Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
feat: Update docs and add analytics dashboards (#1373)
Aman Agarwald0533cb86e84
diff --git a/.gitignore b/.gitignoreindex 19b77e30..0d27fe0b 100644--- a/.gitignore+++ b/.gitignore@@ -9,7 +9,12 @@ path/__pycache__/**bun.lockblogs/+# Secrets / local env (never commit)+.env+**/.env**.env+!.env.example+!**/.env.example.vscode/.idea/.cursor-rules/diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex a7d4fd70..67e327bb 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -8,7 +8,6 @@ Before you begin:- Ensure you have a [GitHub account](https://github.com/join).- Familiarize yourself with git commands in the [Git Documentation](https://git-scm.com/documentation).- Read the README for project setup instructions.-- For Fleet Hub and OpAMP development, review the [OpAMP Deployment Guide](OPAMP_DEPLOYMENT.md) and [Certificate Management Guide](src/opamp-server/CERTIFICATES.md).## Contributing Workflowdiff --git a/OPAMP_DEPLOYMENT.md b/OPAMP_DEPLOYMENT.mddeleted file mode 100644index e531fb5d..00000000--- a/OPAMP_DEPLOYMENT.md+++ /dev/null@@ -1,406 +0,0 @@-# OpenLIT OpAMP Server Deployment Guide--This guide explains how to deploy OpenLIT with the integrated OpAMP server for Fleet Hub management using Docker Compose.--## Overview--The OpenLIT OpAMP server provides secure communication between the OpenLIT platform and OpAMP supervisors managing OpenTelemetry Collectors. This deployment includes:--- **Automatic certificate generation** for secure TLS communication-- **Environment-aware configuration** (development vs production)-- **Dynamic supervisor configuration** based on environment settings-- **Controlled network configuration** with user-configurable security settings--## Quick Start--### 1. Clone and Setup--```bash-git clone https://github.com/openlit/openlit.git-cd openlit-```--### 2. Configure Environment (Optional)--Copy the example environment file and customize:--```bash-cp env.example .env-# Edit .env with your preferred settings-```--### 3. Deploy with Docker Compose--**For Development:**-```bash-docker-compose up -d-```--**For Production:**-```bash-# Production mode is now the default with secure TLS settings-# Set any custom overrides if needed-export OPAMP_ENVIRONMENT=production # Default-export OPAMP_TLS_INSECURE_SKIP_VERIFY=false # Default-export OPAMP_TLS_REQUIRE_CLIENT_CERT=true # Default--docker-compose up -d-```--### 4. Access Services--- **OpenLIT UI**: http://localhost:3000-- **Fleet Hub**: http://localhost:3000/fleet-hub-- **OpAMP Server**: wss://localhost:4320/v1/opamp-- **OpAMP API**: http://localhost:8080--## Environment Configuration--### Available Environment Variables--| Variable | Default | Description |-|----------|---------|-------------|-| `OPAMP_ENVIRONMENT` | `development` | Environment mode (`development`, `production`, `testing`) |-| `OPAMP_TLS_INSECURE_SKIP_VERIFY` | `true` | Skip certificate verification (dev only) |-| `OPAMP_TLS_REQUIRE_CLIENT_CERT` | `false` | Require client certificates |-| `OPAMP_TLS_MIN_VERSION` | `1.2` | Minimum TLS version |-| `OPAMP_TLS_MAX_VERSION` | `1.3` | Maximum TLS version |-| `OPAMP_LOG_LEVEL` | `info` | Logging level |--### Environment Modes--#### Development Mode (Default)-- Certificate verification is skipped (`insecure_skip_verify: true`)-- Client certificates are not required-- Suitable for local development and testing-- **⚠️ Not secure for production use**--#### Production Mode-- Full certificate verification enabled-- Client certificates required by default-- Strict TLS configuration-- Certificate expiry monitoring-- **✅ Production-ready security**--#### Testing Mode-- Similar to development mode-- Optimized for automated testing-- Flexible certificate validation--## Deployment Scenarios--### Scenario 1: Local Development--```bash-# Use default development settings-docker-compose up -d-```--The OpAMP supervisor will automatically connect using `insecure_skip_verify: true`.--### Scenario 2: Production Deployment--```bash-# Set production environment variables-export OPAMP_ENVIRONMENT=production-export OPAMP_TLS_INSECURE_SKIP_VERIFY=false-export OPAMP_TLS_REQUIRE_CLIENT_CERT=true--# Deploy with production settings-docker-compose up -d-```--### Scenario 3: Custom Configuration--Create a `.env` file:--```env-# Custom production configuration-OPAMP_ENVIRONMENT=production-OPAMP_TLS_INSECURE_SKIP_VERIFY=false-OPAMP_TLS_REQUIRE_CLIENT_CERT=true-OPAMP_TLS_MIN_VERSION=1.3-OPAMP_LOG_LEVEL=warn--# Database settings-OPENLIT_DB_PASSWORD=your-secure-password-```--Then deploy:--```bash-docker-compose up -d-```--## Certificate Management--### Automatic Certificate Generation--The deployment automatically generates:-- **CA Certificate**: Self-signed root certificate-- **Server Certificate**: For OpAMP server TLS-- **Client Certificate**: For mutual TLS authentication (required in production)--### Certificate Properties--- **Key Size**: 4096-bit RSA keys-- **Validity**: 10 years (CA), 1 year (server/client)-- **Subject Alternative Names**: Includes localhost, Docker IPs, and container hostnames-- **Extensions**: Proper key usage and extended key usage--### Custom Certificates--For production with custom certificates:--1. **Replace generated certificates** in the container:- ```bash- docker cp your-ca.cert.pem openlit:/app/opamp/certs/cert/ca.cert.pem- docker cp your-server.cert.pem openlit:/app/opamp/certs/server/server.cert.pem- docker cp your-server.key.pem openlit:/app/opamp/certs/server/server.key.pem- ```--2. **Restart the container**:- ```bash- docker-compose restart openlit- ```--## Connecting External Supervisors--### Development Connection--For external OpAMP supervisors in development:--```yaml-# supervisor.yaml-server:- endpoint: wss://your-openlit-host:4320/v1/opamp- tls:- insecure_skip_verify: true--agent:- executable: /path/to/otelcol-contrib- config_files:- - /etc/otel/config.yaml--capabilities:- accepts_remote_config: true- reports_effective_config: true- reports_own_logs: true- reports_health: true-```--### Production Connection--For production deployments with mutual TLS:--1. **Extract certificates**:- ```bash- # Extract CA certificate- docker cp openlit:/app/opamp/certs/cert/ca.cert.pem ./ca.cert.pem-- # Extract client certificates for mutual TLS- docker cp openlit:/app/opamp/certs/client/client.cert.pem ./client.cert.pem- docker cp openlit:/app/opamp/certs/client/client.key.pem ./client.key.pem- ```--2. **Configure supervisor with mutual TLS**:- ```yaml- # supervisor.yaml- server:- endpoint: wss://your-openlit-host:4320/v1/opamp- tls:- insecure_skip_verify: false- ca_file: /path/to/ca.cert.pem- cert_file: /path/to/client.cert.pem- key_file: /path/to/client.key.pem- ```--3. **Deploy supervisor**:- ```bash- ./opampsupervisor --config supervisor.yaml- ```--### Using the Setup Script--Use the included setup script for automated configuration:--```bash-# Extract setup script from container-docker cp openlit:/app/opamp/setup-supervisor.sh ./--# Generate development configuration-./setup-supervisor.sh --development--# Generate production configuration-./setup-supervisor.sh --production --server-url wss://your-server:4320/v1/opamp-```--## Monitoring and Troubleshooting--### Container Logs--```bash-# View all logs-docker-compose logs -f--# View OpAMP specific logs-docker-compose logs -f openlit | grep -E "(OpAMP|OPAMP)"--# View certificate generation logs-docker exec openlit cat /var/log/certs.log-```--### Health Checks--```bash-# Check OpAMP server status-curl -k https://localhost:8080/health--# Test TLS connection-openssl s_client -connect localhost:4320--# View certificate information-docker exec openlit openssl x509 -in /app/opamp/certs/cert/ca.cert.pem -noout -text-```--### Common Issues--#### "certificate signed by unknown authority"--**Cause**: Supervisor doesn't trust the self-signed CA certificate.--**Solutions**:-1. **Development**: Set `insecure_skip_verify: true` in supervisor config-2. **Production**: Provide CA certificate path in supervisor config--#### "connection refused"--**Cause**: OpAMP server not running or network issues.--**Solutions**:-1. Check container status: `docker-compose ps`-2. Verify port mapping: `docker-compose port openlit 4320`-3. Check firewall rules for port 4320--#### "certificate has expired"--**Cause**: Generated certificates have expired.--**Solutions**:-1. Regenerate certificates: `docker-compose restart openlit`-2. Check certificate expiry: `docker exec openlit openssl x509 -in /app/opamp/certs/server/server.cert.pem -noout -dates`--## Security Considerations--### Development Security--- Uses `insecure_skip_verify: true` by default-- Suitable for local development only-- **Never use in production**--### Production Security--- Full certificate verification enabled-- Mutual TLS support available-- Secure cipher suites and TLS versions-- Certificate expiry monitoring--### Best Practices--1. **Use production mode** for production deployments-2. **Rotate certificates** regularly-3. **Secure CA private key** with proper file permissions-4. **Monitor certificate expiry** and set up alerts-5. **Use strong passwords** for database and other services--## Advanced Configuration--### Mutual TLS Configuration--Mutual TLS is enabled by default in production mode. To disable it:--```bash-export OPAMP_TLS_REQUIRE_CLIENT_CERT=false-docker-compose up -d-```--Supervisor configuration with client certificates (required in production):--```yaml-server:- tls:- insecure_skip_verify: false- ca_file: /path/to/ca.cert.pem- cert_file: /path/to/client.cert.pem- key_file: /path/to/client.key.pem-```--### Custom Certificate Validity--Modify certificate validity periods by setting environment variables before deployment:--```bash-export CERT_VALIDITY_DAYS=7300 # 20 years for CA-export SERVER_CERT_DAYS=730 # 2 years for server cert-docker-compose up -d-```--## Migration from Previous Versions--If upgrading from a previous version without OpAMP support:--1. **Backup existing data**:- ```bash- docker-compose down- docker volume create openlit-backup- docker run --rm -v openlit-data:/source -v openlit-backup:/backup alpine cp -r /source/. /backup/- ```--2. **Update configuration**:- ```bash- # Pull latest changes- git pull origin main-- # Update Docker Compose- docker-compose pull- ```--3. **Deploy with new configuration**:- ```bash- docker-compose up -d- ```--## Support and Documentation--- **Fleet Hub Documentation**: [docs/latest/openlit/observability/fleet-hub.mdx](docs/latest/openlit/observability/fleet-hub.mdx)-- **Certificate Management**: [src/opamp-server/CERTIFICATES.md](src/opamp-server/CERTIFICATES.md)-- **OpAMP Specification**: [OpenTelemetry OpAMP](https://github.com/open-telemetry/opamp-spec)--## Environment Variables Reference--### User-Configurable Variables--```bash-# Environment and Security-OPAMP_ENVIRONMENT=development|production|testing-OPAMP_TLS_INSECURE_SKIP_VERIFY=true|false-OPAMP_TLS_REQUIRE_CLIENT_CERT=true|false-OPAMP_TLS_MIN_VERSION=1.0|1.1|1.2|1.3-OPAMP_TLS_MAX_VERSION=1.0|1.1|1.2|1.3-OPAMP_LOG_LEVEL=debug|info|warn|error--# Database Configuration-OPENLIT_DB_NAME=openlit-OPENLIT_DB_USER=default-OPENLIT_DB_PASSWORD=OPENLIT--# Application Configuration-PORT=3000-DOCKER_PORT=3000--# OAuth (Optional)-GOOGLE_CLIENT_ID=your-client-id-GOOGLE_CLIENT_SECRET=your-client-secret-GITHUB_CLIENT_ID=your-client-id-GITHUB_CLIENT_SECRET=your-client-secret-```diff --git a/README.md b/README.mdindex 5b604098..7f6fc4a0 100644--- a/README.md+++ b/README.md@@ -1,7 +1,7 @@<div align="center"><img src="https://github.com/openlit/.github/blob/main/profile/assets/wide-logo-no-bg.png?raw=true" alt="OpenLIT Logo" width="30%">-#### Observability, Evaluations, Rule Engine, Guardrails, Prompts, Vault, Playground, FleetHub+#### Observability, Evaluations, Rule Engine, Guardrails, Prompts, Vault, Playground# Open Source Platform for AI Engineering@@ -49,8 +49,6 @@ This project proudly follows and maintains the [Semantic Conventions](https://gi- 🎮 **Experiment with different LLMs**: Use OpenGround to explore, test and compare various LLMs side by side.-- 🚀 **Fleet Hub for OpAMP Management**: Centrally manage and monitor OpenTelemetry Collectors across your infrastructure using the OpAMP (Open Agent Management Protocol) with secure TLS communication.-## 🚀 Getting Started with LLM Observability```mermaid@@ -347,7 +345,6 @@ We are dedicated to continuously improving OpenLIT. Here's a look at what's been| [Cost Tracking for Custom Models](https://docs.openlit.io/latest/features/pricing) | ✅ Completed || [Real-Time Guardrails Implementation](https://docs.openlit.io/latest/features/guardrails) | ✅ Completed || [Programmatic Evaluation for LLM Response](https://docs.openlit.io/latest/features/evaluations) | ✅ Completed |-| [Fleet Hub for OpAMP Management](https://docs.openlit.io/latest/openlit/observability/fleet-hub) | ✅ Completed || [Auto-Evaluation Metrics Based on Usage](https://github.com/openlit/openlit/issues/470) | 🔜 Coming Soon || [Human Feedback for LLM Events](https://github.com/openlit/openlit/issues/471) | 🔜 Coming Soon || [Dataset Generation Based on LLM Events](https://github.com/openlit/openlit/issues/472) | 🔜 Coming Soon |diff --git a/cli/internal/coding/hook/cursor/handle.go b/cli/internal/coding/hook/cursor/handle.goindex 0686ef05..9bb8377a 100644--- a/cli/internal/coding/hook/cursor/handle.go+++ b/cli/internal/coding/hook/cursor/handle.go@@ -46,6 +46,17 @@ type cursorPayload struct {Text string `json:"text"`DurationMs int64 `json:"duration_ms"`+ // Token usage — Cursor ships these on afterAgentResponse and stop.+ // Absent on older builds and+ // on beforeSubmitPrompt; when present they are authoritative and+ // supersede the char-heuristic estimate. Pointers so we can tell+ // "missing" from an explicit zero.+ InputTokens *int64 `json:"input_tokens"`+ OutputTokens *int64 `json:"output_tokens"`+ CacheReadTokens *int64 `json:"cache_read_tokens"`+ CacheWriteTokens *int64 `json:"cache_write_tokens"`+ Provider string `json:"provider"`+// preToolUse / postToolUse / postToolUseFailureToolName string `json:"tool_name"`ToolUseID string `json:"tool_use_id"`@@ -196,16 +207,36 @@ func handle(ctx context.Context, in normalize.Input) error {// Cursor's `stop` fires when the agent loop terminates. We// emit a small event so dashboards can count loop turns; the// authoritative session end span comes from sessionEnd.+ //+ // When Cursor attaches token counts on stop, stamp them on the event so the UI can show+ // authoritative usage even if afterAgentResponse omitted them.+ attrs := map[string]any{+ "coding_agent.client": in.Vendor,+ "coding_agent.hook.event": event,+ "coding_agent.session.loop.status": p.Status,+ "coding_agent.session.loop.count": p.LoopCount,+ }+ if inTok, outTok, cacheRead, cacheWrite, ok := realTokenUsage(p); ok {+ attrs["gen_ai.usage.input_tokens"] = inTok+ attrs["gen_ai.usage.output_tokens"] = outTok+ attrs["gen_ai.usage.total_tokens"] = inTok + outTok+ if cacheRead > 0 {+ attrs["gen_ai.usage.cache.read_input_tokens"] = cacheRead+ }+ if cacheWrite > 0 {+ attrs["gen_ai.usage.cache.creation_input_tokens"] = cacheWrite+ }+ if p.Model != "" {+ attrs["gen_ai.request.model"] = p.Model+ rate := pricing.Lookup(p.Model)+ attrs["gen_ai.usage.cost"] = rate.Cost(inTok, outTok, cacheRead, cacheWrite)+ }+ }return in.Emit.EmitEvent(normalize.EventEmission{SessionID: sessionID,Name: "coding_agent.session.loop.stop",At: time.Now(),- Attrs: map[string]any{- "coding_agent.client": in.Vendor,- "coding_agent.hook.event": event,- "coding_agent.session.loop.status": p.Status,- "coding_agent.session.loop.count": p.LoopCount,- },+ Attrs: attrs,})case "beforeSubmitPrompt":@@ -510,10 +541,11 @@ func buildPromptTurn(in normalize.Input, p cursorPayload, sessionID string) norm}}now := time.Now()- // Cursor doesn't expose token counts on hook payloads, so we- // estimate from prompt length and apply the static pricing table.- // This is intentionally a lower bound — when the model surfaces- // real usage (Codex rollout, Claude transcript), prefer that.+ // beforeSubmitPrompt does not carry token counts (we observes+ // the same). Stamp a char-heuristic input estimate so the prompt+ // half isn't empty; cost for this half is the input-only estimate+ // and acts as the fallback when afterAgentResponse/stop never+ // report real usage.inputTokens := pricing.EstimateTokens(p.Prompt)rate := pricing.Lookup(p.Model)cost := rate.Cost(inputTokens, 0, 0, 0)@@ -536,9 +568,31 @@ func buildPromptTurn(in normalize.Input, p cursorPayload, sessionID string) normfunc buildResponseTurn(in normalize.Input, p cursorPayload, sessionID string) normalize.LLMTurn {now := time.Now()- outputTokens := pricing.EstimateTokens(p.Text)rate := pricing.Lookup(p.Model)- cost := rate.Cost(0, outputTokens, 0, 0)++ // Prefer Cursor's real token fields when present (same source). These cover the full generation — input,+ // output, and cache — so we bill the complete turn here.+ //+ // Cost note: the matching beforeSubmitPrompt span already billed+ // an input-only estimate. To avoid double-counting input, we bill+ // output + cache from the real counters here and still stamp the+ // real input token count for accurate token dashboards. When real+ // counters are absent we fall back to the char-heuristic on the+ // response text (output only), same as before.+ var (+ inputTokens, outputTokens int64+ cacheRead, cacheWrite int64+ cost float64+ )+ if inTok, outTok, cr, cw, ok := realTokenUsage(p); ok {+ inputTokens, outputTokens = inTok, outTok+ cacheRead, cacheWrite = cr, cw+ cost = rate.Cost(0, outputTokens, cacheRead, cacheWrite)+ } else {+ outputTokens = pricing.EstimateTokens(p.Text)+ cost = rate.Cost(0, outputTokens, 0, 0)+ }+return normalize.LLMTurn{SessionID: sessionID,ConversationID: p.ConversationID,@@ -549,12 +603,37 @@ func buildResponseTurn(in normalize.Input, p cursorPayload, sessionID string) noEndedAt: now,Response: p.Text,AssistantMessageOnly: true,+ InputTokens: inputTokens,OutputTokens: outputTokens,- TotalTokens: outputTokens,+ TotalTokens: inputTokens + outputTokens,+ CacheReadTokens: cacheRead,+ CacheCreationTokens: cacheWrite,CostUSD: cost,}}+// realTokenUsage returns Cursor's authoritative token counters when any+// of the usage fields are present on the payload.+func realTokenUsage(p cursorPayload) (input, output, cacheRead, cacheWrite int64, ok bool) {+ if p.InputTokens == nil && p.OutputTokens == nil &&+ p.CacheReadTokens == nil && p.CacheWriteTokens == nil {+ return 0, 0, 0, 0, false+ }+ if p.InputTokens != nil {+ input = *p.InputTokens+ }+ if p.OutputTokens != nil {+ output = *p.OutputTokens+ }+ if p.CacheReadTokens != nil {+ cacheRead = *p.CacheReadTokens+ }+ if p.CacheWriteTokens != nil {+ cacheWrite = *p.CacheWriteTokens+ }+ return input, output, cacheRead, cacheWrite, true+}+func buildToolCallSuccess(in normalize.Input, p cursorPayload, sessionID string) normalize.ToolCall {startedAt := time.Now().Add(-time.Duration(p.Duration) * time.Millisecond)return normalize.ToolCall{diff --git a/cli/internal/coding/hook/cursor/handle_test.go b/cli/internal/coding/hook/cursor/handle_test.goindex d9daf394..4c16f623 100644--- a/cli/internal/coding/hook/cursor/handle_test.go+++ b/cli/internal/coding/hook/cursor/handle_test.go@@ -4,6 +4,7 @@ import ("context""encoding/json""os"+ "strings""testing""github.com/openlit/openlit/cli/internal/coding/normalize"@@ -210,3 +211,125 @@ func TestCursorSessionIDPrefersConversation(t *testing.T) {em.sessions[0].SessionID, "convo-stable")}}++// TestCursorAfterAgentResponsePrefersRealTokens:+// when Cursor ships input_tokens / output_tokens / cache_* on+// afterAgentResponse, those counters are authoritative and must beat+// the char-heuristic estimate derived from `text`.+func TestCursorAfterAgentResponsePrefersRealTokens(t *testing.T) {+ withIsolatedCache(t)++ em := &recordingEmitter{}+ in := inputBuilder(t, em)++ // Long text would estimate to thousands of tokens; real counters+ // below are tiny so a regression to the heuristic is obvious.+ longText := strings.Repeat("word ", 2000)+ if err := handle(context.Background(), in("afterAgentResponse", map[string]any{+ "hook_event_name": "afterAgentResponse",+ "conversation_id": "convo-1",+ "generation_id": "gen-1",+ "model": "composer-2.5",+ "text": longText,+ "input_tokens": int64(1200),+ "output_tokens": int64(340),+ "cache_read_tokens": int64(800),+ "cache_write_tokens": int64(100),+ })); err != nil {+ t.Fatalf("afterAgentResponse: %v", err)+ }+ if len(em.llmTurns) != 1 {+ t.Fatalf("expected one llm turn; got %d", len(em.llmTurns))+ }+ got := em.llmTurns[0]+ if got.InputTokens != 1200 {+ t.Errorf("InputTokens = %d, want 1200 (real payload)", got.InputTokens)+ }+ if got.OutputTokens != 340 {+ t.Errorf("OutputTokens = %d, want 340 (real payload)", got.OutputTokens)+ }+ if got.CacheReadTokens != 800 {+ t.Errorf("CacheReadTokens = %d, want 800", got.CacheReadTokens)+ }+ if got.CacheCreationTokens != 100 {+ t.Errorf("CacheCreationTokens = %d, want 100", got.CacheCreationTokens)+ }+ if got.TotalTokens != 1540 {+ t.Errorf("TotalTokens = %d, want 1540", got.TotalTokens)+ }+ if got.CostUSD <= 0 {+ t.Errorf("CostUSD = %v, want > 0 from real token pricing", got.CostUSD)+ }+}++// TestCursorAfterAgentResponseFallsBackToEstimate covers older Cursor+// builds that omit token fields — we still estimate from response text.+func TestCursorAfterAgentResponseFallsBackToEstimate(t *testing.T) {+ withIsolatedCache(t)++ em := &recordingEmitter{}+ in := inputBuilder(t, em)++ text := "hello world from cursor" // short, deterministic estimate+ if err := handle(context.Background(), in("afterAgentResponse", map[string]any{+ "hook_event_name": "afterAgentResponse",+ "conversation_id": "convo-2",+ "model": "composer-2.5",+ "text": text,+ })); err != nil {+ t.Fatalf("afterAgentResponse: %v", err)+ }+ if len(em.llmTurns) != 1 {+ t.Fatalf("expected one llm turn; got %d", len(em.llmTurns))+ }+ got := em.llmTurns[0]+ if got.InputTokens != 0 {+ t.Errorf("InputTokens = %d, want 0 when no real counters", got.InputTokens)+ }+ if got.OutputTokens <= 0 {+ t.Errorf("OutputTokens = %d, want estimated > 0", got.OutputTokens)+ }+ if got.CacheReadTokens != 0 || got.CacheCreationTokens != 0 {+ t.Errorf("cache tokens should be 0 on estimate path; got read=%d write=%d",+ got.CacheReadTokens, got.CacheCreationTokens)+ }+}++// TestCursorStopStampsRealTokenAttrs ensures stop surfaces token counters on the loop event.+func TestCursorStopStampsRealTokenAttrs(t *testing.T) {+ withIsolatedCache(t)++ em := &recordingEmitter{}+ in := inputBuilder(t, em)++ if err := handle(context.Background(), in("stop", map[string]any{+ "hook_event_name": "stop",+ "conversation_id": "convo-3",+ "status": "completed",+ "loop_count": 2,+ "model": "composer-2.5",+ "input_tokens": int64(500),+ "output_tokens": int64(100),+ "cache_read_tokens": int64(200),+ "cache_write_tokens": int64(50),+ })); err != nil {+ t.Fatalf("stop: %v", err)+ }+ if len(em.events) != 1 {+ t.Fatalf("expected one stop event; got %d", len(em.events))+ }+ attrs := em.events[0].Attrs+ if attrs["gen_ai.usage.input_tokens"] != int64(500) {+ t.Errorf("input_tokens attr = %v, want 500", attrs["gen_ai.usage.input_tokens"])+ }+ if attrs["gen_ai.usage.output_tokens"] != int64(100) {+ t.Errorf("output_tokens attr = %v, want 100", attrs["gen_ai.usage.output_tokens"])+ }+ if attrs["gen_ai.usage.cache.read_input_tokens"] != int64(200) {+ t.Errorf("cache read attr = %v, want 200", attrs["gen_ai.usage.cache.read_input_tokens"])+ }+ if attrs["gen_ai.usage.cost"] == nil {+ t.Errorf("expected gen_ai.usage.cost on stop when model + tokens present")+ }+}+diff --git a/cli/internal/coding/hook/hook.go b/cli/internal/coding/hook/hook.goindex 1a844643..c121b402 100644--- a/cli/internal/coding/hook/hook.go+++ b/cli/internal/coding/hook/hook.go@@ -331,8 +331,8 @@ func run(cmd *cobra.Command, vendor, event string) (rerr error) {// subagent that knows its parent's chat id, stamp it so the UI// can fold the subagent's spans under the parent's chat row// instead of listing the subagent as a separate session.- if cached.ParentConversationID != "" {- sessionAttrs["coding_agent.agent.parent_id"] = cached.ParentConversationID+ if parentID := foreignParentID(cached.ParentConversationID, cached.ConversationID, sessionID); parentID != "" {+ sessionAttrs["coding_agent.agent.parent_id"] = parentID}// Subagent flag — toggled by Cursor's is_background_agent /// is_parallel_worker on subagent payloads. Used by the Sessions@@ -461,6 +461,21 @@ func pickAdapter(vendor string) (normalize.Adapter, error) {}}+// foreignParentID returns a parent conversation id worth stamping as+// coding_agent.agent.parent_id, or "" when the probe is empty / a+// Cursor self-parent echo (parent_conversation_id == this chat's own+// id). Stamping self-parent made the Sessions list treat the parent+// chat as a subagent and hide it.+func foreignParentID(parentID, conversationID, sessionID string) string {+ if parentID == "" {+ return ""+ }+ if parentID == conversationID || parentID == sessionID {+ return ""+ }+ return parentID+}+// isClaudeCodeVendor returns true when the vendor key resolves to the// Claude Code adapter. Accepts any of the historical aliases the host// plugins have used (`cc`, `claudecode`, `claude_code`, …) so callersdiff --git a/cli/internal/coding/hook/hook_test.go b/cli/internal/coding/hook/hook_test.goindex 2e4393e8..8278a53b 100644--- a/cli/internal/coding/hook/hook_test.go+++ b/cli/internal/coding/hook/hook_test.go@@ -5,6 +5,33 @@ import ("testing")+// TestForeignParentID ensures Cursor's self-parent echo+// (parent_conversation_id == conversation.id) is not stamped as a+// real parent link. That echo previously flipped is_subagent on the+// parent chat and hid it from the Sessions list.+func TestForeignParentID(t *testing.T) {+ cases := []struct {+ name string+ parent string+ conversationID string+ sessionID string+ want string+ }{+ {name: "empty", parent: "", conversationID: "c1", sessionID: "s1", want: ""},+ {name: "foreign", parent: "parent-1", conversationID: "c1", sessionID: "s1", want: "parent-1"},+ {name: "self_eq_conversation", parent: "c1", conversationID: "c1", sessionID: "s1", want: ""},+ {name: "self_eq_session", parent: "s1", conversationID: "c1", sessionID: "s1", want: ""},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ if got := foreignParentID(tc.parent, tc.conversationID, tc.sessionID); got != tc.want {+ t.Fatalf("foreignParentID(%q,%q,%q) = %q, want %q",+ tc.parent, tc.conversationID, tc.sessionID, got, tc.want)+ }+ })+ }+}+// TestIsClaudeCodeVendor covers the small whitelist that drives the// host-mismatch guard's left-hand side. The list is small but the// downstream behaviour (emit vs. drop) is binary, so we want adiff --git a/cli/internal/coding/identity/identity_test.go b/cli/internal/coding/identity/identity_test.goindex b686f0a3..67459898 100644--- a/cli/internal/coding/identity/identity_test.go+++ b/cli/internal/coding/identity/identity_test.go@@ -9,7 +9,7 @@ import (func TestResolveForVendor_ClaudeCodeReadsOAuthEmail(t *testing.T) {tmp := t.TempDir()t.Setenv("HOME", tmp)- const wantEmail = "[email protected]"+ const wantEmail = "[email protected]"body := `{"oauthAccount":{"accountUuid":"abc","emailAddress":"` + wantEmail + `"}}`if err := os.WriteFile(filepath.Join(tmp, ".claude.json"), []byte(body), 0o600); err != nil {t.Fatalf("write .claude.json: %v", err)diff --git a/cli/internal/coding/pricing/pricing.go b/cli/internal/coding/pricing/pricing.goindex efbc8f34..80d0a1bc 100644--- a/cli/internal/coding/pricing/pricing.go+++ b/cli/internal/coding/pricing/pricing.go@@ -2,10 +2,11 @@// small static table of public list prices.//// The CLI is short-lived and offline-safe; we do not fetch live pricing.-// Rates here are last-known public list prices (2026-05) and are kept-// deliberately conservative — we'd rather under-bill than over-bill the-// dashboard. The server can replace these with org-specific contract-// rates at materialization time.+// Rates here are last-known public list prices (2026-07, sourced from+// https://cursor.com/docs/models-and-pricing plus provider list prices)+// and are kept deliberately conservative — we'd rather under-bill than+// over-bill the dashboard. The server can replace these with+// org-specific contract rates at materialization time.//// All rates are USD per 1,000,000 tokens.package pricing@@ -103,37 +104,93 @@ type entry struct {// edits. Order matters — first-match wins, so put more-specific// patterns ahead of broader ones.//-// References (verified May 2026):+// References (verified July 2026 against Cursor docs):+// - Cursor (Composer + Auto + Grok 4.5 + routed API models):+// https://cursor.com/docs/models-and-pricing// - Anthropic: https://www.anthropic.com/pricing#anthropic-api// - OpenAI / Codex: https://openai.com/api/pricing/ +// https://developers.openai.com/codex/pricing// - Google Gemini: https://ai.google.dev/pricing// - xAI Grok: https://docs.x.ai/docs/models-// - Cursor (Composer + routed models):-// https://cursor.com/docs/models-and-pricing// - Moonshot Kimi: https://platform.moonshot.ai/docs/pricing//// IMPORTANT: the order below is load-bearing. We've been bitten more-// than once by a generic `claude-opus-4` pattern eating the new $5/$25-// Opus 4.5+ pricing. ALWAYS list versioned variants before the family-// prefix.+// than once by a generic `claude-opus-4` / `gpt-5` pattern eating a+// newer SKU's pricing (e.g. gpt-5.6-sol matching "gpt-5"). ALWAYS+// list versioned variants before the family prefix.//-// CacheCreationPer1M (Anthropic only) is set to 1.25x InputPer1M per-// Anthropic's prompt caching docs. CachedReadPer1M is ~10% of input.-// OpenAI / Google / xAI / Cursor / Moonshot leave CacheCreationPer1M-// at 0 — their APIs don't surface a separate cache-write count.+// CacheCreationPer1M (Anthropic + GPT-5.6) is set when the provider+// bills cache writes separately. CachedReadPer1M is ~10% of input for+// Anthropic / most OpenAI; Cursor first-party models publish flat+// cache-read rates. Vendors without a separate cache-write signal+// leave CacheCreationPer1M at 0.var table = []entry{+ // --- Cursor first-party pool -----------------------------------+ // Auto / Composer 2.5 / Grok 4.5 per cursor.com/docs/models-and-pricing.+ // Auto is matched only as a whole-word-ish id — put it last among+ // Cursor entries so it can't swallow "…-auto…" suffixes on other+ // vendors. Cursor stamps the literal model id "auto" when Auto is+ // selected.+ {+ match: []string{"composer-2-5", "composer-2.5", "composer-2"},+ rate: Rate{InputPer1M: 0.50, OutputPer1M: 2.50, CachedReadPer1M: 0.20},+ },+ {+ // Legacy Composer 1.5 (no longer on Cursor's public table;+ // kept so historical sessions still price).+ match: []string{"composer-1-5", "composer-1.5"},+ rate: Rate{InputPer1M: 3.50, OutputPer1M: 17.50, CachedReadPer1M: 0.35},+ },+ {+ match: []string{"composer-1", "composer"},+ rate: Rate{InputPer1M: 1.25, OutputPer1M: 10.00, CachedReadPer1M: 0.125},+ },+ {+ match: []string{"grok-4.5", "grok-4-5"},+ rate: Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.50},+ },+ {+ // Older Cursor/xAI Grok ladder ids still seen in the wild.+ match: []string{"grok-4-20", "grok-4.20"},+ rate: Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.50},+ },+ {+ match: []string{"grok-4-3", "grok-4.3"},+ rate: Rate{InputPer1M: 1.25, OutputPer1M: 2.50, CachedReadPer1M: 0.20},+ },+ {+ match: []string{"grok-build-0-1", "grok-build-0.1", "grok-build"},+ rate: Rate{InputPer1M: 1.00, OutputPer1M: 2.00, CachedReadPer1M: 0.20},+ },+ {+ // Cursor Auto pool (also matched by bare "auto" at the end of+ // the table so the substring can't eat unrelated model ids).+ match: []string{"cursor-auto"},+ rate: Rate{InputPer1M: 1.25, OutputPer1M: 6.00, CachedReadPer1M: 0.25, CacheCreationPer1M: 1.25},+ },+// --- Anthropic Claude family -----------------------------------// Opus 4.5+ dropped to $5/$25 in 2025-10 (down from the original// Opus 4 / 4.1 launch price of $15/$75). We special-case the// later versions so Cursor's `claude-opus-4-7-thinking-*` SKUs and// the bare `claude-opus-4-5` / `4-6` / `4-7` / `4-8` ids price- // correctly. Note the "-fast" variants (Anthropic priority tier,- // also exposed via Cursor) carry a ~6x premium.+ // correctly.+ //+ // Fast mode: Cursor docs list Opus 4.7 fast at $30/$150. Opus 4.8+ // fast is documented as 3x cheaper than 4.7 fast → $10/$50.+ {+ match: []string{"claude-opus-4-8-fast"},+ rate: Rate{InputPer1M: 10.00, OutputPer1M: 50.00, CachedReadPer1M: 1.00, CacheCreationPer1M: 12.50},+ },{- match: []string{"claude-opus-4-8-fast", "claude-opus-4-7-fast", "claude-opus-4-6-fast", "claude-opus-4-5-fast"},+ match: []string{"claude-opus-4-7-fast", "claude-opus-4-6-fast", "claude-opus-4-5-fast"},rate: Rate{InputPer1M: 30.00, OutputPer1M: 150.00, CachedReadPer1M: 3.00, CacheCreationPer1M: 37.50},},+ {+ // Claude Fable 5 (~2x Opus 4.8).+ match: []string{"claude-fable-5", "claude-fable", "fable-5"},+ rate: Rate{InputPer1M: 10.00, OutputPer1M: 50.00, CachedReadPer1M: 1.00, CacheCreationPer1M: 12.50},+ },{match: []string{"claude-opus-4-8", "claude-opus-4-7", "claude-opus-4-6", "claude-opus-4-5"},rate: Rate{InputPer1M: 5.00, OutputPer1M: 25.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25},@@ -144,13 +201,23 @@ var table = []entry{rate: Rate{InputPer1M: 15.00, OutputPer1M: 75.00, CachedReadPer1M: 1.50, CacheCreationPer1M: 18.75},},{- // Generic "claude-opus-4" / "claude-4-opus" with no version- // suffix. We bias toward the modern rate because new SDKs and- // Cursor builds default to the latest Opus, and undercharging is- // preferable to a 3x overcharge.+ // Generic "claude-opus-4" with no version suffix. Bias toward+ // the modern rate — undercharging beats a 3x overcharge.match: []string{"claude-opus-4"},rate: Rate{InputPer1M: 5.00, OutputPer1M: 25.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25},},+ {+ // Claude 4 Sonnet 1M context: 2x the standard Sonnet rates.+ match: []string{"claude-4-sonnet-1m", "claude-sonnet-4-1m", "sonnet-4-1m", "sonnet-1m"},+ rate: Rate{InputPer1M: 6.00, OutputPer1M: 22.50, CachedReadPer1M: 0.60, CacheCreationPer1M: 7.50},+ },+ {+ // Claude Sonnet 5 (list $3/$15; Cursor notes a launch promo of+ // $2/$10 through 2026-08-31 — we keep list price so the table+ // stays correct after the promo ends).+ match: []string{"claude-sonnet-5", "claude-5-sonnet"},+ rate: Rate{InputPer1M: 3.00, OutputPer1M: 15.00, CachedReadPer1M: 0.30, CacheCreationPer1M: 3.75},+ },{match: []string{"claude-sonnet-4", "claude-4-sonnet", "claude-4.5-sonnet", "claude-4.6-sonnet", "claude-4.7-sonnet"},rate: Rate{InputPer1M: 3.00, OutputPer1M: 15.00, CachedReadPer1M: 0.30, CacheCreationPer1M: 3.75},@@ -184,10 +251,27 @@ var table = []entry{rate: Rate{InputPer1M: 15.00, OutputPer1M: 75.00, CachedReadPer1M: 1.50, CacheCreationPer1M: 18.75},},// --- OpenAI ----------------------------------------------------- // GPT-5.x ladder. Versions matter: 5.0 / 5.1 share $1.25/$10, 5.2 /- // 5.3 jump to $1.75/$14, 5.4 base sits at $2.50/$15 (the surprise- // price hike), 5.5 jumps again to $5/$30. Mini and nano variants- // price separately, so they go first.+ // GPT-5.x ladder. GPT-5.6 Sol/Terra/Luna MUST come before the bare+ // `gpt-5` catch-all — `strings.Contains("gpt-5.6-sol", "gpt-5")`+ // is true and previously under-billed Sol ~4x.+ {+ match: []string{"gpt-5.6-sol", "gpt-5-6-sol"},+ rate: Rate{InputPer1M: 5.00, OutputPer1M: 30.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25},+ },+ {+ match: []string{"gpt-5.6-terra", "gpt-5-6-terra"},+ rate: Rate{InputPer1M: 2.50, OutputPer1M: 15.00, CachedReadPer1M: 0.25, CacheCreationPer1M: 3.125},+ },+ {+ match: []string{"gpt-5.6-luna", "gpt-5-6-luna"},+ rate: Rate{InputPer1M: 1.00, OutputPer1M: 6.00, CachedReadPer1M: 0.10, CacheCreationPer1M: 1.25},+ },+ {+ // Bare gpt-5.6 with no Sol/Terra/Luna suffix — default to Sol+ // (Cursor's Max-Mode frontier tier for the 5.6 line).+ match: []string{"gpt-5.6", "gpt-5-6"},+ rate: Rate{InputPer1M: 5.00, OutputPer1M: 30.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25},+ },{match: []string{"gpt-5.5-pro", "gpt-5-5-pro"},rate: Rate{InputPer1M: 30.00, OutputPer1M: 180.00},@@ -295,9 +379,10 @@ var table = []entry{rate: Rate{InputPer1M: 1.10, OutputPer1M: 4.40},},// --- Google Gemini ---------------------------------------------+ // Cache-read rates from Cursor's models-and-pricing table.{match: []string{"gemini-3.1-pro", "gemini-3-1-pro"},- rate: Rate{InputPer1M: 2.00, OutputPer1M: 12.00},+ rate: Rate{InputPer1M: 2.00, OutputPer1M: 12.00, CachedReadPer1M: 0.20},},{match: []string{"gemini-3.1-flash-lite", "gemini-3-1-flash-lite"},@@ -305,15 +390,15 @@ var table = []entry{},{match: []string{"gemini-3.5-flash", "gemini-3-5-flash"},- rate: Rate{InputPer1M: 1.50, OutputPer1M: 9.00},+ rate: Rate{InputPer1M: 1.50, OutputPer1M: 9.00, CachedReadPer1M: 0.15},},{match: []string{"gemini-3-pro-image-preview", "gemini-3-pro"},- rate: Rate{InputPer1M: 2.00, OutputPer1M: 12.00},+ rate: Rate{InputPer1M: 2.00, OutputPer1M: 12.00, CachedReadPer1M: 0.20},},{match: []string{"gemini-3-flash"},- rate: Rate{InputPer1M: 0.50, OutputPer1M: 3.00},+ rate: Rate{InputPer1M: 0.50, OutputPer1M: 3.00, CachedReadPer1M: 0.05},},{match: []string{"gemini-2.5-pro", "gemini-2-5-pro"},@@ -325,7 +410,7 @@ var table = []entry{},{match: []string{"gemini-2.5-flash", "gemini-2-5-flash"},- rate: Rate{InputPer1M: 0.30, OutputPer1M: 2.50},+ rate: Rate{InputPer1M: 0.30, OutputPer1M: 2.50, CachedReadPer1M: 0.03},},{match: []string{"gemini-2.0-flash", "gemini-2-0-flash"},@@ -339,37 +424,11 @@ var table = []entry{match: []string{"gemini-1.5-flash", "gemini-1-5-flash"},rate: Rate{InputPer1M: 0.075, OutputPer1M: 0.30},},- // --- xAI Grok --------------------------------------------------- // Grok 4.x ladder per Cursor + xAI docs. No explicit cache rates;- // xAI does not expose cached_tokens.- {- match: []string{"grok-4-20", "grok-4.20"},- rate: Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.20},- },- {- match: []string{"grok-4-3", "grok-4.3"},- rate: Rate{InputPer1M: 1.25, OutputPer1M: 2.50, CachedReadPer1M: 0.20},- },- {- match: []string{"grok-build-0-1", "grok-build-0.1", "grok-build"},- rate: Rate{InputPer1M: 1.00, OutputPer1M: 2.00, CachedReadPer1M: 0.20},- },- // --- Cursor Composer ------------------------------------------- // Cursor's own models per https://cursor.com/docs/models-and-pricing- // (no separate cache write; cache reads are flat).- {- match: []string{"composer-2-5", "composer-2.5", "composer-2"},- rate: Rate{InputPer1M: 0.50, OutputPer1M: 2.50, CachedReadPer1M: 0.20},- },- {- match: []string{"composer-1-5", "composer-1.5"},- rate: Rate{InputPer1M: 3.50, OutputPer1M: 17.50, CachedReadPer1M: 0.35},- },+ // --- Moonshot Kimi --------------------------------------------{- match: []string{"composer-1", "composer"},- rate: Rate{InputPer1M: 1.25, OutputPer1M: 10.00, CachedReadPer1M: 0.125},+ match: []string{"kimi-k2.7", "kimi-k2-7", "kimi-2.7"},+ rate: Rate{InputPer1M: 0.95, OutputPer1M: 4.00, CachedReadPer1M: 0.19},},- // --- Moonshot Kimi --------------------------------------------{match: []string{"kimi-k2-5", "kimi-k2.5", "kimi-2-5"},rate: Rate{InputPer1M: 0.60, OutputPer1M: 3.00, CachedReadPer1M: 0.10},@@ -378,4 +437,12 @@ var table = []entry{match: []string{"kimi-k2", "kimi-2"},rate: Rate{InputPer1M: 0.60, OutputPer1M: 3.00, CachedReadPer1M: 0.10},},+ // --- Cursor Auto (last) ----------------------------------------+ // Bare "auto" MUST be last: Contains("…", "auto") is a broad+ // substring and would otherwise steal any future model id that+ // happens to include those four letters.+ {+ match: []string{"auto"},+ rate: Rate{InputPer1M: 1.25, OutputPer1M: 6.00, CachedReadPer1M: 0.25, CacheCreationPer1M: 1.25},+ },}diff --git a/cli/internal/coding/pricing/pricing_test.go b/cli/internal/coding/pricing/pricing_test.goindex 2c198d49..90c7d305 100644--- a/cli/internal/coding/pricing/pricing_test.go+++ b/cli/internal/coding/pricing/pricing_test.go@@ -22,9 +22,14 @@ func TestLookupAnthropic(t *testing.T) {{"claude-opus-4-7-thinking-xhigh", Rate{InputPer1M: 5.00, OutputPer1M: 25.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25}},{"claude-opus-4-6", Rate{InputPer1M: 5.00, OutputPer1M: 25.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25}},{"claude-opus-4-5", Rate{InputPer1M: 5.00, OutputPer1M: 25.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25}},- // Anthropic priority-tier "-fast" variants carry a 6x markup- // (and roughly 6x cache-write rate); Cursor exposes these.- {"claude-opus-4-8-fast", Rate{InputPer1M: 30.00, OutputPer1M: 150.00, CachedReadPer1M: 3.00, CacheCreationPer1M: 37.50}},+ // Opus 4.7 fast is $30/$150; Opus 4.8 fast is 3x cheaper per+ // Cursor docs ($10/$50).+ {"claude-opus-4-8-fast", Rate{InputPer1M: 10.00, OutputPer1M: 50.00, CachedReadPer1M: 1.00, CacheCreationPer1M: 12.50}},+ {"claude-opus-4-7-fast", Rate{InputPer1M: 30.00, OutputPer1M: 150.00, CachedReadPer1M: 3.00, CacheCreationPer1M: 37.50}},+ // Claude Fable 5 / Sonnet 5 / Sonnet 1M from Cursor pricing table.+ {"claude-fable-5", Rate{InputPer1M: 10.00, OutputPer1M: 50.00, CachedReadPer1M: 1.00, CacheCreationPer1M: 12.50}},+ {"claude-sonnet-5", Rate{InputPer1M: 3.00, OutputPer1M: 15.00, CachedReadPer1M: 0.30, CacheCreationPer1M: 3.75}},+ {"claude-4-sonnet-1m", Rate{InputPer1M: 6.00, OutputPer1M: 22.50, CachedReadPer1M: 0.60, CacheCreationPer1M: 7.50}},// Legacy Opus 4.0 / 4.1 are still on the original $15/$75 list.{"claude-opus-4-0", Rate{InputPer1M: 15.00, OutputPer1M: 75.00, CachedReadPer1M: 1.50, CacheCreationPer1M: 18.75}},{"claude-opus-4-1", Rate{InputPer1M: 15.00, OutputPer1M: 75.00, CachedReadPer1M: 1.50, CacheCreationPer1M: 18.75}},@@ -62,6 +67,11 @@ func TestLookupOpenAI(t *testing.T) {{"gpt-5.4-mini", Rate{InputPer1M: 0.75, OutputPer1M: 4.50, CachedReadPer1M: 0.075}},// GPT-5.5 frontier tier.{"gpt-5.5", Rate{InputPer1M: 5.00, OutputPer1M: 30.00, CachedReadPer1M: 0.50}},+ // GPT-5.6 Sol/Terra/Luna — must NOT fall through to bare gpt-5.+ {"gpt-5.6-sol", Rate{InputPer1M: 5.00, OutputPer1M: 30.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25}},+ {"gpt-5.6-terra", Rate{InputPer1M: 2.50, OutputPer1M: 15.00, CachedReadPer1M: 0.25, CacheCreationPer1M: 3.125}},+ {"gpt-5.6-luna", Rate{InputPer1M: 1.00, OutputPer1M: 6.00, CachedReadPer1M: 0.10, CacheCreationPer1M: 1.25}},+ {"gpt-5.6-sol-medium", Rate{InputPer1M: 5.00, OutputPer1M: 30.00, CachedReadPer1M: 0.50, CacheCreationPer1M: 6.25}},// o-series — confirm o3 isn't being eaten by the gpt-5 family.{"o3", Rate{InputPer1M: 2.00, OutputPer1M: 8.00}},{"o4-mini", Rate{InputPer1M: 1.10, OutputPer1M: 4.40}},@@ -80,12 +90,12 @@ func TestLookupGoogle(t *testing.T) {want Rate}{{"gemini-2.5-pro", Rate{InputPer1M: 1.25, OutputPer1M: 10.00}},- {"gemini-2.5-flash", Rate{InputPer1M: 0.30, OutputPer1M: 2.50}},+ {"gemini-2.5-flash", Rate{InputPer1M: 0.30, OutputPer1M: 2.50, CachedReadPer1M: 0.03}},{"gemini-2.5-flash-lite", Rate{InputPer1M: 0.10, OutputPer1M: 0.40}},- {"gemini-3-flash", Rate{InputPer1M: 0.50, OutputPer1M: 3.00}},- {"gemini-3-pro", Rate{InputPer1M: 2.00, OutputPer1M: 12.00}},- {"gemini-3.5-flash", Rate{InputPer1M: 1.50, OutputPer1M: 9.00}},- {"gemini-3.1-pro", Rate{InputPer1M: 2.00, OutputPer1M: 12.00}},+ {"gemini-3-flash", Rate{InputPer1M: 0.50, OutputPer1M: 3.00, CachedReadPer1M: 0.05}},+ {"gemini-3-pro", Rate{InputPer1M: 2.00, OutputPer1M: 12.00, CachedReadPer1M: 0.20}},+ {"gemini-3.5-flash", Rate{InputPer1M: 1.50, OutputPer1M: 9.00, CachedReadPer1M: 0.15}},+ {"gemini-3.1-pro", Rate{InputPer1M: 2.00, OutputPer1M: 12.00, CachedReadPer1M: 0.20}},}for _, tt := range tests {got := Lookup(tt.model)@@ -100,15 +110,19 @@ func TestLookupXAIAndCursorAndMoonshot(t *testing.T) {model stringwant Rate}{- // xAI Grok ladder.- {"grok-4-20", Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.20}},+ // Cursor first-party + Grok ladder (cursor.com/docs/models-and-pricing).+ {"grok-4.5", Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.50}},+ {"grok-4-5", Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.50}},+ {"grok-4-20", Rate{InputPer1M: 2.00, OutputPer1M: 6.00, CachedReadPer1M: 0.50}},{"grok-4-3", Rate{InputPer1M: 1.25, OutputPer1M: 2.50, CachedReadPer1M: 0.20}},{"grok-build-0-1", Rate{InputPer1M: 1.00, OutputPer1M: 2.00, CachedReadPer1M: 0.20}},- // Cursor Composer.{"composer-2-5", Rate{InputPer1M: 0.50, OutputPer1M: 2.50, CachedReadPer1M: 0.20}},{"composer-1-5", Rate{InputPer1M: 3.50, OutputPer1M: 17.50, CachedReadPer1M: 0.35}},{"composer-1", Rate{InputPer1M: 1.25, OutputPer1M: 10.00, CachedReadPer1M: 0.125}},- // Moonshot.+ {"auto", Rate{InputPer1M: 1.25, OutputPer1M: 6.00, CachedReadPer1M: 0.25, CacheCreationPer1M: 1.25}},+ {"cursor-auto", Rate{InputPer1M: 1.25, OutputPer1M: 6.00, CachedReadPer1M: 0.25, CacheCreationPer1M: 1.25}},+ // Moonshot — K2.7 must not fall through to K2 rates.+ {"kimi-k2.7-code", Rate{InputPer1M: 0.95, OutputPer1M: 4.00, CachedReadPer1M: 0.19}},{"kimi-k2-5", Rate{InputPer1M: 0.60, OutputPer1M: 3.00, CachedReadPer1M: 0.10}},}for _, tt := range tests {diff --git a/cli/internal/coding/pricing/tokens.go b/cli/internal/coding/pricing/tokens.goindex 20180193..39f0d3ff 100644--- a/cli/internal/coding/pricing/tokens.go+++ b/cli/internal/coding/pricing/tokens.go@@ -7,11 +7,13 @@ import "strings"// aware floor (so we never under-bill multi-word prompts).//// This is a fallback for vendors that don't expose token counts on-// their hooks (Cursor's beforeSubmitPrompt / afterAgentResponse give us-// only the text). When the vendor does report usage (Claude Code's-// transcript JSONL, Codex rollout JSONL), prefer that authoritative-// number — this estimator is intentionally conservative so we never-// silently inflate cost.+// every hook event (Cursor's beforeSubmitPrompt still gives us only+// the text; afterAgentResponse / stop may carry real+// input_tokens / output_tokens / cache_* — prefer those when present).+// When the vendor does report usage (Claude Code's transcript JSONL,+// Codex rollout JSONL, Cursor stop/afterAgentResponse), prefer that+// authoritative number — this estimator is intentionally conservative+// so we never silently inflate cost.//// Heuristic chosen to match what Anthropic's tokenizer roughly produces// on English/code prompts (their actual ratio runs 3.5–4.5 chars/tokdiff --git a/cli/internal/otlp/attrs.go b/cli/internal/otlp/attrs.goindex a6d32762..aee81e52 100644--- a/cli/internal/otlp/attrs.go+++ b/cli/internal/otlp/attrs.go@@ -70,24 +70,36 @@ func perEventSpansAllowed(mode string) bool {return mode != semconv.CodingAgentContentCaptureMinimal}-// inferProvider returns the OTel-standard `gen_ai.system` value based-// on the model name and (as a last resort) the vendor. We deliberately-// only handle the model families our adapters report; everything else-// gets the empty string so dashboards know not to render a misleading-// system value. Extend as we add more agents.+// inferProvider returns the OTel-standard `gen_ai.system` /+// `gen_ai.provider.name` value based on the model name and (as a last+// resort) the vendor. We deliberately only handle the model families+// our adapters report; everything else gets the empty string so+// dashboards know not to render a misleading system value.+//+// Cursor stamps ids like `cursor-grok-4.5-high`, `composer-2.5`, and+// `auto` — strip the `cursor-` prefix before matching so first-party+// and routed models still land on the right provider bucket (used by+// the Coding Agents dashboard / traces). LLM dashboard widgets+// exclude coding-agent spans separately.func inferProvider(model, vendor string) string {- m := strings.ToLower(model)+ m := strings.ToLower(strings.TrimSpace(model))+ m = strings.TrimPrefix(m, "cursor-")switch {- case strings.HasPrefix(m, "claude"), strings.HasPrefix(m, "anthropic/"):+ case strings.HasPrefix(m, "claude"), strings.HasPrefix(m, "anthropic/"),+ strings.Contains(m, "claude-"):return "anthropic"case strings.HasPrefix(m, "gpt"), strings.HasPrefix(m, "o1"),strings.HasPrefix(m, "o3"), strings.HasPrefix(m, "o4"),- strings.HasPrefix(m, "openai/"):+ strings.HasPrefix(m, "openai/"), strings.Contains(m, "codex"):return "openai"case strings.HasPrefix(m, "gemini"), strings.HasPrefix(m, "google/"):return "google"- case strings.HasPrefix(m, "grok"):+ case strings.HasPrefix(m, "grok"), strings.Contains(m, "grok"):return "xai"+ case strings.HasPrefix(m, "composer"), m == "auto":+ return "cursor"+ case strings.HasPrefix(m, "kimi"), strings.HasPrefix(m, "moonshot"):+ return "moonshot"case strings.HasPrefix(m, "deepseek"):return "deepseek"}@@ -96,6 +108,8 @@ func inferProvider(model, vendor string) string {return "anthropic"case "codex":return "openai"+ case "cursor":+ return "cursor"}return ""}diff --git a/cli/internal/otlp/attrs_test.go b/cli/internal/otlp/attrs_test.gonew file mode 100644index 00000000..c8851d85--- /dev/null+++ b/cli/internal/otlp/attrs_test.go@@ -0,0 +1,31 @@+package otlp++import "testing"++func TestInferProvider(t *testing.T) {+ tests := []struct {+ model, vendor, want string+ }{+ {"claude-opus-4-8", "cursor", "anthropic"},+ {"claude-opus-4-8-thinking-high", "cursor", "anthropic"},+ {"gpt-5.5", "codex", "openai"},+ {"gpt-5.6-sol-medium", "cursor", "openai"},+ {"cursor-grok-4.5-high", "cursor", "xai"},+ {"grok-4.5", "cursor", "xai"},+ {"composer-2.5", "cursor", "cursor"},+ {"composer-2-5", "cursor", "cursor"},+ {"auto", "cursor", "cursor"},+ {"gemini-3-pro", "", "google"},+ {"kimi-k2.7-code", "cursor", "moonshot"},+ {"", "cursor", "cursor"},+ {"", "codex", "openai"},+ {"", "claude-code", "anthropic"},+ {"totally-unknown", "", ""},+ }+ for _, tt := range tests {+ got := inferProvider(tt.model, tt.vendor)+ if got != tt.want {+ t.Errorf("inferProvider(%q, %q) = %q, want %q", tt.model, tt.vendor, got, tt.want)+ }+ }+}diff --git a/docs/docs.json b/docs/docs.jsonindex 423bf839..ee1931ef 100644--- a/docs/docs.json+++ b/docs/docs.json@@ -9,6 +9,23 @@"dark": "#FFA500"},"favicon": "favicon.png",+ "seo": {+ "indexing": "navigable",+ "metatags": {+ "og:image": "/images/docs-main-banner.jpeg"+ },+ "organization": {+ "name": "OpenLIT",+ "url": "https://openlit.io",+ "logo": "https://docs.openlit.io/images/rectangular-light-bg-logo.jpg",+ "sameAs": [+ "https://github.com/openlit/",+ "https://twitter.com/openlit_io",+ "https://www.linkedin.com/company/openlit/",+ "https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ"+ ]+ }+ },"feedback": {"thumbsRating": true,"suggestEdit": true@@ -48,62 +65,147 @@"pages": ["latest/openlit/quickstart-ai-observability","latest/openlit/quickstart-evals",- "latest/openlit/quickstart-guard",- "latest/openlit/quickstart-gpu",- "latest/openlit/quickstart-vectordb-observability",- "latest/openlit/quickstart-mcp-observability"+ "latest/openlit/prompts-experiments/prompt-hub/overview"]},- "latest/openlit/configuration",- "latest/openlit/oauth",- {- "group": "Organisation",- "icon": "building",- "pages": [- "latest/openlit/organisation/overview",- "latest/openlit/organisation/projects",- "latest/openlit/organisation/database-config"- ]- }+ "latest/openlit/configuration"]},{- "group": "Observability",+ "group": "Monitor","pages": [- "latest/openlit/observability/tracing",- "latest/openlit/observability/metrics",- "latest/openlit/observability/error",- "latest/openlit/observability/fleet-hub",+ {+ "group": "Telemetry",+ "icon": "wave-pulse",+ "pages": [+ "latest/openlit/observability/telemetry/overview",+ "latest/openlit/observability/telemetry/traces",+ "latest/openlit/observability/telemetry/ai-analysis",+ "latest/openlit/observability/telemetry/metrics",+ "latest/openlit/observability/telemetry/logs"+ ]+ },{"group": "Agents",- "icon": "radar",+ "icon": "robot","pages": [- "latest/controller/overview",- "latest/controller/quickstart",- "latest/controller/architecture",- "latest/controller/configuration"+ "latest/openlit/observability/agents/overview",+ "latest/openlit/observability/agents/setup",+ "latest/openlit/observability/agents/analytics",+ "latest/openlit/observability/agents/monitoring",+ "latest/openlit/observability/agents/definition",+ "latest/openlit/observability/agents/configuration",+ "latest/openlit/observability/agents/versioning"]},{"group": "Coding Agents","icon": "code","pages": [- "latest/openlit/coding-agents/onboarding",- "latest/openlit/coding-agents/architecture"+ "latest/openlit/coding-agents/overview",+ "latest/openlit/coding-agents/setup-and-configure",+ "latest/openlit/coding-agents/analytics",+ "latest/openlit/coding-agents/sessions",+ "latest/openlit/coding-agents/users",+ "latest/openlit/coding-agents/privacy-and-governance",+ "latest/openlit/coding-agents/troubleshooting"+ ]+ },+ {+ "group": "Evaluations",+ "icon": "display",+ "pages": [+ "latest/openlit/evaluations/overview",+ "latest/openlit/evaluations/analytics",+ "latest/openlit/evaluations/evaluators",+ "latest/openlit/evaluations/configuration",+ {+ "group": "Evaluation Methods",+ "icon": "ruler",+ "pages": [+ "latest/openlit/evaluations/llm-as-a-judge",+ "latest/openlit/evaluations/programmatic-evals",+ "latest/openlit/evaluations/manual-feedback"+ ]+ }+ ]+ },+ {+ "group": "Costs",+ "icon": "circle-dollar",+ "pages": [+ "latest/openlit/costs/overview",+ "latest/openlit/costs/analytics",+ {+ "group": "Manage models",+ "icon": "sliders",+ "pages": [+ "latest/openlit/costs/manage-models/overview",+ "latest/openlit/costs/manage-models/providers",+ "latest/openlit/costs/manage-models/models",+ "latest/openlit/costs/manage-models/import-and-export",+ "latest/openlit/costs/manage-models/sdk-pricing"+ ]+ },+ "latest/openlit/costs/configuration"]}]},{- "group": "Evaluations",+ "group": "Develop","pages": [- "latest/openlit/evaluations/overview",{- "group": "Evaluation Methods",- "icon": "ruler",+ "group": "Prompt Hub",+ "icon": "shapes",+ "pages": [+ "latest/openlit/prompts-experiments/prompt-hub/overview",+ "latest/openlit/prompts-experiments/prompt-hub/manage-prompts",+ "latest/openlit/prompts-experiments/prompt-hub/versioning",+ "latest/openlit/prompts-experiments/prompt-hub/ai-improvement",+ "latest/openlit/prompts-experiments/prompt-hub/rules",+ "latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt"+ ]+ },+ {+ "group": "Vault",+ "icon": "book",+ "pages": [+ "latest/openlit/developer-resources/vault/overview",+ "latest/openlit/developer-resources/vault/manage-secrets",+ "latest/openlit/developer-resources/vault/security",+ "latest/openlit/developer-resources/vault/retrieve-secrets"+ ]+ },+ {+ "group": "Context",+ "icon": "book-open",+ "pages": [+ "latest/openlit/prompts-experiments/context/overview",+ "latest/openlit/prompts-experiments/context/manage-contexts",+ "latest/openlit/prompts-experiments/context/link-rules",+ "latest/openlit/prompts-experiments/context/retrieve-context"+ ]+ },+ {+ "group": "Rule Engine",+ "icon": "sliders",+ "pages": [+ "latest/openlit/prompts-experiments/rule-engine/overview",+ "latest/openlit/prompts-experiments/rule-engine/create-and-manage-rules",+ "latest/openlit/prompts-experiments/rule-engine/conditions",+ "latest/openlit/prompts-experiments/rule-engine/preview",+ "latest/openlit/prompts-experiments/rule-engine/linked-entities",+ "latest/openlit/prompts-experiments/rule-engine/evaluate-api"+ ]+ },+ {+ "group": "OpenGround",+ "icon": "display","pages": [- "latest/openlit/evaluations/llm-as-a-judge",- "latest/openlit/evaluations/programmatic-evals"+ "latest/openlit/prompts-experiments/openground/overview",+ "latest/openlit/prompts-experiments/openground/run-a-comparison",+ "latest/openlit/prompts-experiments/openground/results-and-history"]}]@@ -127,7 +229,7 @@"pages": ["latest/openlit/dashboards/widgets/overview","latest/openlit/dashboards/widgets/area-chart-widget",- "latest/openlit/dashboards/widgets/bar-chart-widget",+ "latest/openlit/dashboards/widgets/bar-chart-widget","latest/openlit/dashboards/widgets/line-chart-widget","latest/openlit/dashboards/widgets/pie-chart-widget","latest/openlit/dashboards/widgets/stat-widget",@@ -152,29 +254,31 @@{"group": "Otter (AI Copilot)","pages": [- "latest/openlit/chat"- ]- },- {- "group": "Pricing & Models",- "pages": [- "latest/openlit/pricing/manage-models",- "latest/openlit/pricing/cost-recalculation"+ "latest/openlit/chat/overview",+ "latest/openlit/chat/conversations",+ "latest/openlit/chat/settings",+ "latest/openlit/chat/usage"]},{- "group": "Prompts and Experiments",+ "group": "Platform","pages": [- "latest/openlit/prompts-experiments/prompt-hub",- "latest/openlit/prompts-experiments/openground",- "latest/openlit/prompts-experiments/context",- "latest/openlit/prompts-experiments/rule-engine"- ]- },- {- "group": "Developer Resources",- "pages": [- "latest/openlit/developer-resources/vault",+ {+ "group": "Administration",+ "icon": "user-shield",+ "pages": [+ "latest/openlit/oauth",+ {+ "group": "Organisation",+ "icon": "building",+ "pages": [+ "latest/openlit/organisation/overview",+ "latest/openlit/organisation/projects",+ "latest/openlit/organisation/database-config"+ ]+ }+ ]+ },"latest/openlit/developer-resources/multiple-db","latest/openlit/developer-resources/anonymous-telemetry",{@@ -199,10 +303,28 @@"pages": ["latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate"]+ },+ {+ "group": "Manage Models",+ "pages": [+ "latest/openlit/developer-resources/api-reference/endpoint/manage-models/models",+ "latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers",+ "latest/openlit/developer-resources/api-reference/endpoint/manage-models/export",+ "latest/openlit/developer-resources/api-reference/endpoint/manage-models/import",+ "latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing"+ ]}]}]+ },+ {+ "group": "Resources",+ "pages": [+ "latest/glossary/what-is-ai-observability",+ "latest/glossary/what-is-ai-evaluation",+ "latest/glossary/what-is-prompt-engineering"+ ]}]},@@ -239,6 +361,7 @@"latest/sdk/features/pricing","latest/sdk/features/evaluations","latest/sdk/features/guardrails",+ "latest/sdk/features/custom-attributes","latest/sdk/features/rule-engine"]},@@ -251,10 +374,8 @@"icon": "robot","pages": ["latest/sdk/integrations/openai",- "latest/sdk/integrations/go-openai","latest/sdk/integrations/ollama","latest/sdk/integrations/anthropic",- "latest/sdk/integrations/go-anthropic","latest/sdk/integrations/deepseek","latest/sdk/integrations/gpt4all","latest/sdk/integrations/cohere",@@ -326,7 +447,10 @@"latest/sdk/integrations/strands","latest/sdk/integrations/google-adk","latest/sdk/integrations/claude-agent-sdk",- "latest/sdk/integrations/vercel-ai"+ "latest/sdk/integrations/vercel-ai",+ "latest/sdk/integrations/smolagents",+ "latest/sdk/integrations/agent-framework",+ "latest/sdk/integrations/mcp"]},{@@ -410,9 +534,170 @@]}]+ },+ {+ "tab": "CLI",+ "groups": [+ {+ "group": "Getting Started",+ "pages": [+ "latest/cli/overview",+ "latest/cli/installation",+ "latest/cli/configuration"+ ]+ },+ {+ "group": "Reference",+ "pages": [+ "latest/cli/commands"+ ]+ }+ ]}]},+ "redirects": [+ {+ "source": "/latest/openlit/prompts-experiments/prompt-hub",+ "destination": "/latest/openlit/prompts-experiments/prompt-hub/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/developer-resources/vault",+ "destination": "/latest/openlit/developer-resources/vault/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/prompts-experiments/context",+ "destination": "/latest/openlit/prompts-experiments/context/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/prompts-experiments/rule-engine",+ "destination": "/latest/openlit/prompts-experiments/rule-engine/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/prompts-experiments/openground",+ "destination": "/latest/openlit/prompts-experiments/openground/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/tracing",+ "destination": "/latest/openlit/observability/telemetry/traces",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/metrics",+ "destination": "/latest/openlit/observability/telemetry/metrics",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/error",+ "destination": "/latest/openlit/observability/telemetry/traces",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/telemetry",+ "destination": "/latest/openlit/observability/telemetry/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/agents",+ "destination": "/latest/openlit/observability/agents/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/coding-agents/onboarding",+ "destination": "/latest/openlit/coding-agents/setup-and-configure",+ "permanent": true+ },+ {+ "source": "/latest/openlit/coding-agents/architecture",+ "destination": "/latest/openlit/coding-agents/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/agents/agent-detail",+ "destination": "/latest/openlit/observability/agents/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/agents/definition-and-versions",+ "destination": "/latest/openlit/observability/agents/versioning",+ "permanent": true+ },+ {+ "source": "/latest/openlit/observability/agents/coding-agents",+ "destination": "/latest/openlit/coding-agents/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/costs/manage-models",+ "destination": "/latest/openlit/costs/manage-models/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/chat",+ "destination": "/latest/openlit/chat/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/cost-recalculation",+ "destination": "/latest/openlit/costs/configuration",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/manage-models",+ "destination": "/latest/openlit/costs/manage-models/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/manage-models/overview",+ "destination": "/latest/openlit/costs/manage-models/overview",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/manage-models/providers",+ "destination": "/latest/openlit/costs/manage-models/providers",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/manage-models/models",+ "destination": "/latest/openlit/costs/manage-models/models",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/manage-models/import-and-export",+ "destination": "/latest/openlit/costs/manage-models/import-and-export",+ "permanent": true+ },+ {+ "source": "/latest/openlit/pricing/manage-models/sdk-pricing",+ "destination": "/latest/openlit/costs/manage-models/sdk-pricing",+ "permanent": true+ },+ {+ "source": "/latest/openlit/evaluations/core-concepts",+ "destination": "/latest/openlit/evaluations/evaluators",+ "permanent": true+ },+ {+ "source": "/latest/openlit/evaluations/auto-evaluation",+ "destination": "/latest/openlit/evaluations/configuration",+ "permanent": true+ },+ {+ "source": "/latest/openlit/evaluations/settings",+ "destination": "/latest/openlit/evaluations/configuration",+ "permanent": true+ },+ {+ "source": "/latest/openlit/evaluations/types",+ "destination": "/latest/openlit/evaluations/evaluators",+ "permanent": true+ }+ ],"logo": {"light": "/images/rectangular-light-bg-logo.jpg","dark": "/images/rectangular-dark-bg-logo.jpg",@@ -448,6 +733,10 @@}},"contextual": {- "options": ["copy", "chatgpt", "claude"]+ "options": [+ "copy",+ "chatgpt",+ "claude"+ ]}}diff --git a/docs/latest/cli/commands.mdx b/docs/latest/cli/commands.mdxnew file mode 100644index 00000000..59171148--- /dev/null+++ b/docs/latest/cli/commands.mdx@@ -0,0 +1,138 @@+---+title: "Command Reference"+sidebarTitle: "Command Reference"+description: "Every openlit CLI command, subcommand, and flag"+---++The root command is `openlit`. Run `openlit --help` or `openlit <command> --help` for inline usage at any time.++```+openlit+├── coding Coding-agent observability (Claude Code, Cursor, Codex)+│ ├── install Install per-vendor host plugin manifests+│ ├── launch Bootstrap the plugin for an agent and exec it+│ ├── uninstall Remove per-vendor host plugin manifests+│ └── hook Process a hook event (invoked by host plugins)+├── configure Write persistent settings to config.env+├── doctor Diagnose config, OTLP reachability, and plugins+└── version Print the CLI version+```++## openlit coding++Coding-agent observability for Claude Code, Cursor, and Codex. See the [Coding Agents setup guide](/latest/openlit/coding-agents/setup-and-configure) for the full vendor-by-vendor walkthrough.++### openlit coding install++Install the per-vendor host plugin manifest. Idempotent - re-running drops prior `openlit` entries before re-writing, leaving any third-party entries in place.++```bash+openlit coding install --vendor=claude-code+openlit coding install --vendor=all --dry-run # preview without writing+```++| Flag | Description |+|---|---|+| `--vendor` *(required)* | `claude-code` \| `cursor` \| `codex` \| `all` |+| `--dry-run` | Print what would be written without modifying any files |++**Where manifests are written:**++| Vendor | Manifest path |+|---|---|+| Claude Code | `~/.claude/plugins/openlit-cc/` |+| Cursor | `~/.cursor/hooks.json` (user scope, merged) |+| Codex | `~/.codex/plugins/openlit/` |++### openlit coding launch++Bootstrap the plugin for an agent and exec it - telemetry on from the first turn. Auto-installs the manifest if needed.++```bash+openlit coding launch claude+openlit coding launch cursor -- <agent-args...> # pass args through after --+openlit coding launch codex+```++| Argument | Description |+|---|---|+| `<agent>` *(required)* | `claude` \| `claude-code` \| `cc` \| `cursor` \| `codex` (`claude-code` and `cc` are aliases for `claude`) |++Any arguments after `--` are forwarded to the agent's own CLI.++### openlit coding uninstall++Remove the per-vendor plugin manifests written by `install`. Best-effort deregisters the plugin from the vendor's own CLI where applicable. Leaves your shared config alone unless `--purge` is passed.++```bash+openlit coding uninstall --vendor=cursor # one vendor+openlit coding uninstall --vendor=all # every vendor+openlit coding uninstall --vendor=all --purge # also drop ~/.config/openlit + session cache+openlit coding uninstall --vendor=cursor --dry-run # preview without touching disk+```++| Flag | Description |+|---|---|+| `--vendor` *(required)* | `claude-code` \| `cursor` \| `codex` \| `all` |+| `--purge` | Also remove `~/.config/openlit` and the session-state cache |+| `--dry-run` | Print what would be removed without modifying any files |++### openlit coding hook++<Note>+You will almost never run this yourself. The host plugin manifests written by `openlit coding install` invoke it once per agent event, piping the event payload on stdin.+</Note>++```bash+openlit coding hook --vendor=claude-code --event=SessionStart < payload.json+```++| Flag | Description |+|---|---|+| `--vendor` *(required)* | `cc` \| `claude-code` \| `cursor` \| `codex` |+| `--event` | Hook event name (vendor-specific; e.g. `SessionStart`, `PreToolUse`) |++The hook is crash-isolated: a 5-second hard timeout (~3s reserved for OTLP flush) and it **always exits 0** even when telemetry fails - a broken pipeline never blocks your coding session. It never writes to stdout (agents like Claude Code parse stdout for JSON). Set `OPENLIT_DEBUG_PAYLOAD_DIR` to tee raw payloads for debugging.++## openlit configure++Write persistent settings to `~/.config/openlit/config.env`. See [Configuration](/latest/cli/configuration) for the full flag table, precedence, and content-capture modes.++```bash+openlit configure --endpoint https://openlit.example.com:4318 --api-key $OPENLIT_API_KEY+openlit configure --show # print resolved config and its source, write nothing+```++## openlit doctor++Diagnose the CLI's setup. Reads the resolved config, TCP-dials the OTLP endpoint (2s timeout), and lists installed vendor plugins.++```bash+openlit doctor+```++Exit code `0` means the CLI is wired up (possibly with warnings); `1` means at least one check failed. This is the first thing to run when something isn't showing up on `/agents`.++## openlit version++Print the CLI version and build commit.++```bash+openlit version+# openlit v1.2.3 (abc1234)+```++## Reserved (future) commands++The CLI is built to grow into OpenLIT's full control plane. These command groups are reserved in the codebase and will land in future releases:++| Command | Planned subcommands |+|---|---|+| `openlit prompts` | `pull`, `push`, `list`, `diff` |+| `openlit traces` | `tail`, `query`, `export` |+| `openlit eval` | `run`, `list` |+| `openlit migrate` | - |++<Info>+These are not available yet. They are documented here only to signal the CLI's direction - new groups plug in by registering on the root command without breaking the coding-agent integration.+</Info>diff --git a/docs/latest/cli/configuration.mdx b/docs/latest/cli/configuration.mdxnew file mode 100644index 00000000..81baa89f--- /dev/null+++ b/docs/latest/cli/configuration.mdx@@ -0,0 +1,100 @@+---+title: "CLI Configuration"+sidebarTitle: "Configuration"+description: "Configure the openlit CLI - endpoint, API key, content capture, and precedence rules"+---++The CLI reads its settings from flags, environment variables, and a config file. The fastest way to set it up is `openlit configure`.++## Configure once++```bash+openlit configure \+ --endpoint https://openlit.example.com:4318 \+ --api-key $OPENLIT_API_KEY+```++This writes a `KEY=VALUE` file to:++- **Linux / macOS** - `~/.config/openlit/config.env` (honors `$XDG_CONFIG_HOME`)+- **Windows** - `%APPDATA%\openlit\config.env`++The file is written with mode `0600` (owner-readable only).++To inspect the effective configuration without writing anything - including which source each value came from - use `--show`:++```bash+openlit configure --show+```++## `configure` flags++| Flag | Maps to | Description |+|---|---|---|+| `--endpoint` | `OPENLIT_OTLP_ENDPOINT` | OTLP/HTTP endpoint URL (e.g. `http://localhost:4318`) |+| `--api-key` | `OPENLIT_API_KEY` | API key, sent as `Authorization: Bearer <key>` |+| `--environment` | `OPENLIT_ENVIRONMENT` | Deployment environment (e.g. `production`, `staging`) |+| `--application-name` | `OPENLIT_APPLICATION_NAME` | Application name (default `openlit-cli`) |+| `--content-capture` | `OPENLIT_CODING_CONTENT_CAPTURE` | Coding-agent content capture: `minimal` \| `metadata_only` \| `full` |+| `--show` | - | Print resolved config (and its source) without writing |++## Precedence++Settings resolve highest-to-lowest:++1. **CLI flags** (`--endpoint`, `--api-key`, …)+2. **`OPENLIT_*` environment variables**+3. **`OTEL_EXPORTER_OTLP_*` environment variables** (standard OpenTelemetry fallback)+4. **Config file** (`~/.config/openlit/config.env`, allow-listed keys only)+5. **Built-in defaults**++For headless / fleet rollouts, prefer environment variables or flags - they take precedence over the file. Existing OpenTelemetry users can reuse their standard `OTEL_EXPORTER_OTLP_*` setup.++## Environment variables++| Variable | Purpose | Default |+|---|---|---|+| `OPENLIT_OTLP_ENDPOINT` | OTLP/HTTP base URL | `http://127.0.0.1:4318` |+| `OPENLIT_API_KEY` | Authenticates the OTLP exporter (`Authorization: Bearer <key>`) | - |+| `OPENLIT_ENVIRONMENT` | Deployment environment label | `default` |+| `OPENLIT_APPLICATION_NAME` | App name for OTel resource attributes | `openlit-cli` |+| `OPENLIT_CODING_CONTENT_CAPTURE` | Coding-agent content capture mode | `full` |+| `OPENLIT_CODING_REPO_ALLOWLIST` | Comma/newline-separated repo patterns for personal-vs-work classification | - |+| `OPENLIT_USER` | Override the resolved user identity (`gen_ai.user.name`) | auto |+| `OTEL_EXPORTER_OTLP_ENDPOINT` | Standard OTel endpoint (fallback) | - |+| `OTEL_EXPORTER_OTLP_HEADERS` | Extra headers as comma-separated `k=v` pairs | - |+| `OTEL_RESOURCE_ATTRIBUTES` | Standard OTel resource attributes | - |+| `OPENLIT_DEBUG_PAYLOAD_DIR` | Tee raw hook payloads to JSONL for debugging | - |++<Warning>+The `OPENLIT_API_KEY` value is never promoted into a launched agent's child-process environment, and the config file is written `0600`. The CLI never reads your shell rc files (`~/.zshrc`, `~/.bashrc`) - if you set `OPENLIT_*` vars in a shell rc, source them before launching the agent.+</Warning>++## Content capture modes++`OPENLIT_CODING_CONTENT_CAPTURE` controls how much of each coding-agent interaction is captured. Tier-1 secret redaction runs on every export regardless of mode.++| Mode | What's captured |+|---|---|+| `minimal` | Session bookends and counters only - good for cost + activity dashboards |+| `metadata_only` | Per-event spans **without** prompt bodies or tool-arg bodies |+| `full` *(default)* | Complete prompt + response bodies and tool I/O, secret-scrubbed on the way out |++<Tip>+`full` is the default so the trace-detail view is useful on first use. Switch to `metadata_only` when rolling out across a team where prompts may carry confidential material, or `minimal` if you only need cost and activity rollups.+</Tip>++## Endpoint normalization++The CLI accepts scheme-less, `http://`, or `https://` endpoints, with or without a path/trailing slash. For non-loopback hosts it auto-upgrades plain `http://` to `https://`. Loopback and private-range hosts (`localhost`, `127.0.0.1`, `::1`, `10.*`, `192.168.*`, `172.16–31.*`, `169.254.*`) are left on `http://`.++## Next steps++<CardGroup cols={2}>+ <Card title="Command reference" href="/latest/cli/commands" icon="terminal">+ Every command, subcommand, and flag+ </Card>+ <Card title="Wire your coding agents" href="/latest/openlit/coding-agents/setup-and-configure" icon="code">+ The end-to-end setup guide+ </Card>+</CardGroup>diff --git a/docs/latest/cli/installation.mdx b/docs/latest/cli/installation.mdxnew file mode 100644index 00000000..39a1d8e0--- /dev/null+++ b/docs/latest/cli/installation.mdx@@ -0,0 +1,104 @@+---+title: "Install the CLI"+sidebarTitle: "Installation"+description: "Install the openlit CLI on macOS, Linux, Windows, Docker, or from source"+---++All install paths produce the same `openlit` binary. Pick the one that matches your OS.++<CodeGroup>+```bash macOS+brew install openlit/openlit/openlit+```++```bash Linux+curl -fsSL https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.sh | sh+```++```powershell Windows+iwr -useb https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.ps1 | iex+```++```bash Docker+docker run --rm -it ghcr.io/openlit/openlit-cli:latest --help+```++```bash From source+go install github.com/openlit/openlit/cli/cmd/openlit@latest+```+</CodeGroup>++## Install locations & PATH++| Method | Install location | PATH handling |+|---|---|---|+| **Homebrew** | Managed by Homebrew | Automatic |+| **Linux script** | `~/.openlit/bin/openlit` | Prints a PATH-add hint if the directory isn't on `$PATH` |+| **Windows script** | `%USERPROFILE%\.openlit\bin\openlit.exe` | Updates user-scope PATH automatically - open a new terminal to pick it up |+| **`go install`** | `$(go env GOPATH)/bin/openlit` | Add `GOPATH/bin` to `$PATH` if not already |++<Tip>+The CLI binary **must be on `$PATH`** for the vendor plugin installs to work. The `openlit coding install` command rewrites the `openlit` token in each generated hook manifest to the absolute path of the running binary, which sidesteps the minimal PATH that GUI-launched agents (like Cursor) inherit - but the binary still has to be found at install time.+</Tip>++### Linux/macOS script options++The `install.sh` script downloads the matching tarball from the latest `cli-*.*.*` GitHub Release of `openlit/openlit`, verifies its `.sha256` sidecar (best-effort), and installs to `~/.openlit/bin/openlit`. It detects your OS (macOS, Linux) and architecture (amd64, arm64).++| Environment variable | Purpose | Default |+|---|---|---|+| `OPENLIT_INSTALL_DIR` | Target install directory | `$HOME/.openlit/bin` |+| `OPENLIT_VERSION` | Release tag **without** the `cli-` prefix (e.g. `1.2.3`) | latest release |++```bash+# Pin a specific version into a custom directory+OPENLIT_INSTALL_DIR=/usr/local/bin OPENLIT_VERSION=1.2.3 \+ sh -c "$(curl -fsSL https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.sh)"+```++## Build from source++If you have the repository checked out and Go installed, build a local binary:++```bash+sh cli/scripts/install-local.sh+```++This builds with `go build -trimpath -ldflags "-s -w"` from `cli/cmd/openlit` and installs to `~/.openlit/bin/openlit`.++## Verify the install++```bash+openlit version+# openlit v1.2.3 (abc1234)++openlit doctor+# Prints resolved config, dials your OTLP endpoint, lists installed plugins+```++## Uninstall++The `openlit` binary is never touched by `openlit coding uninstall` (which only removes vendor plugin manifests). Remove the binary the way you installed it:++```bash+brew uninstall openlit # Homebrew+rm ~/.openlit/bin/openlit # curl|sh installer (Linux/macOS)+rm $(go env GOPATH)/bin/openlit # go install+```++On Windows, run the uninstall script:++```powershell+iwr -useb https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/uninstall.ps1 | iex+```++## Next steps++<CardGroup cols={2}>+ <Card title="Configuration" href="/latest/cli/configuration" icon="sliders">+ Point the CLI at your OpenLIT collector+ </Card>+ <Card title="Wire your coding agents" href="/latest/openlit/coding-agents/setup-and-configure" icon="code">+ Claude Code, Cursor, and Codex in under a minute+ </Card>+</CardGroup>diff --git a/docs/latest/cli/overview.mdx b/docs/latest/cli/overview.mdxnew file mode 100644index 00000000..fbc4b696--- /dev/null+++ b/docs/latest/cli/overview.mdx@@ -0,0 +1,74 @@+---+title: "OpenLIT CLI"+sidebarTitle: "Overview"+description: "The command-line interface for OpenLIT - observe coding agents today, control the whole platform tomorrow"+---++The **`openlit` CLI** is a single Go binary that connects your local tooling to an OpenLIT instance. Today it ships first-class observability for AI **coding agents** - Claude Code, Cursor, and Codex - by wiring into each vendor's hook system, normalizing events onto OpenTelemetry's `gen_ai.*` and OpenLIT's `coding_agent.*` semantic conventions, and exporting them via OTLP to your collector.++It talks **only** to your OpenLIT endpoint. There is no managed cloud - your telemetry stays on your infrastructure.++<Note>+The CLI is the future control plane for OpenLIT. It's built so new command groups can plug in over time - `prompts`, `traces`, `eval`, and `migrate` are already reserved in the codebase and will light up in subsequent releases without breaking the coding-agent integration. Today, the CLI is focused on **coding-agent observability**.+</Note>++## What you can do today++<CardGroup cols={2}>+ <Card title="Observe coding agents" icon="code">+ Wire Claude Code, Cursor, or Codex into OpenLIT with one idempotent command per vendor - no SDK, no code changes in your repos.+ </Card>+ <Card title="Configure once" icon="gear">+ Set your OTLP endpoint, API key, environment, and content-capture mode in a single config file or via environment variables.+ </Card>+ <Card title="Launch agents instrumented" icon="rocket">+ `openlit coding launch` bootstraps the plugin and execs your agent in one step - telemetry on from the first turn.+ </Card>+ <Card title="Diagnose your setup" icon="stethoscope">+ `openlit doctor` prints the resolved config, dials your OTLP endpoint, and lists installed plugins so you can confirm everything is wired up.+ </Card>+</CardGroup>++## Command groups++| Command | Purpose |+|---|---|+| [`openlit coding`](/latest/cli/commands#openlit-coding) | Install, launch, and manage coding-agent telemetry (Claude Code, Cursor, Codex) |+| [`openlit configure`](/latest/cli/commands#openlit-configure) | Write persistent settings to `~/.config/openlit/config.env` |+| [`openlit doctor`](/latest/cli/commands#openlit-doctor) | Diagnose config, OTLP reachability, and installed plugins |+| [`openlit version`](/latest/cli/commands#openlit-version) | Print the CLI version and build commit |++## How it works++```+Coding agent (Claude Code / Cursor / Codex)+ +-- fires hook events (SessionStart, PreToolUse, Stop, ...)+ |+ v+openlit coding hook --vendor=<v> --event=<name> (invoked by the host plugin manifest)+ +-- normalizes to gen_ai.* / coding_agent.* semantic conventions+ +-- tier-1 secret redaction+ +-- 5s hard timeout, always exits 0 (never blocks the developer)+ |+ v+OTLP/HTTP --> your OpenLIT collector --> /agents page+```++The hook is the CLI's hot path - invoked once per agent event. It is crash-isolated: a 5-second hard timeout with ~3 seconds reserved for OTLP flush, and it **always exits 0** even when telemetry fails, so a broken pipeline never blocks your coding session.++## Get started++<CardGroup cols={2}>+ <Card title="Install the CLI" href="/latest/cli/installation" icon="circle-down">+ macOS, Linux, Windows, Docker, or build from source+ </Card>+ <Card title="Configuration" href="/latest/cli/configuration" icon="sliders">+ Endpoint, API key, content capture, and precedence rules+ </Card>+ <Card title="Command reference" href="/latest/cli/commands" icon="terminal">+ Every command, subcommand, and flag+ </Card>+ <Card title="Coding agents setup" href="/latest/openlit/coding-agents/setup-and-configure" icon="code">+ The end-to-end guide to wiring each vendor+ </Card>+</CardGroup>diff --git a/docs/latest/controller/architecture.mdx b/docs/latest/controller/architecture.mdxdeleted file mode 100644index 5cc82639..00000000--- a/docs/latest/controller/architecture.mdx+++ /dev/null@@ -1,130 +0,0 @@-----title: "Controller Architecture"-sidebarTitle: "Architecture"-description: "How the OpenLIT Controller discovers, instruments, and reconciles AI services"------## High-Level Architecture--```mermaid-flowchart TB- subgraph "Controller (per node)"- Scanner["eBPF Scanner\n(proc + socket inspection)"]- OBI["OBI Engine\n(eBPF instrumentation)"]- SDKInj["SDK Injector\n(Python SDK)"]- Poller["Poll Loop\n(reports to dashboard)"]- end-- subgraph "OpenLIT Dashboard"- Hub["Agents"]- Actions["Action Queue"]- Desired["Desired State Store"]- CH[(ClickHouse)]- end-- Scanner -->|Discovered services| Poller- Poller -->|POST /api/controller/poll| Hub- Hub -->|Pending actions| Poller- Poller -->|instrument| OBI- Poller -->|enable_python_sdk| SDKInj- Hub -->|Read/write| CH- Hub -->|User intent| Desired- Desired -->|Reconcile| Actions- OBI -->|OTLP traces| CH- SDKInj -->|OTLP traces| CH-```--## Core Components--### Service Discovery--The Controller continuously scans for processes that communicate with known LLM API endpoints. It inspects:--- **Network connections** — Identifies TCP connections to known LLM provider domains (OpenAI, Anthropic, Google AI, AWS Bedrock, etc.)-- **Process metadata** — Extracts PID, executable path, command line, and language runtime-- **Kubernetes metadata** — Resolves pod name, namespace, deployment/daemonset/statefulset ownership-- **Docker metadata** — Reads container name, ID, and labels-- **Workload key** — Generates a stable identifier (e.g., `k8s:namespace:deployment:service`) that persists across pod restarts--### LLM Observability (OBI Engine)--OBI (OpenLIT Binary Instrumentation) uses **eBPF** to intercept LLM API traffic at the kernel level:--1. The Controller tells OBI which processes to instrument-2. OBI attaches eBPF probes to TLS read/write syscalls-3. Captured data is parsed to extract model name, token counts, latency, and errors-4. Traces are sent to the OTLP endpoint in OpenTelemetry format--**Key properties:**-- Zero application overhead (kernel-level interception)-- Works with any language runtime (Python, Node.js, Go, Java, etc.)-- No code changes, no SDK, no sidecar proxies-- Captures traffic that SDKs cannot (e.g., internal library calls)--### Agent Observability (SDK Injector)--For deeper agent framework visibility, the Controller injects the **OpenLIT Python SDK** into running applications:--| Mode | Injection mechanism |-|---|---|-| **Kubernetes** | Patches Deployment/DaemonSet/StatefulSet spec with init container + env vars, triggers rolling update |-| **Docker** | Recreates the container with SDK volume mount + env vars |-| **Linux** | Creates a systemd drop-in that sets `PYTHONPATH` and env vars, then restarts the service |--The injected SDK auto-instruments agent frameworks (LangChain, CrewAI, LangGraph, OpenAI Agents, etc.) and sends spans to the OTLP endpoint.--## Poll-Based Control Loop--The Controller communicates with the OpenLIT dashboard via a **poll-based** control loop:--```mermaid-sequenceDiagram- participant C as Controller- participant D as Dashboard- participant CH as ClickHouse-- loop Every poll_interval seconds- C->>D: POST /api/controller/poll- Note over C,D: Reports discovered services,<br/>instrumentation status,<br/>action results- D->>CH: Upsert services- D->>CH: Read desired states- D->>CH: Queue reconciliation actions- D-->>C: Response with pending actions- C->>C: Execute actions<br/>(instrument, uninstrument,<br/>enable_sdk, disable_sdk)- end-```--### Why poll-based?--- Works across NAT, firewalls, and air-gapped environments-- No inbound ports required on the controller-- Dashboard is the single source of truth for desired state-- Simple, resilient, and easy to debug--## Desired State Reconciliation--When a user enables observability for a service, the dashboard stores the **desired state** in a dedicated ClickHouse table. On every poll cycle, the Controller compares the reported state against the desired state and takes corrective action:--| Desired | Current | Action |-|---|---|---|-| `instrumented` | `discovered` | Queue `instrument` action |-| `none` | `instrumented` | Queue `uninstrument` action |-| `enabled` (agent) | `disabled` | Queue `enable_python_sdk` action |-| `none` (agent) | `enabled` | Queue `disable_python_sdk` action |--This reconciliation ensures that:-- **Pod restarts** — New pods are automatically re-instrumented-- **Container recreates** — Docker containers regain their instrumentation-- **Process restarts** — Linux services are re-instrumented via systemd-- **Controller restarts** — The Controller reads desired state from ClickHouse and re-applies it-- **Node failures** — When a node recovers, the Controller on that node reconciles all services--## Security Model--| Concern | Approach |-|---|---|-| **Privileged access** | Required for eBPF probes (`hostPID: true`, `privileged: true` in K8s) |-| **Blast radius** | Controller failure does not affect application behavior — eBPF probes are read-only, SDK injection is reversible |-| **Network** | Controller only makes outbound HTTP calls to the dashboard. No inbound ports |-| **RBAC (K8s)** | Controller needs `get/list/watch` on pods and `update/patch` on deployments/daemonsets for SDK injection |-| **Docker socket** | Required in Docker mode for container inspection and SDK injection |diff --git a/docs/latest/controller/configuration.mdx b/docs/latest/controller/configuration.mdxdeleted file mode 100644index dfaf0d98..00000000--- a/docs/latest/controller/configuration.mdx+++ /dev/null@@ -1,142 +0,0 @@-----title: "Controller Configuration"-sidebarTitle: "Configuration"-description: "Configuration options for the OpenLIT Controller"------The Controller is configured via a YAML file mounted at `/etc/openlit/config.yaml` or through environment variables.--## Configuration File--```yaml /etc/openlit/config.yaml-openlit_url: "http://openlit:3000"-otlp_endpoint: "http://openlit:4318"-poll_interval: 60s-environment: "production"-api_listen: ":4321"-```--## Configuration Reference--| Field | Env Variable | Default | Description |-|---|---|---|---|-| `openlit_url` | `OPENLIT_URL` | (required) | URL of the OpenLIT dashboard for the poll API |-| `otlp_endpoint` | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTLP HTTP endpoint for sending traces and metrics |-| `poll_interval` | `OPENLIT_POLL_INTERVAL` | `60s` | Duration between poll cycles (e.g., `60s`, `2m`) |-| `environment` | `OPENLIT_ENVIRONMENT` | `default` | Deployment environment label (e.g., `production`, `staging`) |-| `api_listen` | `OPENLIT_API_LISTEN` | `:4321` | Address for the Controller's REST API |-| `cluster_id` | `OPENLIT_CLUSTER_ID` | `default` | Identifier for multi-cluster setups |-| `api_key` | `OPENLIT_API_KEY` | — | API key for authenticating with the dashboard |-| `sdk_version` | `OPENLIT_SDK_VERSION` | (auto) | Pin a specific OpenLIT Python SDK version |-| — | `OPENLIT_INSTANCE_ID` | (auto) | Override the controller's identity. By default, the instance ID is derived automatically (hostname on Linux, host hostname on Docker, `NODE_NAME` on K8s). Set this to force a stable, custom identity — useful when the automatic ID is not stable across restarts |--Environment variables take precedence over config file values.--## Deployment Mode--The Controller auto-detects its deployment mode:--- **Kubernetes** — Detected when the `KUBERNETES_SERVICE_HOST` environment variable is present-- **Docker** — Detected when `/var/run/docker.sock` is accessible-- **Linux** — Default fallback when neither Kubernetes nor Docker is detected--You can override auto-detection with the `OPENLIT_DEPLOY_MODE` environment variable:--```bash-OPENLIT_DEPLOY_MODE=docker # Force Docker mode-OPENLIT_DEPLOY_MODE=linux # Force Linux mode-```--## Kubernetes Configuration--### Required RBAC--The Controller needs the following ClusterRole to discover and modify workloads:--```yaml-rules:- - apiGroups: [""]- resources: ["pods", "pods/log"]- verbs: ["get", "list", "watch", "delete", "create"]- - apiGroups: ["apps"]- resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]- verbs: ["get", "list", "watch", "update", "patch"]-```--- `get/list/watch` on pods — for service discovery-- `delete/create` on pods — for naked pod Agent Observability (requires pod restart)-- `update/patch` on deployments/daemonsets/statefulsets — for Agent Observability SDK injection--### Volume Mounts--| Mount | Host path | Purpose |-|---|---|---|-| `/host/proc` | `/proc` (read-only) | Scans `/proc/net/tcp` to discover processes with connections to LLM APIs, reads `/proc/<pid>/cgroup` for container metadata |-| `/sys/kernel/debug` | `/sys/kernel/debug` (read-only) | Required by the kernel for attaching eBPF kprobes to `tcp_connect` |-| `/sys/fs/bpf` | `/sys/fs/bpf` (read-write) | eBPF map pinning — allows the OBI engine to persist eBPF maps across probe reloads |-| `/etc/openlit-controller` | ConfigMap | Configuration file (managed by Helm chart) |--### Security Context--The Controller requires `privileged: true` and `hostPID: true` because eBPF kprobe attachment and `/proc` scanning across all namespaces require elevated kernel access. The Controller itself is read-only (it inspects traffic, it doesn't modify it).--## Docker Configuration--### Required Volumes--```yaml-volumes:- - /proc:/host/proc:ro # /proc/net/tcp scanning for LLM connection discovery- - /sys/kernel/debug:/sys/kernel/debug:ro # eBPF kprobe attachment- - /sys/fs/bpf:/sys/fs/bpf:rw # eBPF map pinning- - /var/run/docker.sock:/var/run/docker.sock # Container discovery + SDK injection-```--- **`/proc`** — Scan process network connections to detect LLM API calls-- **`/sys/kernel/debug`** + **`/sys/fs/bpf`** — Required for eBPF operation (LLM traffic interception)-- **Docker socket** — Discover running containers, read labels/metadata, and inject the Python SDK for Agent Observability--## Linux Configuration--### Systemd Integration--For Agent Observability on Linux, the Controller manages systemd drop-in files to inject the SDK. It needs:--- Write access to `/etc/systemd/system/` for creating drop-in configurations-- Ability to run `systemctl daemon-reload` and `systemctl restart`--### Proc Filesystem--The Controller reads `/proc` to discover running processes and their network connections. Ensure `/host/proc` is mounted (or the Controller runs on the host directly with access to `/proc`).--## Controller Identity--Each controller registers itself in the dashboard with a unique `instance_id`. This ID determines whether a restarted controller appears as the same entry or a new one on the Agents page.--| Platform | Default `instance_id` | Behavior on restart |-|---|---|---|-| **Linux** | Machine hostname | Stable — hostname persists across restarts |-| **Docker** | Host hostname + container hash | Stable across `docker restart` / `stop+start`. New entry on `docker rm && run` (old entry auto-detected as stale) |-| **Kubernetes** | `NODE_NAME` (downward API) | Stable — same node, same ID across pod restarts |--To override the automatic identity on any platform, set `OPENLIT_INSTANCE_ID`:--```bash-OPENLIT_INSTANCE_ID="my-controller-1" openlit-controller-```--The dashboard automatically detects stale controllers by checking heartbeat age — controllers that haven't reported in over 10 minutes are marked as stale and dimmed in the UI.--## Multi-Cluster Setup--To run Controllers across multiple Kubernetes clusters reporting to the same OpenLIT dashboard, set a unique `cluster_id` for each cluster:--```yaml-# Cluster A-cluster_id: "us-east-prod"--# Cluster B-cluster_id: "eu-west-prod"-```--Services from different clusters will appear on the Agents page with their cluster label, and desired states are scoped per cluster.diff --git a/docs/latest/controller/overview.mdx b/docs/latest/controller/overview.mdxdeleted file mode 100644index 9a559805..00000000--- a/docs/latest/controller/overview.mdx+++ /dev/null@@ -1,172 +0,0 @@-----title: "OpenLIT Controller"-sidebarTitle: "Overview"-description: "Zero-code LLM and Agent observability for Kubernetes, Docker, and Linux"------The OpenLIT Controller is a lightweight agent that discovers and instruments AI applications without requiring any code changes, SDK integration, or container image modifications. It runs alongside your applications and uses **eBPF** for LLM Observability and **Python SDK injection** for Agent Observability.--## Why the Controller?--Traditional observability requires adding SDKs to every service — changing code, rebuilding images, and redeploying. The Controller eliminates all of that:--| | SDK approach | Controller approach |-|---|---|---|-| Code changes | Required | None |-| Image rebuild | Required | None |-| Redeploy | Required | None |-| Covers existing apps | No | Yes |-| Agent framework spans | Yes | Yes (via injected SDK) |-| LLM traffic (tokens, cost, latency) | Yes | Yes (via eBPF) |--<Tip>-The Controller and SDKs are complementary. Use the **Controller** for zero-code LLM Observability across all services, and add the **SDK** when you need deeper application-level tracing, evaluations, or guardrails.-</Tip>--## How It Works--The Controller provides two types of observability:--### LLM Observability (eBPF-based)--Uses eBPF to intercept LLM API calls at the network level — no application changes required. Captures:-- Model name, provider, and endpoint-- Token usage (prompt + completion)-- Request latency and error rates-- Cost estimation--### Agent Observability (SDK injection)--Automatically injects the OpenLIT Python SDK into running Python applications to capture:-- Agent framework spans (LangChain, CrewAI, LangGraph, etc.)-- Tool calls and chain-of-thought traces-- Vector database operations--## Deployment Modes--The Controller auto-detects its environment and adapts accordingly:--<CardGroup cols={3}>- <Card title="Kubernetes" icon="dharmachakra">- Runs as a **DaemonSet** on every node. Discovers Pods, Deployments, DaemonSets, and StatefulSets. SDK injection triggers rolling updates.- </Card>- <Card title="Docker" icon="docker" iconType="brands">- Runs as a **sidecar container** alongside your services. Discovers containers via the Docker socket. SDK injection recreates the container.- </Card>- <Card title="Linux" icon="linux" iconType="brands">- Runs as a **systemd service** or standalone process. Discovers bare-metal processes on the host. SDK injection uses systemd drop-in files.- </Card>-</CardGroup>--## Architecture--```mermaid-flowchart LR- subgraph "Your Infrastructure"- direction TB- App1[Python App]- App2[LLM Service]- App3[Agent App]- end-- subgraph "OpenLIT Controller"- direction TB- Scanner[Service Discovery]- OBI[eBPF Engine]- SDK[SDK Injector]- end-- subgraph "OpenLIT Dashboard"- direction TB- UI[Agents]- CH[(ClickHouse)]- OTEL[OTEL Collector]- end-- Scanner -->|Discovers| App1- Scanner -->|Discovers| App2- Scanner -->|Discovers| App3- OBI -->|LLM traces| OTEL- SDK -->|Agent spans| OTEL- OTEL -->|Stores| CH- UI -->|Reads| CH- UI -->|Controls| Scanner-```--1. **Service Discovery** — The Controller scans for processes making LLM API calls (OpenAI, Anthropic, Gemini, Bedrock, etc.)-2. **Agents** — Discovered services appear in the OpenLIT dashboard where you can enable/disable observability-3. **LLM Observability** — One-click eBPF instrumentation for LLM traffic metrics-4. **Agent Observability** — One-click SDK injection for agent framework traces-5. **Reconciliation** — The Controller automatically restores your desired state after pod restarts, container recreates, or process restarts--## The Agents Page--The **Agents** page is the dashboard page where you manage all Controller-discovered services. Once the Controller is running, navigate to **Agents** in the OpenLIT sidebar.--### Services view--The main view shows every service the Controller has discovered across your infrastructure:--| Column | Description |-|---|---|-| **Service** | Service name (derived from Kubernetes workload name, Docker container name, or process executable) |-| **Platform** | Kubernetes, Docker, or Linux |-| **LLM Providers** | Detected LLM API connections (OpenAI, Anthropic, Gemini, etc.) |-| **LLM Observability** | Toggle to enable/disable eBPF-based LLM tracing |-| **Agent Observability** | Toggle to enable/disable Python SDK injection (Python services only) |-| **Status** | Current instrumentation state (`Discovered`, `Instrumenting`, `Active`, `Error`) |--### Enabling observability--Click **Enable** next to any service. The Controller will:--1. **LLM Observability** — Attach eBPF probes to intercept LLM API traffic. Takes effect within seconds, no restart required.-2. **Agent Observability** — Inject the OpenLIT Python SDK. In Kubernetes this triggers a rolling update; in Docker the container is recreated; on Linux the systemd service is restarted.--### Disabling observability--Click **Disable** to remove instrumentation:--- **LLM Observability** — eBPF probes are detached immediately. No application restart.-- **Agent Observability** — The injected SDK environment variables are removed. In Kubernetes a rolling update restores the original pod spec. In Docker the container is recreated without the SDK. On Linux the systemd drop-in is removed and the service is restarted.--### Service detail page--Click on a service name to see its detail page:--- **Controller instance** — Which Controller node is managing this service-- **Workload info** — Namespace, deployment name, container image (Kubernetes), container ID (Docker), or PID/executable (Linux)-- **Instrumentation status** — Current state for both LLM and Agent Observability-- **SDK version** — The OpenLIT Python SDK version injected (for Agent Observability)-- **Last seen** — When the Controller last reported this service--### Controller instances--The **Controller** tab shows all running Controller instances with their:--- Node/host name-- Deploy mode (Kubernetes, Docker, Linux)-- Poll interval and last poll time-- Number of services discovered-- Version--## Troubleshooting--| Symptom | Likely cause | Fix |-|---|---|---|-| Service not appearing in the Hub | Controller not running, or service is not making LLM API calls | Check controller logs (`kubectl logs`, `docker logs`, or `journalctl`). The service must have active TCP connections to LLM API endpoints to be discovered. |-| LLM Observability enabled but no traces | eBPF probe failed to attach | Check controller logs for eBPF errors. Ensure the Controller has privileged access and required volume mounts. |-| Agent Observability shows "Error" | SDK injection failed | Check controller logs. For Kubernetes, verify RBAC allows patching deployments. For Docker, verify the Docker socket is mounted. |-| Service keeps reappearing after disable | Desired state mismatch | The Controller reconciles desired state from ClickHouse. If the service reappears, check that the disable action completed successfully in the dashboard. |-| Traces appear but with wrong service name | Service name detection issue | The Controller derives service names from workload metadata. You can override using the `OTEL_SERVICE_NAME` environment variable in your application. |--## Getting Started--<CardGroup cols={2}>- <Card title="Quickstart" href="/latest/controller/quickstart" icon="bolt">- Get the Controller running in under 5 minutes- </Card>- <Card title="Configuration" href="/latest/controller/configuration" icon="gear">- Customize discovery, polling, endpoints, and more- </Card>-</CardGroup>diff --git a/docs/latest/controller/quickstart.mdx b/docs/latest/controller/quickstart.mdxdeleted file mode 100644index 9be5d1a3..00000000--- a/docs/latest/controller/quickstart.mdx+++ /dev/null@@ -1,48 +0,0 @@-----title: "Controller Quickstart"-sidebarTitle: "Quickstart"-description: "Get zero-code LLM and Agent observability running in under 5 minutes"------import ControllerQuickstart from '/snippets/quickstart-controller.mdx';--The OpenLIT Controller uses eBPF to automatically discover and instrument LLM traffic across Kubernetes, Docker, and Linux — no SDK, code changes, or image rebuilds required.--Deploy OpenLIT with the Controller on your platform, then enable observability from the **Agents** page.--<ControllerQuickstart />--## Authenticate with an API Key--If you've created an API key in **Settings → API Keys**, configure the Controller to use it. This secures the poll endpoint so only authorized controllers can register services and receive commands.--| Platform | How to pass the key |-|---|---|-| **Kubernetes** | `--set openlit-controller.apiKey="YOUR_KEY"` in Helm |-| **Docker** | `OPENLIT_API_KEY` environment variable |-| **Linux** | `Environment="OPENLIT_API_KEY=YOUR_KEY"` in the systemd unit |--<Info>-On a fresh install with no API keys, the Controller can connect without authentication. Once you create your first API key, all controllers must authenticate. See [Configuration](/latest/controller/configuration) for the full reference.-</Info>--## What's Next?--Once services are discovered, use the **Agents** page to:--1. **Enable LLM Observability** — Click **Enable** next to any service to start capturing LLM metrics via eBPF-2. **Enable Agent Observability** — For Python services, click **Enable** to inject the OpenLIT SDK for agent framework traces-3. **View traces and metrics** — Head to the **Requests** and **Dashboards** sections to see your data--<Tip>-The Controller automatically reconciles desired state. If a pod restarts, a container is recreated, or a process bounces, the Controller will re-enable the observability you configured.-</Tip>--<CardGroup cols={2}>- <Card title="Configuration Reference" href="/latest/controller/configuration" icon="gear">- Customize polling intervals, OTLP endpoints, and more- </Card>- <Card title="Architecture Deep Dive" href="/latest/controller/architecture" icon="sitemap">- How eBPF discovery, SDK injection, and reconciliation work- </Card>-</CardGroup>diff --git a/docs/latest/glossary/what-is-ai-evaluation.mdx b/docs/latest/glossary/what-is-ai-evaluation.mdxnew file mode 100644index 00000000..c8543765--- /dev/null+++ b/docs/latest/glossary/what-is-ai-evaluation.mdx@@ -0,0 +1,38 @@+---+title: 'What is AI Evaluation?'+description: 'AI evaluation explained: how LLM-as-a-judge scoring works, online vs. offline evaluation, and how to catch quality regressions'+sidebarTitle: 'AI Evaluation'+---++AI evaluation is the process of scoring LLM outputs for quality, safety, and correctness - using an LLM-as-a-judge, rule-based checks, or human review - so you can catch bad responses before or after they reach users.++## Why does AI evaluation matter?++An LLM can produce a fluent, confident-sounding response that's factually wrong, biased, or off-topic, without ever raising an error. AI evaluation gives you a quality signal traditional monitoring can't: a score for hallucination, bias, toxicity, relevance, and more, attached to the exact prompt version and model that produced the response.++## What is LLM-as-a-judge?++LLM-as-a-judge uses a separate LLM call to grade a response against a rubric - for example, "does this response contain a hallucinated fact?" - and returns a score and explanation. It scales further than human review and can run automatically on every response in production.++## What's the difference between online and offline evaluation?++- **Online evaluation** runs automatically against live production traces - manually triggered, or on an automatic schedule with sampling to control cost.+- **Offline evaluation** runs programmatically, typically in CI/CD, against a fixed dataset before you ship a prompt or model change - catching regressions before they reach production.++## How does OpenLIT provide AI evaluation?++OpenLIT includes 11 built-in evaluation types (hallucination, bias, toxicity, and more), runs them automatically in production from [Configuration](/latest/openlit/evaluations/configuration) (Auto Evaluation), and exposes the same evaluators [programmatically via the SDK](/latest/sdk/features/evaluations) for offline testing in CI/CD.++---++<CardGroup cols={3}>+ <Card title="Evaluations overview" href="/latest/openlit/evaluations/overview" icon='gavel'>+ The 11 built-in evaluation types and how automated scoring works+ </Card>+ <Card title="Configuration" href="/latest/openlit/evaluations/configuration" icon='gear'>+ Auto Evaluation schedule and judge model under Monitor+ </Card>+ <Card title="Offline evaluations (SDK)" href="/latest/sdk/features/evaluations" icon='code'>+ Run evaluations programmatically in CI/CD and development pipelines+ </Card>+</CardGroup>diff --git a/docs/latest/glossary/what-is-ai-observability.mdx b/docs/latest/glossary/what-is-ai-observability.mdxnew file mode 100644index 00000000..db111c0e--- /dev/null+++ b/docs/latest/glossary/what-is-ai-observability.mdx@@ -0,0 +1,40 @@+---+title: 'What is AI Observability?'+description: 'AI observability explained: what it is, why it matters, and how to add it to LLM apps and agents with zero code changes'+sidebarTitle: 'AI Observability'+---++AI observability is the practice of collecting traces, metrics, and logs from LLM applications and AI agents so you can see exactly what happened on every request - which model was called, how long it took, what it cost, and whether it failed.++## Why does AI observability matter?++LLM applications fail in ways traditional monitoring doesn't catch: a model can return a slow, expensive, or low-quality response without ever throwing an error. Without AI observability, you only find out from a user complaint. With it, you can see the exact prompt, model, latency, and cost behind every call - and catch regressions before they reach production.++## What data does AI observability cover?++- **Traces**: the full request flow - model calls, tool calls, retries, and agent steps, in order.+- **Metrics**: aggregated numbers over time, like latency, token throughput, and error rate.+- **Logs**: discrete events emitted by your application and its dependencies.+- **Cost and token usage**: spend per request, model, and provider, computed from token counts and per-model pricing.++## How is AI observability different from traditional APM?++Traditional application performance monitoring (APM) tracks HTTP requests, database queries, and infrastructure health. AI observability adds what APM tools don't understand: which model and provider handled a call, the prompt and completion content, token usage, per-call cost, and the tool-call graph an agent followed to produce its answer.++## How does OpenLIT provide AI observability?++OpenLIT auto-instruments 90+ LLMs, agent frameworks, and vector databases via OpenTelemetry - with zero code changes - and surfaces traces, metrics, and logs in one [Telemetry](/latest/openlit/observability/telemetry/overview) page, plus a trace-derived call graph for every [agent](/latest/openlit/observability/agents/overview) it observes.++---++<CardGroup cols={3}>+ <Card title="Telemetry" href="/latest/openlit/observability/telemetry/overview" icon='chart-line'>+ View traces, metrics, logs, and exceptions in one place+ </Card>+ <Card title="Agents" href="/latest/openlit/observability/agents/overview" icon='robot'>+ See every agent's trace-derived call graph, tools, and versions+ </Card>+ <Card title="Quickstart: AI Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>+ Instrument an LLM app with zero code changes in 2 steps+ </Card>+</CardGroup>diff --git a/docs/latest/glossary/what-is-prompt-engineering.mdx b/docs/latest/glossary/what-is-prompt-engineering.mdxnew file mode 100644index 00000000..e5572814--- /dev/null+++ b/docs/latest/glossary/what-is-prompt-engineering.mdx@@ -0,0 +1,36 @@+---+title: 'What is Prompt Engineering?'+description: 'Prompt engineering explained: versioning, testing, and deploying prompts as managed, trackable artifacts'+sidebarTitle: 'Prompt Engineering'+---++Prompt engineering is the practice of designing, testing, and iterating on the instructions you send to an LLM to get reliable, high-quality output. In production, that means treating prompts as versioned artifacts you can edit, test, roll back, and link to the results they produced - not just strings hardcoded in your application.++## Why does prompt engineering need tooling?++A prompt that works well in testing can quietly degrade after a small wording change, a model swap, or a new edge case in production traffic. Without version history and a link back to the traces and evaluations a prompt produced, it's hard to tell what changed or why quality shifted.++## What does managing prompts as artifacts involve?++- **Versioning**: track major, minor, and patch changes to a prompt over time.+- **Variables**: use `{{placeholder}}` syntax so one prompt template serves many runtime inputs.+- **Testing before shipping**: compare how a prompt performs across different models before deploying it.+- **Traceability**: link each prompt version to the traces and evaluation scores it produced in production.++## How does OpenLIT support prompt engineering?++[Prompt Hub](/latest/openlit/prompts-experiments/prompt-hub/overview) manages prompts as versioned artifacts with variable substitution and full history, [OpenGround](/latest/openlit/prompts-experiments/openground/overview) lets you test a prompt across multiple models side-by-side before shipping, and every prompt version is linked back to the traces and evaluation results it produced in production.++---++<CardGroup cols={3}>+ <Card title="Prompt Hub" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>+ Version, deploy, and collaborate on prompts with centralized management and tracking+ </Card>+ <Card title="OpenGround" href="/latest/openlit/prompts-experiments/openground/overview" icon='flask'>+ Test the same prompt across multiple LLM providers and compare cost, speed, and quality+ </Card>+ <Card title="Rule Engine" href="/latest/openlit/prompts-experiments/rule-engine/overview" icon='gears'>+ Match runtime conditions to the right prompt, context, or evaluation config+ </Card>+</CardGroup>diff --git a/docs/latest/gpu-collector/configuration.mdx b/docs/latest/gpu-collector/configuration.mdxindex 8c3ba544..8e5744f1 100644--- a/docs/latest/gpu-collector/configuration.mdx+++ b/docs/latest/gpu-collector/configuration.mdx@@ -20,14 +20,14 @@ The collector is configured entirely via environment variables. All variables fo## Common configurations-### Minimal — send to a local OTel Collector+### Minimal - send to a local OTel Collector```bashOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \./opentelemetry-gpu-collector```-### Production — with service name, environment, and auth header+### Production - with service name, environment, and auth header```bashOTEL_SERVICE_NAME=gpu-worker \diff --git a/docs/latest/gpu-collector/gpus/intel.mdx b/docs/latest/gpu-collector/gpus/intel.mdxindex 45e2d34d..96ab22e3 100644--- a/docs/latest/gpu-collector/gpus/intel.mdx+++ b/docs/latest/gpu-collector/gpus/intel.mdx@@ -6,7 +6,7 @@ description: 'Monitor Intel GPU metrics via sysfs/hwmon using the OpenTelemetryThe collector monitors Intel GPUs via the Linux kernel's sysfs, hwmon, and DRM interfaces exposed by the i915 and Xe drivers. No Intel GPU tools, no OneAPI, and no user-space libraries are required.<Note>-Intel GPU support provides thermal, power, energy, and clock metrics. Utilization and memory metrics are not available via the sysfs/hwmon interface — these would require the [Intel XPU Manager](https://github.com/intel/xpumanager) or similar tooling.+Intel GPU support provides thermal, power, energy, and clock metrics. Utilization and memory metrics are not available via the sysfs/hwmon interface - these would require the [Intel XPU Manager](https://github.com/intel/xpumanager) or similar tooling.</Note>## Requirementsdiff --git a/docs/latest/gpu-collector/gpus/nvidia.mdx b/docs/latest/gpu-collector/gpus/nvidia.mdxindex cdb966c2..265bb7eb 100644--- a/docs/latest/gpu-collector/gpus/nvidia.mdx+++ b/docs/latest/gpu-collector/gpus/nvidia.mdx@@ -102,7 +102,7 @@ spec:```<Note>-The collector does not need privileged mode for NVML — just access to `libnvidia-ml.so`. eBPF tracing requires `CAP_BPF` + `CAP_PERFMON`.+The collector does not need privileged mode for NVML - just access to `libnvidia-ml.so`. eBPF tracing requires `CAP_BPF` + `CAP_PERFMON`.</Note>---diff --git a/docs/latest/gpu-collector/installation.mdx b/docs/latest/gpu-collector/installation.mdxindex a7c39cb3..0b4e2eb5 100644--- a/docs/latest/gpu-collector/installation.mdx+++ b/docs/latest/gpu-collector/installation.mdx@@ -113,7 +113,7 @@ rm /usr/local/bin/opentelemetry-gpu-collector## Troubleshooting<AccordionGroup>- <Accordion title="No GPU metrics — collector starts but reports no hw.gpu.* metrics">+ <Accordion title="No GPU metrics - collector starts but reports no hw.gpu.* metrics">- Confirm the host has a supported GPU: `lspci | grep -E 'VGA|3D|Display'`- For NVIDIA: verify `libnvidia-ml.so` is present: `ldconfig -p | grep nvidia-ml`- For Docker: ensure `--gpus all` (NVIDIA) or `--device /dev/dri` (AMD/Intel) is passeddiff --git a/docs/latest/gpu-collector/metrics.mdx b/docs/latest/gpu-collector/metrics.mdxindex 81df8d8d..1648961a 100644--- a/docs/latest/gpu-collector/metrics.mdx+++ b/docs/latest/gpu-collector/metrics.mdx@@ -14,19 +14,19 @@ Collected for each detected GPU on Linux. Availability depends on vendor and GPU| Metric | Type | Unit | Description | NVIDIA | AMD | Intel ||---|---|---|---|:---:|:---:|:---:|-| `hw.gpu.utilization` | Gauge | `1` | GPU compute/encoder/decoder utilization (0.0–1.0) | Yes | Yes | — |-| `hw.gpu.memory.utilization` | Gauge | `1` | Memory controller utilization (0.0–1.0) | Yes | Yes | — |-| `hw.gpu.memory.limit` | UpDownCounter | `By` | Total GPU memory | Yes | Yes | — |-| `hw.gpu.memory.usage` | UpDownCounter | `By` | Used GPU memory | Yes | Yes | — |-| `hw.gpu.memory.free` | UpDownCounter | `By` | Free GPU memory | Yes | Yes | — |+| `hw.gpu.utilization` | Gauge | `1` | GPU compute/encoder/decoder utilization (0.0–1.0) | Yes | Yes | - |+| `hw.gpu.memory.utilization` | Gauge | `1` | Memory controller utilization (0.0–1.0) | Yes | Yes | - |+| `hw.gpu.memory.limit` | UpDownCounter | `By` | Total GPU memory | Yes | Yes | - |+| `hw.gpu.memory.usage` | UpDownCounter | `By` | Used GPU memory | Yes | Yes | - |+| `hw.gpu.memory.free` | UpDownCounter | `By` | Free GPU memory | Yes | Yes | - || `hw.gpu.temperature` | Gauge | `Cel` | Die or memory temperature | Yes | Yes | Yes || `hw.gpu.fan_speed` | Gauge | `{rpm}` | Fan speed | Yes | Yes | Yes* || `hw.gpu.power.draw` | Gauge | `W` | Current power draw | Yes | Yes | Yes || `hw.gpu.power.limit` | Gauge | `W` | Power limit/cap | Yes | Yes | Yes || `hw.gpu.energy.consumed` | Counter | `J` | Cumulative energy consumed | Yes | Yes | Yes || `hw.gpu.clock.graphics` | Gauge | `MHz` | Graphics/SM clock frequency | Yes | Yes | Yes* |-| `hw.gpu.clock.memory` | Gauge | `MHz` | Memory clock frequency | Yes | Yes | — |-| `hw.errors` | Counter | `{error}` | ECC and PCIe error count | Yes | — | — |+| `hw.gpu.clock.memory` | Gauge | `MHz` | Memory clock frequency | Yes | Yes | - |+| `hw.errors` | Counter | `{error}` | ECC and PCIe error count | Yes | - | - |\* Intel fan speed requires Linux kernel 6.16+. Intel graphics clock requires the Xe driver.diff --git a/docs/latest/gpu-collector/overview.mdx b/docs/latest/gpu-collector/overview.mdxindex d381ec1d..760c7b4f 100644--- a/docs/latest/gpu-collector/overview.mdx+++ b/docs/latest/gpu-collector/overview.mdx@@ -3,29 +3,29 @@ title: 'Overview'description: 'OpenTelemetry-native GPU and host metrics collector for NVIDIA, AMD, and Intel GPUs'----The **OpenTelemetry GPU Collector** is a lightweight, single-binary metrics collector written in Go. It exports GPU hardware telemetry, host system metrics, and process metrics via OpenTelemetry (OTLP) — with no Python dependencies, no DCGM daemon, and no vendor-specific agents.+The **OpenTelemetry GPU Collector** is a lightweight, single-binary metrics collector written in Go. It exports GPU hardware telemetry, host system metrics, and process metrics via OpenTelemetry (OTLP) - with no Python dependencies, no DCGM daemon, and no vendor-specific agents.It is fully configured via standard OpenTelemetry environment variables and follows the [OTel semantic conventions for hardware metrics](https://opentelemetry.io/docs/specs/semconv/hardware/gpu/).## Goals-* **OpenTelemetry-native** — uses standard `OTEL_*` env vars, exports via OTLP gRPC or HTTP to any OTel-compatible backend-* **Cross-vendor GPU support** — NVIDIA (NVML), AMD (sysfs/hwmon), and Intel (i915/Xe sysfs) from a single binary-* **OTel semantic conventions** — `hw.gpu.*` metric names, `hw.id` / `hw.name` / `hw.vendor` attributes per spec-* **Zero dependencies** — no DCGM, no Python, no CUDA toolkit needed at runtime for hardware metrics-* **Resilient** — continues exporting host metrics even when no GPUs are present; retries GPU discovery every 30s+* **OpenTelemetry-native** - uses standard `OTEL_*` env vars, exports via OTLP gRPC or HTTP to any OTel-compatible backend+* **Cross-vendor GPU support** - NVIDIA (NVML), AMD (sysfs/hwmon), and Intel (i915/Xe sysfs) from a single binary+* **OTel semantic conventions** - `hw.gpu.*` metric names, `hw.id` / `hw.name` / `hw.vendor` attributes per spec+* **Zero dependencies** - no DCGM, no Python, no CUDA toolkit needed at runtime for hardware metrics+* **Resilient** - continues exporting host metrics even when no GPUs are present; retries GPU discovery every 30s## What it collects<CardGroup cols={3}><Card title="GPU Hardware Telemetry" icon="microchip">- Utilization, memory, temperature, power draw, energy, clock speeds, ECC errors, PCIe errors — for NVIDIA, AMD, and Intel GPUs+ Utilization, memory, temperature, power draw, energy, clock speeds, ECC errors, PCIe errors - for NVIDIA, AMD, and Intel GPUs</Card><Card title="Host System Metrics" icon="server">- CPU utilization, memory usage, disk I/O, filesystem usage, and network I/O — on Linux, macOS, and Windows+ CPU utilization, memory usage, disk I/O, filesystem usage, and network I/O - on Linux, macOS, and Windows</Card><Card title="eBPF CUDA Tracing" icon="code">- Kernel launch counts, grid/block sizes, memory allocations, and memory copies — via uprobes on libcudart.so (opt-in, Linux only)+ Kernel launch counts, grid/block sizes, memory allocations, and memory copies - via uprobes on libcudart.so (opt-in, Linux only)</Card></CardGroup>@@ -33,8 +33,8 @@ It is fully configured via standard OpenTelemetry environment variables and foll| Vendor | Backend | Metrics available ||---|---|---|-| **NVIDIA** | NVML via [go-nvml](https://github.com/NVIDIA/go-nvml) — loads `libnvidia-ml.so` at runtime | Utilization, memory, temperature, power, energy, clocks, ECC errors, PCIe errors, fan speed |-| **AMD** | sysfs + hwmon — zero external dependencies | Utilization, memory, temperature, power, energy, fan speed |+| **NVIDIA** | NVML via [go-nvml](https://github.com/NVIDIA/go-nvml) - loads `libnvidia-ml.so` at runtime | Utilization, memory, temperature, power, energy, clocks, ECC errors, PCIe errors, fan speed |+| **AMD** | sysfs + hwmon - zero external dependencies | Utilization, memory, temperature, power, energy, fan speed || **Intel** | sysfs + hwmon + DRM (i915/Xe driver) | Temperature, power draw/limit, cumulative energy, graphics clock, fan speed (kernel 6.16+) |## Platform support@@ -43,8 +43,8 @@ It is fully configured via standard OpenTelemetry environment variables and foll|---|:---:|:---:|:---:|| System metrics (CPU, memory, disk, network) | Yes | Yes | Yes || Process metrics (CPU, memory, threads, FDs) | Yes | Yes | Yes |-| GPU metrics (NVIDIA, AMD, Intel) | Yes | — | — |-| eBPF CUDA tracing | Yes | — | — |+| GPU metrics (NVIDIA, AMD, Intel) | Yes | - | - |+| eBPF CUDA tracing | Yes | - | - |## How it works@@ -70,13 +70,13 @@ ExportOn startup, the collector scans `/sys/bus/pci/devices/` for PCI class codes `0x0300` (VGA), `0x0302` (3D controller), and `0x0380` (display controller), identifying GPU vendor from the PCI vendor ID.</Step><Step title="Initialises vendor backend">- Each detected GPU is handed to its vendor-specific backend. NVIDIA loads `libnvidia-ml.so` via NVML. AMD and Intel read directly from kernel sysfs/hwmon — no additional libraries needed.+ Each detected GPU is handed to its vendor-specific backend. NVIDIA loads `libnvidia-ml.so` via NVML. AMD and Intel read directly from kernel sysfs/hwmon - no additional libraries needed.</Step><Step title="Registers OTel instruments">Observable gauge and counter instruments are registered with the OTel SDK meter. On each collection tick, the SDK calls back into the collector to read fresh values from each GPU.</Step><Step title="Exports via OTLP">- Metrics are exported via OTLP to any compatible backend — OpenLIT, Grafana, Datadog, New Relic, or a standard OTel Collector.+ Metrics are exported via OTLP to any compatible backend - OpenLIT, Grafana, Datadog, New Relic, or a standard OTel Collector.</Step></Steps>diff --git a/docs/latest/openlit/chat.mdx b/docs/latest/openlit/chat.mdxdeleted file mode 100644index b607d89a..00000000--- a/docs/latest/openlit/chat.mdx+++ /dev/null@@ -1,239 +0,0 @@-----title: 'Otter — AI Chat Copilot'-description: 'Meet Otter, your AI copilot for querying data, managing resources, and creating dashboards'------Meet **Otter** 🦦 — your AI-powered observability copilot built into OpenLIT. Otter lets you interact with your observability platform using natural language. It can query your telemetry data, create and manage platform resources, and generate complete dashboards — all through conversation.--## Meet Otter--Otter is always just a click away via the floating chat button (bottom-right corner of every page) or from the **Chat** entry in the sidebar. Otter paddles through your traces, metrics, and logs so you don't have to.--<Frame>- <img src="/images/otter.svg" width="80" />-</Frame>--## Key Features--- **Natural Language Queries**: Ask questions about your traces, metrics, costs, and tokens — the AI converts them to ClickHouse SQL and executes them-- **Trace and Span Analysis**: Run structured AI analysis on a full trace hierarchy or an individual span, with findings grouped by performance, reliability, cost, token efficiency, and execution path-- **Prompt Improvement**: Ask Otter to review an existing Prompt Hub prompt and suggest improvements without automatically saving changes-- **Resource Management**: Create and manage rules, contexts, prompts, vault secrets, and custom models through conversation-- **Dashboard Generation**: Describe the dashboard you want and get a complete importable layout with working queries-- **SQL Execution**: Generated queries run automatically with results displayed inline as tables and charts-- **Save as Widget**: Save any query result as a reusable dashboard widget--## Get Started--<Steps>- <Step title="Configure AI Provider">- Navigate to **Chat → Settings** (gear icon in chat sidebar) and configure:- 1. **Provider** — Select from 14 supported providers (OpenAI, Anthropic, Google, etc.)- 2. **Model** — Choose a model from the provider's list (includes custom models from Manage Models)- 3. **API Key** — Select a vault secret containing your provider's API key-- <Info>- Store your API key in the **Vault** first, then select it in Chat Settings.- </Info>- </Step>-- <Step title="Start a Conversation">- Click **New Chat** in the sidebar or use the floating chat button available on any page.-- Try example prompts like:- - "What are my top 5 most expensive models this week?"- - "Show me error rate by provider over the last 24 hours"- - "Create a rule that triggers when request duration exceeds 5 seconds"- </Step>-- <Step title="View Results">- - **SQL queries** appear in a collapsible accordion with syntax highlighting- - **Query results** display automatically as tables, bar charts, line charts, or pie charts- - **Entity cards** link to created resources (rules, contexts, prompts)- - **Dashboard cards** show an import button for generated dashboards- </Step>-</Steps>--## Data Queries--Ask questions about your OpenTelemetry data and the AI generates ClickHouse SQL with dynamic time filters.--### Supported Tables--| Table | Description |-|-------|-------------|-| `otel_traces` | Traces and spans (LLM calls, API requests, etc.) |-| `otel_metrics_gauge` | Gauge metrics (GPU temperature, memory, etc.) |-| `otel_metrics_sum` | Counter/sum metrics (token counts, etc.) |-| `otel_metrics_histogram` | Histogram metrics (request durations, etc.) |--### Example Queries--```-"How many requests did I get today?"-"What is my total cost breakdown by provider?"-"Show me the slowest requests in the last hour"-"Which models have the highest error rate?"-```--### Save as Widget--Click **Save as Widget** on any query result to:-1. Choose a widget type (Stat Card, Bar Chart, Line Chart, Pie Chart, Area Chart, Table)-2. Select a dashboard to add it to-3. The query is automatically converted to use dynamic time filters (`{{filter.timeLimit.start/end}}`)--## AI Analysis for Traces and Spans--Otter can analyze telemetry directly from chat or from the trace detail view. Use this when you want a structured explanation of what happened in a trace, why a span is slow or expensive, or which part of an agent path should be improved.--### What Otter Reviews--Trace and span analysis evaluates:--- **Performance**: slow spans, blocking operations, duration hotspots, and inefficient execution paths.-- **Reliability**: failed spans, exception signals, retry patterns, and unstable downstream calls.-- **Cost**: model usage, high-cost spans, token usage, and estimated spend.-- **Token efficiency**: input/output token balance, cache usage, repeated context, and response verbosity.-- **Prompt and model behavior**: prompt structure, model selection, and output quality signals.-- **Telemetry quality**: missing span attributes, weak resource metadata, and incomplete trace context.-- **Actionability**: each analysis section is reviewed by a grader pass to make recommendations clearer and more useful.--### How to Run It--You can ask Otter questions like:--```-"Analyze this trace and explain the cost drivers"-"Review span <span_id> and suggest improvements"-"Find token waste in traces for session.id dc7b1ba1-..."-"Analyze the slowest trace from the last hour"-```--When a trace or span is referenced from a telemetry view, Otter opens the same analysis experience with the relevant context already selected. Existing analysis runs are reused unless you explicitly ask for a fresh run.--### Output--The response streams progress while each analysis dimension runs, then collapses into a completed analysis summary. Findings include the affected spans, severity, supporting evidence, and suggested fixes.--## Prompt Improvement from Chat--Otter can review Prompt Hub prompts by name or ID. For review-only requests, it loads the prompt and suggests changes without mutating the saved prompt.--Example prompts:--```-"Can you help me improve prompt with prompt name music_recommend?"-"Review prompt customer_summary and make it more concise"-"Suggest safer output-format instructions for prompt_id <uuid>"-```--<Note>- Otter only updates a Prompt Hub version when you explicitly ask it to save, apply, update, publish, or create a new version.-</Note>--## Resource Management--The chat can create and manage platform resources through natural language. All operations execute immediately.--### Supported Operations--| Resource | Operations |-|----------|-----------|-| **Rules** | Create, update, delete, list, get details, link/unlink entities |-| **Contexts** | Create, update, delete, list |-| **Prompts** | Create, read, improve, update version, delete, list |-| **Vault Secrets** | Create, update, delete, list |-| **Custom Models** | Create, update, delete, list |--### Examples--```-"Create a vault secret called OPENAI_KEY with value sk-..."-"Create a rule named high-cost-alert that triggers when cost exceeds 0.1"-"Create a prompt template for customer feedback summarization"-"Link the customer-context to the high-cost-alert rule"-"List all my vault secrets"-```--<Note>- Vault key names are automatically normalized to UPPER_SNAKE_CASE (e.g., "my openai key" becomes "MY_OPENAI_KEY").-</Note>--## Dashboard Generation--Describe the dashboard you want and the AI generates a complete importable layout.--### How it Works--1. Ask: *"Create a GPU monitoring dashboard with utilization, memory, and temperature widgets"*-2. The AI generates a dashboard JSON with:- - Widget definitions (type, title, SQL query, display properties)- - Grid layout positions- - Dynamic time filters (`{{filter.timeLimit.start/end}}`)-3. A **dashboard card** appears with:- - Dashboard title and widget count- - Widget preview chips- - **Import Dashboard** button- - **Download JSON** button-4. Click **Import Dashboard** to create it instantly--### Dashboard Queries--All generated queries use the Mustache template pattern for dynamic time filtering:--```sql-WITH- parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,- parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time-SELECT ...-FROM otel_traces-WHERE Timestamp >= start_time AND Timestamp <= end_time-```--This ensures dashboards respond to the time filter controls in the UI.--## Cost & Token Tracking--Each conversation tracks:-- **Per-message**: Input tokens, output tokens, estimated cost-- **Per-conversation**: Accumulated totals displayed in the sidebar--The cost formula:-```-cost = (input_tokens / 1M) × input_price + (output_tokens / 1M) × output_price-```--Hover over the info icon next to the chat input to see your current provider, model, and pricing details.--## Conversation Management--- **URL-based navigation**: Each conversation has a unique URL (`/chat?id=<uuid>`)-- **Auto-generated titles**: Titles are generated after the first message exchange-- **Persistent history**: Conversations and messages are stored in ClickHouse-- **Search**: Filter conversations by title in the sidebar-- **Delete**: Hover over a conversation and click the trash icon--## Configuration--### Chat Settings--Access via the gear icon in the chat sidebar or navigate to `/chat/settings`.--| Setting | Description |-|---------|-------------|-| **Provider** | LLM provider (OpenAI, Anthropic, Google, Mistral, Cohere, Groq, etc.) |-| **Model** | Model from the provider's list (includes custom models) |-| **API Key** | Vault secret reference for the provider's API key |--### Supported Providers--OpenAI, Anthropic, Google, Mistral, Cohere, Groq, Perplexity, Azure OpenAI, Together AI, Fireworks AI, DeepSeek, xAI, Hugging Face, Replicate--## Security--- All SQL queries execute with `readonly` mode — no write operations possible-- Queries are validated against an allowlist of tables (`otel_traces`, `otel_metrics_*`)-- DDL/DML statements (INSERT, UPDATE, DELETE, DROP, etc.) are rejected-- API keys are stored in the Vault and never sent to the client-- Query results have a LIMIT cap of 10,000 rowsdiff --git a/docs/latest/openlit/chat/conversations.mdx b/docs/latest/openlit/chat/conversations.mdxnew file mode 100644index 00000000..2743632b--- /dev/null+++ b/docs/latest/openlit/chat/conversations.mdx@@ -0,0 +1,51 @@+---+title: "Chat with Otter"+sidebarTitle: "Chat"+description: "Use Otter's AI chat assistant to query observability data with natural language, save widgets, generate dashboards, and manage OpenLIT resources"+---++The main Otter screen is `/chat` (optional `?id=<conversation-id>` for a specific thread). Empty-state title: **Your AI-Powered Observability Copilot**.++## Conversations++- **New Chat** starts a conversation; titles are generated after the first exchange.+- Search with **Search conversations...** in the sidebar.+- Delete a conversation from its row (trash on hover).+- Each conversation has its own URL so you can bookmark or share the link within your team.++Placeholder for the composer: **Ask a question about your data...** (Enter to send, Shift+Enter for a new line).++Empty-state example chips include analyzing slow traces from the last 24 hours, token usage and cost by model, recent errors for a busy service, creating a dashboard for volume/errors/latency/cost, improvement analysis on high-cost traces, and creating a rule for failures or slow requests.++## Data questions and SQL++When Otter answers with a query, you see a collapsible **SQL Query** block, **Copy** / **Execute** / **Running...**, and results you can view as **Table**, **Bar**, **Line**, or **Pie** (plus an auto **Stat Card** when it fits). Use **Save as Widget** to enter a required **Title**, choose a widget type (**Stat Card**, **Bar Chart**, **Line Chart**, **Pie Chart**, **Area Chart**, **Table**), and optionally add it to a dashboard.++Queries are constrained to telemetry data (traces and metrics tables) in read-only mode with a result row cap - Otter is not a general-purpose database admin console.++## Resources and dashboards++Ask Otter to create or list rules, contexts, prompts, Vault secrets, custom models, and related resources. Created entities show up as cards with links into the product. For dashboards, Otter can return an importable layout with **Import Dashboard**, **Download JSON**, and **Open Dashboard** after import.++Vault key names are normalized to `UPPER_SNAKE_CASE` when Otter creates secrets.++## Analysis and prompt improvement++You can ask Otter to analyze traces or spans (including by session or other attributes) or to review a Prompt Hub prompt by name or ID. Review-only asks suggest edits without saving; Otter only writes a new prompt version when you explicitly ask it to save, apply, update, publish, or create a version.++For the telemetry-side UI of the same analysis pipeline, see [AI Analysis](/latest/openlit/observability/telemetry/ai-analysis).++## Cost on each message++Per-message tokens and estimated cost appear on each reply; the conversation list can show cost when it is greater than zero. Hover the info control near the input for **Active Configuration** details (provider, model, and pricing). Those prices come from [Manage Models](/latest/openlit/costs/manage-models/overview).++---++<CardGroup cols={2}>+ <Card title="Chat Settings" href="/latest/openlit/chat/settings" icon="gear">+ Provider, model, and Vault API key+ </Card>+ <Card title="Otter usage" href="/latest/openlit/chat/usage" icon="chart-column">+ Aggregate tokens and cost across Otter features+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/chat/overview.mdx b/docs/latest/openlit/chat/overview.mdxnew file mode 100644index 00000000..070395ed--- /dev/null+++ b/docs/latest/openlit/chat/overview.mdx@@ -0,0 +1,58 @@+---+title: "Overview"+description: "Otter is OpenLIT's AI chat assistant for observability - ask natural language questions about traces, costs, and tokens, and manage resources through conversation"+---++**Otter** is OpenLIT's AI chat assistant for observability data. Ask questions in plain language about traces, costs, and tokens; create dashboards, rules, prompts, and Vault secrets through conversation; and run the same AI analysis you get from the telemetry detail views. Product copy describes it as: *Ask questions about your observability data using natural language*.++Open Otter from the **Otter** mode toggle in the sidebar (`/chat`), or from the floating Otter button on other playground pages (hidden while you are already on `/chat*`). Configure a provider first under [Chat Settings](/latest/openlit/chat/settings), then start chatting. Track spend on the [Otter usage](/latest/openlit/chat/usage) page.++Trends for narrow phrases like "AI copilot for observability" are low-volume; broader interest sits around AI chat assistants and natural-language querying - Otter is the observability-native version of that workflow.++<CardGroup cols={2}>+ <Card title="Chat with Otter" href="/latest/openlit/chat/conversations" icon="comment">+ Conversations, SQL results, widgets, and resource actions+ </Card>+ <Card title="Chat Settings" href="/latest/openlit/chat/settings" icon="gear">+ Pick AI Provider, Model, and a Vault API key+ </Card>+ <Card title="Otter usage" href="/latest/openlit/chat/usage" icon="chart-column">+ Token and cost attribution by feature, provider, model, and date+ </Card>+ <Card title="AI Analysis" href="/latest/openlit/observability/telemetry/ai-analysis" icon="wand-magic-sparkles">+ Trace and span analysis Otter can also run from chat+ </Card>+</CardGroup>++## What Otter can do++- **Natural language data questions** - Otter turns questions into read-only SQL against your telemetry tables and shows results as tables or charts.+- **Save as Widget** - turn a query result into a dashboard widget.+- **Resource management** - create and manage rules, contexts, prompts, Vault secrets, and custom models through conversation.+- **Dashboard generation** - describe a dashboard and import the generated layout.+- **Trace / span analysis and prompt improvement** - same capabilities as the telemetry AI Analysis and Prompt Hub Otter flows, reachable from chat.++## Get started++<Steps>+ <Step title="Configure Chat Settings">+ Open `/chat/settings` (gear in the Otter sidebar). Choose **AI Provider**, **Model**, and **API Key (from Vault)**, then **Save Configuration** (or **Update Configuration** if one already exists). Store the key in [Vault](/latest/openlit/developer-resources/vault/overview) first, or use **Create new** on the settings form.+ </Step>+ <Step title="Start a conversation">+ Open `/chat`, click **New Chat**, and try an empty-state example such as analyzing slow traces from the last 24 hours or breaking down token usage and cost by model.+ </Step>+ <Step title="Review usage">+ Open **Otter usage** (`/chat/usage`) to see tokens and cost by provider, model, and where Otter was used (chat, trace analysis, span analysis, prompt improvement).+ </Step>+</Steps>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Do I need Chat Settings before I can talk to Otter?">+ Yes. Until a provider, model, and Vault API key are saved, Otter prompts you to configure Chat Settings.+ </Accordion>+ <Accordion title="Can Otter change my data or run destructive SQL?">+ Query tools run read-only against an allowlisted set of telemetry tables. Resource tools only change platform entities (prompts, rules, and so on) when you ask Otter to create or update them.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/chat/settings.mdx b/docs/latest/openlit/chat/settings.mdxnew file mode 100644index 00000000..6bf4302a--- /dev/null+++ b/docs/latest/openlit/chat/settings.mdx@@ -0,0 +1,35 @@+---+title: "Chat Settings"+description: "Configure Otter's AI Provider, Model, and Vault API key on the Chat Settings page before using the AI chat assistant"+---++**Chat Settings** (`/chat/settings`) configures which model Otter uses. Page title: **Chat Settings**. Description: *Configure the AI provider for the chat feature*.++Open it from the gear control in the Otter sidebar, or go directly to `/chat/settings`.++## Fields++| Setting | Label in UI | Notes |+|---------|-------------|--------|+| Provider | **AI Provider** | Providers come from [Manage Models](/latest/openlit/costs/manage-models/overview) (built-in plus any you added). |+| Model | **Model** | Models for the selected provider, including custom ones. Hint: *Models are managed in Manage Models. Custom models you add there will appear here.* |+| API key | **API Key (from Vault)** | Pick a Vault secret, or use **Create new** to open **Create a new secret** (Key, Value, Tags) without leaving Settings. |++Click **Save Configuration** the first time, or **Update Configuration** when editing an existing setup. Success toast: **Chat configuration saved**.++You must fill all three fields. If no provider is selected yet, the model field prompts you to select a provider first. A **Configured** chip shows the current provider / model / vault key when settings are already saved.++## Related++Models and prices are edited on Manage Models; secrets are stored in Vault. Otter only stores a reference to the Vault secret - the key is not sent to the browser as part of chat messages.++---++<CardGroup cols={2}>+ <Card title="Manage Models" href="/latest/openlit/costs/manage-models/overview" icon="sliders">+ Add providers and models Otter can select+ </Card>+ <Card title="Vault" href="/latest/openlit/developer-resources/vault/overview" icon="vault">+ Store the provider API key Otter should use+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/chat/usage.mdx b/docs/latest/openlit/chat/usage.mdxnew file mode 100644index 00000000..b45b67a1--- /dev/null+++ b/docs/latest/openlit/chat/usage.mdx@@ -0,0 +1,36 @@+---+title: "Otter usage"+description: "Track Otter AI chat assistant token and cost attribution by feature, provider, model, and date"+---++**Otter usage** (`/chat/usage`) shows how much Otter has spent across features. Header: **Otter usage**. Description: *Token and cost attribution by feature, provider, model, and date.*++## What you see++- Summary tiles: **Total tokens**, **Total cost**, **Prompt / completion**, **Otter actions**+- Chat rollups: **Chat conversations**, **Chat messages**, **Avg tokens / chat**, **Avg cost / chat**+- **Provider and model spend** - breakdown by provider and model+- **Where Otter was used** - activity list with type labels:+ - **Chat**+ - **Trace analysis**+ - **Span analysis**+ - **Prompt improvement**++Filter with the time tabs (**24H**, **7D**, **1M**, **3M**, **CUSTOM**) and click **Refresh** to reload.++Empty state: **No Otter usage recorded yet** - *Run a chat or AI analysis to see provider, model, token, and cost attribution here.*++## Why it matters++Otter calls your configured provider for chat, telemetry AI Analysis, and Prompt Hub improvements. This page is the single place to see that spend without digging through individual conversations.++---++<CardGroup cols={2}>+ <Card title="Chat with Otter" href="/latest/openlit/chat/conversations" icon="comment">+ Start conversations that show up in usage+ </Card>+ <Card title="Chat Settings" href="/latest/openlit/chat/settings" icon="gear">+ Change provider or model to control future cost+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/coding-agents/analytics.mdx b/docs/latest/openlit/coding-agents/analytics.mdxnew file mode 100644index 00000000..3c59f309--- /dev/null+++ b/docs/latest/openlit/coding-agents/analytics.mdx@@ -0,0 +1,48 @@+---+title: "Analytics"+description: "The seeded Coding Agents dashboard - sessions, cost, acceptance, top tools and repos, and session outcomes, scoped per vendor or per user"+---++Every vendor row on the [Agents page](/latest/openlit/observability/agents/overview)'s Coding Agents tab opens to an **Overview** tab: a seeded, read-only dashboard scoped to that vendor and honoring the global time-range picker. The same dashboard is reused, scoped to a single user, when you click into a row on the [Users](/latest/openlit/coding-agents/users) tab.++## Widgets++- **Stat cards**: Total sessions, Active users, Total cost, Total tool calls, Lines of code accepted, Acceptance %, Commits, Pull requests.+- **Trends**: Sessions over time, Lines over time (accepted vs. rejected).+- **Breakdowns**: Sessions by vendor, Top tools, Top models, Top repositories, Top users, Top users by accepted lines, Acceptance breakdown, Session outcomes (merged, committed, abandoned, cancelled), Edit decisions (auto-applied vs. user-reviewed), MCP servers in use.++Clicking **Top users** or **Top repositories** drills into that user's or repository's own scoped view of the same dashboard.++## How the numbers are computed++- **Acceptance %** is computed from edit-decision counts - `accept` and `auto_accepted` decisions against `reject` decisions - not from a raw lines-changed ratio.+- **Commits** and **Pull requests** are detected heuristically from shell/Bash tool calls (`git commit`, `gh pr ...`), since this isn't a native VCS integration. Treat them as a strong signal rather than an exact count.+- Every widget reads the same `{{filter.timeLimit.start}}` / `{{filter.timeLimit.end}}` bindings as the rest of OpenLIT's dashboards, so changing the time range picker updates the whole board consistently.++## Query the same data yourself++The Overview dashboard reads from the same `otel_traces` table as everything else in OpenLIT, so you can build your own [custom dashboard widgets](/latest/openlit/dashboards/overview) against it:++```sql+SELECT *+FROM otel_traces+WHERE SpanName IN (+ 'coding_agent.session',+ 'coding_agent.edit.decision',+ 'coding_agent.git.commit',+ 'coding_agent.git.pull_request'+)+```++The metrics pipeline emits the same rollups as counters, if you'd rather point a Prometheus/Mimir/Datadog backend at the OpenLIT collector directly: `coding_agent.session.count`, `coding_agent.session.duration`, `coding_agent.tool.call.count`, `coding_agent.lines_of_code.count`, `coding_agent.edit.decision.count`, `coding_agent.commit.count`, `coding_agent.pull_request.count`.++---++<CardGroup cols={2}>+ <Card title="Sessions" href="/latest/openlit/coding-agents/sessions" icon='list'>+ Drill from an aggregate metric down to the individual sessions behind it+ </Card>+ <Card title="Users" href="/latest/openlit/coding-agents/users" icon='users'>+ The same dashboard, scoped to one developer+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/coding-agents/architecture.mdx b/docs/latest/openlit/coding-agents/architecture.mdxdeleted file mode 100644index 01b34c82..00000000--- a/docs/latest/openlit/coding-agents/architecture.mdx+++ /dev/null@@ -1,398 +0,0 @@-----title: "Coding Agents — Architecture & Vendor Authoring"-sidebarTitle: "Architecture"-description: "How OpenLit ingests, normalises, and rolls up coding-agent telemetry — and how to add a new vendor without breaking what's already there."------This page is for contributors. If you just want to send Claude Code or-Cursor telemetry into OpenLit, the-[Onboarding](/latest/features/coding-agents/onboarding) page is what you-want. Read on if you're:--- Adding a new vendor adapter (Claude Code, Codex, Windsurf, …).-- Touching the rollup / materializer that powers the `/agents` and- `/coding-agents` views.-- Debugging missing rows, empty pills, or doubled sessions.--<Note>-This doc is paired with the in-repo Cursor rule-`.cursor/rules/coding-agents-hook.mdc`. The rule auto-loads when you-edit anything under `cli/internal/coding/`, `sdk/go/semconv/`, or the-coding-agents query layer; the rule is the canonical authoring-checklist, this doc is the conceptual overview.-</Note>--## Data flow--```-Vendor hook event ──► openlit CLI ──► OTLP/HTTP ──► OpenLit collector- │ │- ▼ ▼- sessionstate cache otel_traces (ClickHouse)- (XDG cache dir) │- ▼- materializer (cron)- │- ▼- openlit_agents_summary- │- ▼- /agents + /coding-agents UI-```--Each developer machine runs one `openlit` CLI binary. The vendor's-plugin / hook system invokes `openlit hook <vendor>` on every event;-the CLI is short-lived (one process per event) so anything stateful-lives in the session-state cache at-`$XDG_CACHE_HOME/openlit/sessions/<session-id>.json`.--## Identity model--| Concept | Where it lives | Stable across |-| ------- | -------------- | ------------- |-| `session.id` | Span + resource attr `coding_agent.session.id` | One process / hook invocation |-| `conversation.id` | Resource attr `gen_ai.conversation.id` | Multiple sessions when the vendor reports a stable chat id |-| `chat_id` (UI) | Derived: `coalesce(parent_id, session_id)` | One chat thread including its subagents |-| `agent_key` | Computed `computeAgentKey(cluster, env, vendor)` | One per vendor per environment |-| `user` | Resource attr `gen_ai.user.name` | One developer per machine |--The UI rolls up at `chat_id`. The materializer rolls up at `chat_id`,-then by `vendor`. The hub on `/agents` shows one row per vendor.--## Capture modes--`OPENLIT_CODING_CONTENT_CAPTURE` controls what lands on spans. Pick one:--| Mode | Identifiers | Tool names + paths | File diffs / message bodies |-| --------------- | ----------- | ------------------ | --------------------------- |-| `minimal` | ✅ | ❌ | ❌ |-| `metadata_only` | ✅ | ✅ | ❌ |-| `full` | ✅ | ✅ | ✅ |--Two redaction tiers always run, regardless of mode: a token-pattern-scrubber (`tier 1`) and a body-scope scrubber (`tier 2`, only active in-`full`). See `cli/internal/redact/redact.go`.--The mode is stamped as a resource attribute-`coding_agent.content_capture_mode` so audit trails can prove what the-session was recorded under.--## Vendor adapter contract--Each adapter under `cli/internal/coding/hook/<vendor>/` is responsible-for:--1. Parsing the vendor's JSON payload into Go structs.-2. Picking the **session id** (vendor-specific key name).-3. Mapping the vendor's event semantics onto our `normalize` types:- - `Session` — sessionStart / sessionEnd lifecycle.- - `LLMTurn` — one LLM call (prompt + completion).- - `ToolCall` — one tool invocation by the agent.- - `EditDecision` — accept / reject / undo of a code edit.- - `ShellRequest` — agent-issued shell command.- - `Event` — generic counter or "something happened" marker.-4. Calling the matching `emit.Emit*` method on the OTLP emitter.--The adapter does **not** decide what to redact, what to drop, or what-trace id to use. Those live in the OTLP layer-(`cli/internal/otlp/{attrs,exporter,sampler,tracecontext}.go`) so-adapters stay narrow and consistent.--## Common pitfalls (and what to do instead)--### Empty `/agents` "Coding Agents" tab--Cause: the materializer's discovery query has an aggregate in `GROUP BY`,-or a CTE that inlines into nested aggregates. The materializer's-`/api/agents/materialize` route returns `processed: N` for *other*-rows while the coding-vendor row silently errors.--Fix: rewrite the discovery query to use a per-row chat_id expression-(map lookups + coalesce, no `any()`) and group by it directly. See-`discoverCodingAgents` in `src/client/src/lib/platform/agents/materialize.ts`.--### Empty Repository / Working Folder pills on tool / llm spans--Cause: Cursor only sends `workspace_roots` on session lifecycle events.-Per-tool hooks come in without any cwd field, and `os.Getwd()` wasn't-persisted to the session-state cache.--Fix: in `hook.go`, when `cached.CWD == ""`, resolve `os.Getwd()` and-write it back to `cached.CWD` before calling `git.Snapshot`. The save-gate already covers `cached.CWD != ""`.--### One chat appears as N sessions--Cause: a subagent is reporting its own `session_id` but not the-parent's `parent_conversation_id`, so the chat-thread coalesce falls-through to per-process session_id.--Fix: confirm `peekContext` is picking the vendor's parent-id field,-and that it's promoted to the `coding_agent.agent.parent_id` resource-attribute in the `sessionAttrs` block of `hook.go`.--### Double session-root span--Cause: `EmitSession` is emitting a root span on both `sessionStart`-("started") and `sessionEnd` (outcome="completed" / "errored").--Fix: only emit on `End` events. The `started` / `in_progress` outcomes-are no-ops in `EmitSession` — they only update the session-state cache.--### `coding_active_users_24h` is always 0 on a single-user install--Cause: `COHORT_K_FLOOR` was applied at materialize time. Since the-single developer is below the floor, the count is masked to 0 in-the materialized table.--Fix: don't apply the floor in the materializer. Store the raw count.-Apply the floor at query time in `queries.ts` where auth context is-available, so admins see the truth and viewers see the masked view.--### `active_users_24h` shows 0 in queries even after fix--Cause: `USER_EXPR` doesn't include the vendor's identity key. For-Cursor, identity arrives as `user.email` on tool spans and-`gen_ai.user.name` as a resource attribute on every span.--Fix: confirm the resource attribute is set (check-`sessionstate/<sid>.json` for a `user` field). If not, the vendor's-payload didn't include identity and we need to widen `peekContext`'s-key list.--### Agent detail page renders the SDK layout for a coding agent--You click into a Cursor (or Claude Code, …) row in the **Coding-Agents** tab and instead of the Overview / Sessions / Users tabs you-get the generic Overview / Dashboard / Monitoring / Definition /-Configuration tabs from the SDK/Controller detail page. Refresh a few-times and the layout flips back.--Cause: two rows exist in `openlit_agents_summary` with the **same-agent_key** but different `source` — one `coding` and one `sdk`.-The CLI sets `service.name = '<vendor>'` and emits through the-openlit-go SDK, so the SDK discovery in `materialize.ts:discoverAgents`-picks the same `service.name` up unless explicitly excluded. The-summary table is a `ReplacingMergeTree(updated_at)`, so `FINAL` on-read picks whichever row was written most recently — and the detail-page chooses its layout from `agent.source`.--Fix: the SDK discovery query must filter out CLI rows by checking-`ResourceAttributes['coding_agent.session.id']` AND-`SpanAttributes['coding_agent.session.id']` are both empty.--When adding a new vendor, sanity-check that exactly one row exists-per agent in the summary table:--```sql-SELECT agent_key, source, count() AS rows-FROM openlit_agents_summary-WHERE service_name = '<vendor>'-GROUP BY agent_key, source;-```--A single row with `source='coding'` is correct. Two rows-(`coding` + anything else) means the exclusion isn't catching the-vendor's spans — usually because the vendor's CLI is emitting some-spans before the session id is stamped as a resource attribute.--## Claude Code: dual-path coalesce--Claude Code is the first vendor where users can send telemetry to-OpenLit through **two independent paths** at the same time:--1. **Hook path** (recommended primary). The `openlit-cc` plugin sits- in `~/.claude/plugins/openlit-cc/` and is invoked on every- `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`,- `Stop`, `SubagentStop`, and `SessionEnd` hook event. The adapter- in `cli/internal/coding/hook/claudecode/handle.go` normalises each- event into our canonical `coding_agent.*` / `gen_ai.*` schema. It- tails the transcript JSONL for authoritative token + cost numbers- at session end.-2. **Native OTel path** (optional, complementary). When the user- exports `CLAUDE_CODE_ENABLE_TELEMETRY=1` (plus the usual OTLP- endpoint / headers), Claude Code itself emits metrics, events,- and — with `CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1` — spans, all- under `service.name=claude-code` and the `claude_code.*`- attribute namespace.--The two paths overlap. OpenLit dedupes per `session.id` at read-time. The query layer (`queries.ts`, `materialize.ts`) coalesces both-schemas into one chat row:--- **Cost / tokens**: native wins when present (authoritative- Anthropic API numbers). Hook tail-of-transcript is the fallback- estimate.-- **Repository / working folder / branch**: hook wins. Claude Code's- native exporter does not see the user's `cwd` or run- `git.Snapshot(cwd)`; only the openlit hook does.-- **User identity**: either path. When the user is OAuth-authenticated- in Claude Code, the native path supplies `user.email` as a resource- attribute; the hook path supplies `gen_ai.user.name` from the local- env. Both are coalesced into the canonical user column.-- **Vendor identifier**: any of `coding_agent.client = 'claude-code'`- (stamped by hook), `gen_ai.agent.name = 'claude-code'`, or- `service.name = 'claude-code'` (native default). The materializer- recognises all three.--Every adapter stamps `coding_agent.signal_source = "hook"` on its-resource attributes. The native path leaves the attribute unset; the-query layer infers `"native"` from `service.name = 'claude-code'`. The-SDK discovery in `materialize.ts:discoverAgents` excludes any span-that carries `coding_agent.session.id` AND `telemetry.sdk.name =-'openlit'`, so the openlit hook spans are never double-materialized-as SDK rows. Native Claude Code spans use Claude Code's own SDK name-(not `openlit`), so they too are excluded from SDK discovery.--### Native Claude Code metric mapping--Claude Code's native exporter emits a stable set of metrics under the-`claude_code.*` namespace. The openlit CLI emits the same observable-behavior under the canonical `coding_agent.*` namespace so a single-OpenLit installation can ingest both streams interchangeably and any-custom Grafana / Mimir dashboard the operator already has continues-to work. The mapping:--| Native Claude Code metric | Canonical openlit-CLI metric | Notes |-|---|---|---|-| `claude_code.session.count` | `coding_agent.session.count` | Counter; vendor-tagged via `coding_agent.client`. Native path is unitless; openlit-CLI counter is `{session}`. |-| `claude_code.code_edit_tool.decision` | `coding_agent.edit.decision.count` | Counter, tags include `coding_agent.edit.decision` (`accept` / `auto_accepted` / `reject`) and `coding_agent.edit.source`. |-| `claude_code.lines_of_code.count` | `coding_agent.lines_of_code.count` | Counter, tags include `coding_agent.edit.decision` so the same metric powers "lines added" and "lines accepted vs rejected" without two emitters. |-| `claude_code.commit.count` | `coding_agent.commit.count` | Counter; the openlit-CLI adapters detect commits from Bash / shell / local_shell hooks and stamp the same canonical tags. |-| `claude_code.pull_request.count` | `coding_agent.pull_request.count` | Counter; gated on `gh pr create / gh pr new` detection so it survives copy-pasted aliases. |-| `claude_code.token.usage` | `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | OpenLit follows the OTel GenAI semantic-conventions for token usage. Tags: `gen_ai.system`, `gen_ai.request.model`. |-| `claude_code.cost.usage` | `gen_ai.usage.cost` | OTel GenAI semantic-conventions counter, USD. |-| `claude_code.active_time.total` | (derived) | Active time is rolled up at query time as `duration_ms = ended_at - started_at` on the session-root span; we deliberately don't emit it as a separate counter because it is recoverable from the span timings. |--All canonical counters are emitted via the openlit-go MeterProvider-in `cli/internal/otlp/metrics.go` and pass through the same OTLP-exporter as the spans, so a Prometheus / Mimir / Datadog backend-attached to the OpenLit collector sees the same data as the trace-backend.--**Does coalesce-at-read scale?** Yes for current OpenLit volumes-(under ~1M coding-agent spans per tenant per day). ClickHouse-evaluates the per-row coalesce in a few milliseconds; the query-bottleneck is the wider session scan, not the attribute lookup. Above-that volume we'll ship a `coding_agent_sessions` materialized view-that normalises both schemas at ingest. That deferral is documented-inline in `queries.ts` next to the `E1 deferral note`.--## Adding a new vendor--The Cursor, Claude Code, and Codex adapters are the three reference-implementations. Each one models a different vendor shape:--- **Cursor** — IDE extension; 13+ named hook events, no per-session- transcript file, prompt/response/thought arrive as separate hooks.-- **Claude Code** — CLI / IDE plugin; richer hook events plus a- per-session JSONL transcript that's the authoritative source for- per-turn LLM content + token usage. The adapter streams the- transcript with byte-offset bookkeeping.-- **Codex** — CLI; per-turn hook protocol with `turn_id` on every- event and a separate rollout JSONL under- `~/.codex/sessions/YYYY/MM/DD/` that carries the `token_count`- events. The adapter accumulates a per-turn fragment in- sessionstate and drains it on `Stop`.--To add another vendor (Windsurf / OpenCode / …):--1. **Read the convention rule** — `.cursor/rules/coding-agents-convention.mdc`.- It defines the strict attribute matrix every adapter must follow.- New vendors must NOT introduce a new namespace for facts that- already have a canonical key.-2. **Create the adapter package** under `cli/internal/coding/hook/<vendor>/`.- Map the vendor's hook payload keys onto the canonical `coding_agent.*`- / `gen_ai.*` schema. Use `setStr(..., scrub)` for any string —- never bypass the redaction layer.-3. **Extend `peekContext`** in `hook.go` so the vendor's session id,- user, cwd, permission mode, model, and parent-conversation id- resolve from the payload. The function is intentionally a thin- string-pick — add new key names to the `pickString` calls.-4. **Wire the plugin manifest** under `plugins/<vendor>/`. Mirror- `plugins/claude-code/` for Anthropic-style hooks or `plugins/cursor/`- for IDE-extension-style hooks. Add a marketplace entry in- `plugins/.claude-plugin/marketplace.json`.-5. **Bound transcript reads** with `tailfile.Tail` if the vendor- exposes a per-session log file. Never read an unbounded file in a- hook process.-6. **Stamp `gen_ai.system`** correctly: `anthropic` for Claude Code,- `openai` for Codex, `google` for Gemini-backed agents.- Use the `inferProvider(model, vendor)` helper.-7. **Honor capture modes** — drop bodies in `minimal` and- `metadata_only`. The `bodyAllowed(mode)` and `setStr(..., scrub)`- helpers in `cli/internal/otlp/attrs.go` are mandatory.-8. **Add UI icon + label**: register an inline SVG in- `src/client/src/components/svg/coding-agents.tsx` and add a switch- case in `CodingAgentVendorIcon` AND `hasCodingAgentVendorIcon`.- The existing call sites pick the icon up automatically once the- helpers know about the vendor.-9. **Add the onboarding snippet** to `docs/features/coding-agents/onboarding.mdx`.- Document any vendor-specific env vars (`OTEL_*` overrides,- transcript paths, dual-path notes if applicable).-10. **Sanity-check end to end**:- - Fire a real session through the vendor.- - Confirm `otel_traces` has spans with- `SpanAttributes['coding_agent.client'] = '<vendor>'`.- - Trigger the materializer- (`curl -X POST http://localhost:3000/api/agents/materialize -H 'X-CRON-JOB: true'`).- - Confirm a row lands in `openlit_agents_summary` with- `coding_agent_vendor = '<vendor>'` and `source = 'coding'`.- - Open `/agents`, click into the row, verify the Sessions and- Users tabs render without errors and that the trace-detail- pills (Repository, Working Folder, Branch, Mode, Terminal)- populate.--## Threat model & accepted risks--The CLI runs **on the developer's machine** alongside the coding-agent. Anyone who can run `openlit coding hook` on that machine can-emit arbitrary `coding_agent.*` spans into the configured collector-— including spans claiming to be from a different session, vendor,-or user. We accept this:--- **Spoofing scope is bounded by who already has shell access.** A- developer who can run the hook can equally well run any other- process under their own user. There is no privilege boundary the- CLI is the only gate on.-- **The OTLP endpoint authenticates the host, not the event.** The- Authorization header carries an org-scoped API key, so spans- reaching the backend are at least "from some machine our org- authorised". Per-event signing is intentionally out of scope for- v1 — the cost (key distribution, rotation, hook-latency budget)- outweighs the benefit for the threat model of in-org telemetry.-- **Materialization runs k=5 cohort floors** so a single user- cannot inflate their own visible metrics by spamming events; the- per-user page refuses to render below five sessions.-- **Repos and API keys drive the personal-vs-work classifier**, not- hook content. A user cannot relabel their own `personal` activity- as `work` by tampering with hook payloads.--Operators that need stricter guarantees (e.g. an untrusted developer-fleet) should:--- Front the OTLP endpoint with a collector that drops events- whose `coding_agent.client.user.id` doesn't match the- authenticated key's owner.-- Disable the hook tier and rely on vendor-native OTel only- (Claude Code's `CLAUDE_CODE_ENABLE_TELEMETRY=1` path) where the- vendor signs its own egress.--## Where to read next--- `.cursor/rules/coding-agents-convention.mdc` — the strict schema- contract every vendor must follow. The attribute matrix, the- Cursor + Claude Code inbound mappings, and the dual-path coalesce- rules all live here.-- `.cursor/rules/coding-agents-hook.mdc` — canonical authoring rules- + failure modes, auto-loaded by Cursor when editing hook code.-- `cli/internal/coding/hook/cursor/handle.go` — reference adapter.-- `cli/internal/coding/hook/claudecode/handle.go` — second reference- adapter, demonstrates dual-path-aware design.-- `cli/internal/coding/normalize/normalize.go` — schema all adapters- emit into.-- `cli/internal/otlp/attrs.go` — capture-mode gating and scrubbing.-- `src/client/src/lib/platform/agents/materialize.ts` — rollup that- populates `openlit_agents_summary`.-- `src/client/src/lib/platform/coding-agents/queries.ts` — Sessions,- Users, Overview query layer.diff --git a/docs/latest/openlit/coding-agents/onboarding.mdx b/docs/latest/openlit/coding-agents/onboarding.mdxdeleted file mode 100644index f75e3416..00000000--- a/docs/latest/openlit/coding-agents/onboarding.mdx+++ /dev/null@@ -1,319 +0,0 @@-----title: "Coding Agents — Onboarding"-sidebarTitle: "Coding Agents Onboarding"-description: "Send Claude Code, Cursor, and Codex telemetry into OpenLit in under a minute."------OpenLit ships first-class observability for AI coding agents. The-`openlit` CLI ingests events from each vendor's hook system, normalizes-them onto OpenTelemetry's `gen_ai.*` and OpenLit's `coding_agent.*`-semantic conventions, and ships them to your existing OpenLit-collector. The `/agents` page detects them automatically — no-controller, no SDK, no code change in your repos.--<Note>-You only need an OpenLit instance and the `openlit` CLI on each-developer's machine. The CLI talks **only** to your OpenLit endpoint.-There is no managed cloud for coding-agent telemetry.-</Note>--## Prerequisites--- A running OpenLit instance — see- [Installation](/latest/openlit/installation) if you don't have one.-- An OpenLit API key (Settings → API Keys).-- The OTLP endpoint of your OpenLit collector (defaults to- `http://localhost:4318` for local installs).--## 1. Install the CLI--Pick the path that matches your OS. All paths install the same-`openlit` binary built by `cli-release.yml`.--<CodeGroup>-```bash macOS-brew install openlit/openlit/openlit-```--```bash Linux-curl -fsSL https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.sh | sh-```--```powershell Windows-iwr -useb https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.ps1 | iex-```--```bash docker-docker run --rm -it ghcr.io/openlit/openlit-cli:latest --help-```-</CodeGroup>--The Linux and Windows scripts install to `~/.openlit/bin/openlit` and-`%USERPROFILE%\.openlit\bin\openlit.exe` respectively. The Linux-script prints a PATH-add hint if the directory is not already on-`$PATH`; the Windows script updates user-scope PATH automatically-(open a new terminal to pick it up).--If you prefer to build from source: `go install github.com/openlit/openlit/cli/cmd/openlit@latest`.--## 2. Configure once--The fastest path is `openlit configure` — it answers the same-questions a CI / Helm chart would set via env. The result is written-to `~/.config/openlit/config.env` (Linux/macOS) or-`%APPDATA%\openlit\config.env` (Windows). For headless / fleet-rollouts, prefer env vars or flags directly — they take precedence-over the file (flags > env > config file).--```bash-openlit configure \- --endpoint https://openlit.example.com:4318 \- --api-key $OPENLIT_API_KEY-```--Equivalent environment variables (precedence: flags > env > config file):--| Variable | Purpose |-|---|---|-| `OPENLIT_OTLP_ENDPOINT` | OTLP/HTTP endpoint of your collector |-| `OPENLIT_API_KEY` | Authenticates the OTLP exporter |-| `OPENLIT_USER` | Optional override for `gen_ai.user.name` (email or login) |-| `OPENLIT_CODING_CONTENT_CAPTURE` | `minimal` \| `metadata_only` \| `full` *(default: `full`)* |--Standard `OTEL_EXPORTER_OTLP_*` variables are honoured as fallbacks-so existing OTel users can reuse their setup.--<Tip>-`OPENLIT_CODING_CONTENT_CAPTURE` defaults to `full` so the trace-detail view is useful on first use — prompt bodies, agent replies,-and tool I/O are stamped onto every per-event span, scrubbed by the-tier-1 secret redactor on the way out. Switch to `metadata_only`-when rolling out across a team where prompts may carry confidential-material (the trace-detail page will surface a note on every span-explaining how). Use `minimal` if you only need cost + activity-dashboards and not the per-event timeline.-</Tip>--## 3. Wire each vendor--`openlit coding install` writes the right hook config for the agent-you pass and is fully idempotent. You can also point at the-[plugin marketplace](https://github.com/openlit/openlit/tree/main/plugins)-and let the agent pull the manifest itself, or invoke the agent-through `openlit coding launch` which auto-installs and execs.--<Tabs>-<Tab title="Claude Code">--```bash-# One-time install-openlit coding install --vendor=claude-code--# Or run Claude Code through the launcher (auto-installs)-openlit coding launch claude-code-```--Manifest path: `~/.claude/plugins/openlit-cc/`. Hooks wired:-`SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`,-`Stop`, `SubagentStop`, `SessionEnd`. Claude Code's transcript JSONL-(`transcript_path` in the hook payload) is tailed on `SessionStart`-for early model attribution and on `SessionEnd` for authoritative-token usage and cost.--**Optional: Claude Code's native OpenTelemetry exporter.** If you'd-rather not install the plugin, or you want a second signal source-for cross-checking, you can point Claude Code's built-in OTel-exporter at your OpenLit collector instead. Both paths can run side-by side — OpenLit dedupes per `session.id` (Claude Code's authoritative-cost / tokens from the native path win, while the hook path provides-the repository / working folder context that Claude Code itself-can't see).--```bash-export CLAUDE_CODE_ENABLE_TELEMETRY=1-export OTEL_METRICS_EXPORTER=otlp-export OTEL_LOGS_EXPORTER=otlp-export OTEL_EXPORTER_OTLP_PROTOCOL=grpc-export OTEL_EXPORTER_OTLP_ENDPOINT=$OPENLIT_OTLP_ENDPOINT-export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OPENLIT_API_KEY"-# Optional: also enable traces (beta — gives per-tool / per-LLM-turn spans)-export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1-export OTEL_TRACES_EXPORTER=otlp-```--See [Claude Code's monitoring docs](https://code.claude.com/docs/en/monitoring-usage)-for the full environment-variable surface (managed settings, dynamic-headers, cardinality controls, content-capture gates). When both-paths are active OpenLit folds them into one chat thread; see the-"Dual-path coalesce" section of the [architecture guide](./architecture)-for the details.--</Tab>-<Tab title="Cursor">--```bash-openlit coding install --vendor=cursor-```--Manifest path: `~/.cursor/hooks.json` (user scope — applies to every-workspace you open in Cursor; works the same on macOS, Linux, and-Windows). All Cursor hook events (sessionStart/End, prompt /-response / thought, tool + shell + MCP phases, subagent lifecycle,-file edits, preCompact) invoke-`openlit coding hook --vendor=cursor --event=<name>` directly.-At install time the `openlit` token in each `command` is rewritten-to the absolute path of the CLI binary that ran-`openlit coding install`, which sidesteps the minimal-PATH that-GUI-launched Cursor windows inherit on macOS and Linux, and makes-the same manifest work on Windows where there is no shell wrapper.--<Note>-We merge into `~/.cursor/hooks.json` (user scope) rather than drop-a plugin tree elsewhere because Cursor only auto-discovers plugins-that were registered through its in-app `/add-plugin` flow.-User-scope hooks are the supported way to install agent-wide hooks-without an in-app step, and re-running-`openlit coding install --vendor=cursor` is idempotent: prior-openlit entries are dropped before the new ones are appended, and-any third-party entries you (or another tool) have added are left-in place.-</Note>--</Tab>-<Tab title="Codex">--```bash-openlit coding install --vendor=codex-```--Manifest path: `~/.codex/plugins/openlit/`. Hooks wired:-`SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`,-`Stop`. Codex scopes every hook to a `turn_id` so we emit one-`coding_agent.llm.turn` span per turn (built on `Stop`), with the-prompt, last-assistant message, and the per-tool calls produced-during that turn folded into the OTel-canonical-`gen_ai.input.messages` / `gen_ai.output.messages` envelopes.--The CLI tails `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` on-`Stop` to compute the per-turn token delta (input, output, cached,-reasoning) from Codex's running `token_count` events, so cost-rollups in OpenLit reflect the same per-turn deltas Codex itself-records.--Subagent linkage is read from the transcript's `session_meta`-record at `SessionStart`; child turns inherit-`coding_agent.agent.parent_id` automatically so the UI folds them-under the spawning chat. Codex doesn't expose a `SessionEnd` event,-so the session row updates incrementally as each `Stop` lands.--If `codex features list` doesn't already enable `codex_hooks` on-your build, add this to `~/.codex/config.toml`:--```toml-[features]-codex_hooks = true--[plugins."openlit"]-enabled = true-```--Older Codex builds use `hooks = true` and `plugin_hooks = true`-instead of `codex_hooks`.--</Tab>-</Tabs>--## 4. Verify--Run any short coding-agent session, then refresh `/agents` in your-OpenLit UI. You should see a new row labelled with the vendor and a-"Coding" badge. Click in for the dedicated Overview / Sessions /-Dashboard tabs.--## What gets captured--| Surface | What you get out of the box |-|---|---|-| `/agents` row | Per-vendor row with sessions, active users, cost, lines added/removed, acceptance %, commits and PRs — all evaluated over the time range selected in the global filter picker (the materialized rollups are overlaid with a live recompute for any custom window). |-| Overview tab | The seeded "Coding Agents" dashboard, scoped to this vendor, honoring the picker. |-| Sessions tab | `coding_agent.session.id` rows within the selected window — outcome, cost, classification, per-session lines (+/−), acceptance %, commits, PRs. |-| Dashboard tab | Seeded board (vendor breakdown, top tools, top users, top repos, classification, MCP servers, edit decisions, lines accepted, acceptance %, commits, PRs, lines-over-time area, top users by accepted, acceptance breakdown pie). Every widget reads `{{filter.timeLimit.start}}` / `{{filter.timeLimit.end}}`. |-| Per-user page | Code-impact stat row (lines added, lines accepted, acceptance %, commits, PRs) above the user-scoped dashboard. The stat row re-fetches when the picker changes, so the row and the widgets below it stay in lockstep. |-| Custom dashboards | Same data — query `otel_traces WHERE SpanName IN ('coding_agent.session', 'coding_agent.edit.decision', 'coding_agent.git.commit', 'coding_agent.git.pull_request', …)` |-| Metrics pipeline | `coding_agent.session.count`, `coding_agent.session.duration`, `coding_agent.tool.call.count`, `coding_agent.lines_of_code.count`, `coding_agent.edit.decision.count`, `coding_agent.commit.count`, `coding_agent.pull_request.count` counters emitted via the openlit-go MeterProvider |--## Privacy & governance--- **Tier-1 redaction** runs on every export and scrubs API keys,- tokens, and known secret patterns without breaking JSON structure.-- **Personal vs Work classification** is stamped at hook time using- the org's API-key and repo-origin allowlists (configured in OpenLit- Settings). Disputes can be filed via- `POST /api/coding-agents/classification/dispute`; surfacing a- dispute button on the Sessions tab is planned for the next release.-- **k=5 cohort floor**: viewer-tier accounts cannot see per-user- attribution below five sessions. Admins can.--## Stop tracking / uninstall--The inverse of `openlit coding install`. Removes the per-vendor host-plugin manifests written by install, deregisters the plugin from the-vendor's own CLI where applicable (Claude Code, Codex), and leaves-your shared config alone by default so you can re-onboard without-re-entering credentials.--```bash-openlit coding uninstall --vendor=cursor # one vendor-openlit coding uninstall --vendor=all # every vendor at once-openlit coding uninstall --vendor=all --purge # also drop ~/.config/openlit + session-state cache-openlit coding uninstall --vendor=cursor --dry-run # preview without touching disk-```--What each flag removes:--| Path | Removed by `--vendor=<v>` | Removed by `--purge` |-|---|---|---|-| `~/.claude/plugins/openlit-cc/` | ✓ (when `<v>` is `claude-code` or `all`) | — |-| `~/.local/share/openlit/claude-marketplace/` | ✓ (when `<v>` is `claude-code` or `all`) | — |-| openlit entries in `~/.cursor/hooks.json` (other tools' entries preserved; file deleted if it ends up empty) | ✓ (when `<v>` is `cursor` or `all`) | — |-| `~/.local/share/openlit/codex-marketplace/` | ✓ (when `<v>` is `codex` or `all`) | — |-| `claude plugin uninstall openlit-cc@openlit` | ✓ best-effort | — |-| `codex plugin remove openlit@openlit` + `codex plugin marketplace remove openlit` | ✓ best-effort | — |-| `~/.config/openlit/` (your `configure` output) | — | ✓ |-| `<UserCacheDir>/openlit/sessions/` (session-state cache) | — | ✓ |--The `openlit` binary itself is **not** touched. Uninstall it the way-you installed it:--```bash-brew uninstall openlit # Homebrew-rm ~/.openlit/bin/openlit # curl|sh installer-rm $(go env GOPATH)/bin/openlit # go install-```--## Troubleshooting--- **First stop** — run `openlit doctor`. It prints the resolved- config, the cached session count, the installed plugin paths, and- TCP-dials your OTLP endpoint. Exit code 1 means at least one- failure; 0 (possibly with warnings) means the CLI is wired up.-- **No row on /agents** — the hook never writes to a log file; it- emits to stderr, which is captured by the host coding agent's- own logs (Claude Code: `~/.claude/logs/`; Cursor: Developer- Tools → Console; Codex: `~/.codex/`). Set- `OPENLIT_DEBUG_PAYLOAD_DIR=/tmp/openlit-debug` to capture the- exact JSON the vendor is sending the hook — useful when fields- appear to be missing on the resulting spans.-- **Hooks ran but no spans** — confirm `openlit configure --show`- reports the right endpoint. The CLI never reads `~/.zshrc`; if you- set OPENLIT_* in your shell rc, source it before launching the agent.-- **"Database config not found" on /agents** — your OpenLit- installation hasn't run the latest migrations. Restart the- `openlit` container or run the migration manually- (`pnpm --filter client run migrate`).--## What's next--- The CLI reserves subcommands for `prompts`, `traces`, and `eval`;- these will light up in subsequent releases without breaking the- coding-agent integration.diff --git a/docs/latest/openlit/coding-agents/overview.mdx b/docs/latest/openlit/coding-agents/overview.mdxnew file mode 100644index 00000000..8ed268c3--- /dev/null+++ b/docs/latest/openlit/coding-agents/overview.mdx@@ -0,0 +1,39 @@+---+title: "Overview"+description: "OpenLIT ships first-class observability for AI coding agents - Claude Code, Cursor, Codex, and Windsurf - with no SDK and no code changes"+---++OpenLIT ships first-class observability for AI coding agents. The `openlit` CLI ingests events from each vendor's hook system, normalizes them onto OpenTelemetry's `gen_ai.*` and OpenLIT's `coding_agent.*` semantic conventions, and ships them to your existing OpenLIT collector. The [Agents page](/latest/openlit/observability/agents/overview)'s Coding Agents tab detects them automatically - no controller, no SDK, no code change in your repos.++<Note>+You only need an OpenLIT instance and the `openlit` CLI on each developer's machine. The CLI talks **only** to your OpenLIT endpoint. There is no managed cloud for coding-agent telemetry.+</Note>++## What you get++<CardGroup cols={2}>+ <Card title="Setup & Configure" href="/latest/openlit/coding-agents/setup-and-configure" icon='download'>+ Install the CLI, wire Claude Code, Cursor, or Codex, and stop tracking when you need to+ </Card>+ <Card title="Analytics" href="/latest/openlit/coding-agents/analytics" icon='chart-line'>+ The seeded dashboard - sessions, cost, acceptance, top tools and repos, session outcomes+ </Card>+ <Card title="Sessions" href="/latest/openlit/coding-agents/sessions" icon='list'>+ Every coding session, with a full conversational drill-in per session+ </Card>+ <Card title="Users" href="/latest/openlit/coding-agents/users" icon='users'>+ Per-developer rollups of activity, cost, and code impact+ </Card>+ <Card title="Privacy & Governance" href="/latest/openlit/coding-agents/privacy-and-governance" icon='shield'>+ Content capture modes, redaction tiers, and attribution safeguards+ </Card>+ <Card title="Troubleshooting" href="/latest/openlit/coding-agents/troubleshooting" icon='wrench'>+ Diagnose a missing row, silent hooks, or a stale database config+ </Card>+</CardGroup>++## Prerequisites++- A running OpenLIT instance - see [Installation](/latest/openlit/installation) if you don't have one.+- An OpenLIT API key (Settings → API Keys).+- The OTLP endpoint of your OpenLIT collector (defaults to `http://localhost:4318` for local installs).diff --git a/docs/latest/openlit/coding-agents/privacy-and-governance.mdx b/docs/latest/openlit/coding-agents/privacy-and-governance.mdxnew file mode 100644index 00000000..e6d42a2d--- /dev/null+++ b/docs/latest/openlit/coding-agents/privacy-and-governance.mdx@@ -0,0 +1,41 @@+---+title: "Privacy & Governance"+description: "Content capture modes, redaction, and attribution safeguards for coding-agent telemetry"+---++Coding-agent sessions can carry sensitive material - prompts, file contents, proprietary code. OpenLIT gives you control over how much of that gets captured, and applies redaction regardless of the mode you choose.++## Content capture modes++`OPENLIT_CODING_CONTENT_CAPTURE` controls what lands on spans:++| Mode | Identifiers | Tool names + paths | File diffs / message bodies |+|---|---|---|---|+| `minimal` | ✅ | ❌ | ❌ |+| `metadata_only` | ✅ | ✅ | ❌ |+| `full` (default) | ✅ | ✅ | ✅ |++Set it once via `openlit configure` or the `OPENLIT_CODING_CONTENT_CAPTURE` environment variable - see [Setup & Configure](/latest/openlit/coding-agents/setup-and-configure). `full` is the default so the trace detail view is useful out of the box; switch to `metadata_only` when rolling out across a team where prompts may carry confidential material, or `minimal` if you only need cost and activity dashboards and not the per-event timeline. Sessions captured under `minimal` or `metadata_only` show a banner in their [session detail view](/latest/openlit/coding-agents/sessions) explaining what wasn't recorded.++## Redaction++Two redaction tiers run on every export, regardless of capture mode:++- **Tier 1** - a token-pattern scrubber that always runs, stripping API keys, tokens, and known secret patterns without breaking JSON structure.+- **Tier 2** - a body-scope scrubber, active only in `full` mode, that applies the same scrubbing to file diffs and message bodies before they're exported.++The active capture mode is stamped as a resource attribute (`coding_agent.content_capture_mode`) on every span, so you can audit what a given session was recorded under after the fact.++## Attribution safeguards++Per-user attribution is capped by a cohort floor: viewer-tier accounts cannot see per-user metrics for a user with fewer than five sessions in the selected window. Admin-tier accounts see the full breakdown. This prevents singling out an individual developer from a small sample on the [Users](/latest/openlit/coding-agents/users) tab.++## Threat model++The `openlit` CLI runs on the developer's own machine, alongside the coding agent, and authenticates to your collector with an org-scoped API key - the header proves the event came from a machine your org authorized, not that a specific event wasn't spoofed by anyone with shell access on that machine. In practice this is the same trust boundary as any other locally-run developer tool: someone who can run the hook can equally well run any other process under their own user. If you need a stricter guarantee for an untrusted developer fleet, front the OTLP endpoint with a collector that validates the authenticated key's owner against the event's user attribute, or disable the hook tier entirely and rely on a vendor's native OTel exporter (where the vendor signs its own egress) instead.++---++<Card title="Troubleshooting" href="/latest/openlit/coding-agents/troubleshooting" icon='wrench'>+ Diagnose missing rows, silent hooks, or a stale database config+</Card>diff --git a/docs/latest/openlit/coding-agents/sessions.mdx b/docs/latest/openlit/coding-agents/sessions.mdxnew file mode 100644index 00000000..ad95838b--- /dev/null+++ b/docs/latest/openlit/coding-agents/sessions.mdx@@ -0,0 +1,26 @@+---+title: "Sessions"+description: "Every coding session for a vendor, with a full conversational drill-in per session"+---++The **Sessions** tab on a coding agent's detail page lists every individual coding session for that vendor, one row per `coding_agent.session.id` (folded up to its chat thread when subagents are linked in).++## Sessions list++Columns: Session, User, Started, Duration, Model, Tool calls, Code changed (`+added/-removed`, with accepted/rejected broken out in a tooltip), Acceptance %, Commits, Pull Requests, Tokens, Cost, and Outcome.++## Session detail++Clicking a session opens the same trace detail view used for [Traces](/latest/openlit/observability/telemetry/traces), rooted at that session. It defaults to the **Chat** view since a conversational read fits an IDE/CLI session better than a span tree - you'll see prompts, tool calls, and edits laid out like a transcript, with the [AI Analysis](/latest/openlit/observability/telemetry/ai-analysis), Timeline, and Graph views still available if you want a different angle.++The header adds session-specific meta pills on top of the usual Duration, Tokens, Cost, and Model stats: Coding Agent (vendor), User, Working Folder, Repository, Branch, Terminal, Commits, Lines added/deleted, Acceptance, Pull Requests, and Outcome. Use **Prev / Next** to step through sessions without returning to the list.++<Note>+If a session was captured under the `minimal` or `metadata_only` content-capture mode, the detail view shows a banner explaining that prompt bodies and tool I/O weren't recorded for that session - see [Privacy & Governance](/latest/openlit/coding-agents/privacy-and-governance).+</Note>++---++<Card title="Analytics" href="/latest/openlit/coding-agents/analytics" icon='chart-line'>+ Zoom back out to vendor-wide trends and breakdowns+</Card>diff --git a/docs/latest/openlit/coding-agents/setup-and-configure.mdx b/docs/latest/openlit/coding-agents/setup-and-configure.mdxnew file mode 100644index 00000000..47c650e9--- /dev/null+++ b/docs/latest/openlit/coding-agents/setup-and-configure.mdx@@ -0,0 +1,177 @@+---+title: "Setup & Configure"+description: "Install the openlit CLI, wire each vendor's hooks, then configure the endpoint, content capture, and stopping tracking"+---++## Setup++<Steps>+ <Step title="Install the CLI">+ Pick the path that matches your OS. All paths install the same `openlit` binary built by `cli-release.yml`.++ <CodeGroup>+ ```bash macOS+ brew install openlit/openlit/openlit+ ```++ ```bash Linux+ curl -fsSL https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.sh | sh+ ```++ ```powershell Windows+ iwr -useb https://raw.githubusercontent.com/openlit/openlit/main/cli/scripts/install.ps1 | iex+ ```++ ```bash docker+ docker run --rm -it ghcr.io/openlit/openlit-cli:latest --help+ ```+ </CodeGroup>++ The Linux and Windows scripts install to `~/.openlit/bin/openlit` and `%USERPROFILE%\.openlit\bin\openlit.exe` respectively. The Linux script prints a PATH-add hint if the directory is not already on `$PATH`; the Windows script updates user-scope PATH automatically (open a new terminal to pick it up).++ If you prefer to build from source: `go install github.com/openlit/openlit/cli/cmd/openlit@latest`.+ </Step>++ <Step title="Wire each vendor">+ `openlit coding install` writes the right hook config for the agent you pass and is fully idempotent. You can also point at the [plugin marketplace](https://github.com/openlit/openlit/tree/main/plugins) and let the agent pull the manifest itself, or invoke the agent through `openlit coding launch` which auto-installs and execs.++ <Tabs>+ <Tab title="Claude Code">++ ```bash+ # One-time install+ openlit coding install --vendor=claude-code++ # Or run Claude Code through the launcher (auto-installs)+ openlit coding launch claude-code+ ```++ Manifest path: `~/.claude/plugins/openlit-cc/`. Hooks wired: `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `Stop`, `SubagentStop`, `SessionEnd`. Claude Code's transcript JSONL (`transcript_path` in the hook payload) is tailed on `SessionStart` for early model attribution and on `SessionEnd` for authoritative token usage and cost.++ **Optional: Claude Code's native OpenTelemetry exporter.** If you'd rather not install the plugin, or you want a second signal source for cross-checking, you can point Claude Code's built-in OTel exporter at your OpenLIT collector instead. Both paths can run side by side - OpenLIT dedupes per `session.id` (Claude Code's authoritative cost/tokens from the native path win, while the hook path provides the repository/working-folder context that Claude Code itself can't see).++ ```bash+ export CLAUDE_CODE_ENABLE_TELEMETRY=1+ export OTEL_METRICS_EXPORTER=otlp+ export OTEL_LOGS_EXPORTER=otlp+ export OTEL_EXPORTER_OTLP_PROTOCOL=grpc+ export OTEL_EXPORTER_OTLP_ENDPOINT=$OPENLIT_OTLP_ENDPOINT+ export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $OPENLIT_API_KEY"+ # Optional: also enable traces (beta - gives per-tool / per-LLM-turn spans)+ export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1+ export OTEL_TRACES_EXPORTER=otlp+ ```++ See [Claude Code's monitoring docs](https://code.claude.com/docs/en/monitoring-usage) for the full environment-variable surface (managed settings, dynamic headers, cardinality controls, content-capture gates). When both paths are active, OpenLIT folds them into one chat thread automatically.++ </Tab>+ <Tab title="Cursor">++ ```bash+ openlit coding install --vendor=cursor+ ```++ Manifest path: `~/.cursor/hooks.json` (user scope - applies to every workspace you open in Cursor; works the same on macOS, Linux, and Windows). All Cursor hook events (sessionStart/End, prompt/response/thought, tool + shell + MCP phases, subagent lifecycle, file edits, preCompact) invoke `openlit coding hook --vendor=cursor --event=<name>` directly. At install time the `openlit` token in each `command` is rewritten to the absolute path of the CLI binary that ran `openlit coding install`, which sidesteps the minimal-PATH that GUI-launched Cursor windows inherit on macOS and Linux, and makes the same manifest work on Windows where there is no shell wrapper.++ <Note>+ We merge into `~/.cursor/hooks.json` (user scope) rather than drop a plugin tree elsewhere because Cursor only auto-discovers plugins that were registered through its in-app `/add-plugin` flow. User-scope hooks are the supported way to install agent-wide hooks without an in-app step, and re-running `openlit coding install --vendor=cursor` is idempotent: prior openlit entries are dropped before the new ones are appended, and any third-party entries you (or another tool) have added are left in place.+ </Note>++ </Tab>+ <Tab title="Codex">++ ```bash+ openlit coding install --vendor=codex+ ```++ Manifest path: `~/.codex/plugins/openlit/`. Hooks wired: `SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `Stop`. Codex scopes every hook to a `turn_id` so we emit one `coding_agent.llm.turn` span per turn (built on `Stop`), with the prompt, last-assistant message, and the per-tool calls produced during that turn folded into the OTel-canonical `gen_ai.input.messages` / `gen_ai.output.messages` envelopes.++ The CLI tails `~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl` on `Stop` to compute the per-turn token delta (input, output, cached, reasoning) from Codex's running `token_count` events, so cost rollups in OpenLIT reflect the same per-turn deltas Codex itself records.++ Subagent linkage is read from the transcript's `session_meta` record at `SessionStart`; child turns inherit `coding_agent.agent.parent_id` automatically so the UI folds them under the spawning chat. Codex doesn't expose a `SessionEnd` event, so the session row updates incrementally as each `Stop` lands.++ If `codex features list` doesn't already enable `codex_hooks` on your build, add this to `~/.codex/config.toml`:++ ```toml+ [features]+ codex_hooks = true++ [plugins."openlit"]+ enabled = true+ ```++ Older Codex builds use `hooks = true` and `plugin_hooks = true` instead of `codex_hooks`.++ </Tab>+ </Tabs>+ </Step>++ <Step title="Verify">+ Run any short coding-agent session, then refresh the [Agents page](/latest/openlit/observability/agents/overview) in your OpenLIT UI. You should see a new row labelled with the vendor and a "Coding" badge. Click in for the dedicated Overview / Sessions / Users tabs.+ </Step>+</Steps>++## Configure++Everything below is a setting you can change at any point after setup - none of it needs to happen up front.++### Endpoint, API key, and content capture++The fastest path is `openlit configure` - it answers the same questions a CI / Helm chart would set via env. The result is written to `~/.config/openlit/config.env` (Linux/macOS) or `%APPDATA%\openlit\config.env` (Windows). For headless / fleet rollouts, prefer env vars or flags directly - they take precedence over the file (flags > env > config file).++```bash+openlit configure \+ --endpoint https://openlit.example.com:4318 \+ --api-key $OPENLIT_API_KEY+```++Equivalent environment variables (precedence: flags > env > config file):++| Variable | Purpose |+|---|---|+| `OPENLIT_OTLP_ENDPOINT` | OTLP/HTTP endpoint of your collector |+| `OPENLIT_API_KEY` | Authenticates the OTLP exporter |+| `OPENLIT_USER` | Optional override for `gen_ai.user.name` (email or login) |+| `OPENLIT_CODING_CONTENT_CAPTURE` | `minimal` \| `metadata_only` \| `full` *(default: `full`)* - see [Privacy & Governance](/latest/openlit/coding-agents/privacy-and-governance) |++Standard `OTEL_EXPORTER_OTLP_*` variables are honoured as fallbacks so existing OTel users can reuse their setup. Re-run `openlit configure` (or update the env vars) any time - changes apply to the next session, no reinstall of the vendor hooks needed.++### Stop tracking / uninstall++The inverse of `openlit coding install`. Removes the per-vendor host plugin manifests written by install, deregisters the plugin from the vendor's own CLI where applicable (Claude Code, Codex), and leaves your shared config alone by default so you can re-onboard without re-entering credentials.++```bash+openlit coding uninstall --vendor=cursor # one vendor+openlit coding uninstall --vendor=all # every vendor at once+openlit coding uninstall --vendor=all --purge # also drop ~/.config/openlit + session-state cache+openlit coding uninstall --vendor=cursor --dry-run # preview without touching disk+```++What each flag removes:++| Path | Removed by `--vendor=<v>` | Removed by `--purge` |+|---|---|---|+| `~/.claude/plugins/openlit-cc/` | ✓ (when `<v>` is `claude-code` or `all`) | - |+| `~/.local/share/openlit/claude-marketplace/` | ✓ (when `<v>` is `claude-code` or `all`) | - |+| openlit entries in `~/.cursor/hooks.json` (other tools' entries preserved; file deleted if it ends up empty) | ✓ (when `<v>` is `cursor` or `all`) | - |+| `~/.local/share/openlit/codex-marketplace/` | ✓ (when `<v>` is `codex` or `all`) | - |+| `claude plugin uninstall openlit-cc@openlit` | ✓ best-effort | - |+| `codex plugin remove openlit@openlit` + `codex plugin marketplace remove openlit` | ✓ best-effort | - |+| `~/.config/openlit/` (your `configure` output) | - | ✓ |+| `<UserCacheDir>/openlit/sessions/` (session-state cache) | - | ✓ |++The `openlit` binary itself is **not** touched. Uninstall it the way you installed it:++```bash+brew uninstall openlit # Homebrew+rm ~/.openlit/bin/openlit # curl|sh installer+rm $(go env GOPATH)/bin/openlit # go install+```++If you only want to pause a vendor temporarily rather than uninstall it, re-run `openlit coding install --vendor=<v>` later - it's idempotent and will re-wire the same manifest.++---++<Card title="Troubleshooting" href="/latest/openlit/coding-agents/troubleshooting" icon='wrench'>+ No row showing up, or hooks running with no spans landing? Start here.+</Card>diff --git a/docs/latest/openlit/coding-agents/troubleshooting.mdx b/docs/latest/openlit/coding-agents/troubleshooting.mdxnew file mode 100644index 00000000..638b41b8--- /dev/null+++ b/docs/latest/openlit/coding-agents/troubleshooting.mdx@@ -0,0 +1,26 @@+---+title: "Troubleshooting"+description: "Diagnose a missing row, silent hooks, or a stale database config for coding-agent telemetry"+---++## First stop: run `openlit doctor`++It prints the resolved config, the cached session count, the installed plugin paths, and TCP-dials your OTLP endpoint. Exit code `1` means at least one failure; `0` (possibly with warnings) means the CLI is wired up correctly.++## No row on the Agents page++The hook never writes to a log file - it emits to stderr, which is captured by the host coding agent's own logs (Claude Code: `~/.claude/logs/`; Cursor: Developer Tools → Console; Codex: `~/.codex/`). Set `OPENLIT_DEBUG_PAYLOAD_DIR=/tmp/openlit-debug` to capture the exact JSON the vendor is sending the hook - useful when fields appear to be missing on the resulting spans.++## Hooks ran but no spans landed++Confirm `openlit configure --show` reports the right endpoint. The CLI never reads `~/.zshrc`; if you set `OPENLIT_*` in your shell rc, source it before launching the agent.++## "Database config not found" on the Agents page++Your OpenLIT installation hasn't run the latest migrations. Restart the `openlit` container or run the migration manually (`pnpm --filter client run migrate`).++---++<Card title="Setup & Configure" href="/latest/openlit/coding-agents/setup-and-configure" icon='download'>+ Re-check install, wiring, and endpoint configuration+</Card>diff --git a/docs/latest/openlit/coding-agents/users.mdx b/docs/latest/openlit/coding-agents/users.mdxnew file mode 100644index 00000000..39b8a815--- /dev/null+++ b/docs/latest/openlit/coding-agents/users.mdx@@ -0,0 +1,18 @@+---+title: "Users"+description: "Per-developer rollups of coding-agent activity, cost, and code impact"+---++The **Users** tab on a coding agent's detail page lists every developer who has used that vendor, with per-user rollups: Sessions, Tool calls, Cost, Tokens, Lines added (accepted/rejected in a tooltip), Acceptance %, and Commits.++Sort by Last seen, Sessions, Tool calls, Cost, or Tokens. Clicking a user opens the same [Analytics](/latest/openlit/coding-agents/analytics) dashboard used at the vendor level, scoped down to that one developer - so you get the identical stat cards, trends, and breakdowns, just filtered to their activity.++<Note>+Per-user attribution below a five-session cohort floor is hidden from viewer-tier accounts to avoid singling out low-activity individuals from small samples; admin-tier accounts see the full breakdown.+</Note>++---++<Card title="Sessions" href="/latest/openlit/coding-agents/sessions" icon='list'>+ See the individual sessions behind any user's rollup+</Card>diff --git a/docs/latest/openlit/configuration.mdx b/docs/latest/openlit/configuration.mdxindex 580c7c6b..0f8484d2 100644--- a/docs/latest/openlit/configuration.mdx+++ b/docs/latest/openlit/configuration.mdx@@ -124,10 +124,141 @@ For detailed OAuth setup instructions, see the [OAuth Authentication Setup](/lat```</ResponseField>+## Server variables++<ResponseField name="PORT">+ Sets the port the OpenLIT server listens on. For **Docker Compose** deployments, set this in the `.env` file next to `docker-compose.yml` - Compose maps it to both the host port and the container's internal `DOCKER_PORT` for you. For **Kubernetes** or a raw container run, set `DOCKER_PORT` directly instead (see below); the container's entrypoint always derives its actual listening port from `DOCKER_PORT`, defaulting to `3000` if unset.++ **Example** (Docker Compose `.env`):+ ```yaml+ PORT=3000+ ```+</ResponseField>++<ResponseField name="DOCKER_PORT">+ Sets the port the OpenLIT container listens on internally. Only relevant for Kubernetes or a raw container run where there's no Docker Compose translating `PORT` for you.++ **Example**:+ ```yaml+ export DOCKER_PORT=3000+ ```+</ResponseField>++<ResponseField name="API_URL">+ Sets the base URL OpenLIT uses to call back into its own API - required by the Auto Evaluation, Auto Pricing, Agents materialization, and telemetry-snapshot cron jobs, which run as separate processes and call this URL directly. Defaults to `http://localhost:$PORT`. Set this if you run behind a reverse proxy, a non-default host, or a different container/pod hostname - otherwise these background jobs will fail silently after every restart.++ **Example**:+ ```yaml+ export API_URL=https://openlit.internal.example.com+ ```+</ResponseField>++<ResponseField name="OPENLIT_EDITION">+ Sets which OpenLIT build is running. Defaults to `oss` if unset.++ **Example**:+ ```yaml+ export OPENLIT_EDITION=oss+ ```+</ResponseField>++## Agents variables++Tune the background job that materializes the [Agents](/latest/openlit/observability/agents/overview) page's call graphs and versions from trace data.++<ResponseField name="AGENTS_MATERIALIZE_SCHEDULE">+ Cron schedule for the Agents materialization job. Defaults to `* * * * *` (every minute); the job self-throttles when there's no new trace data to process.++ **Example**:+ ```yaml+ export AGENTS_MATERIALIZE_SCHEDULE="* * * * *"+ ```+</ResponseField>++<ResponseField name="AGENTS_MATERIALIZE_MAX_PER_TICK">+ Maximum number of agents materialized per scheduled run. Defaults to `100`.++ **Example**:+ ```yaml+ export AGENTS_MATERIALIZE_MAX_PER_TICK=100+ ```+</ResponseField>++<ResponseField name="AGENTS_MATERIALIZE_PARALLEL">+ Maximum number of agents materialized concurrently per run. Defaults to `4`.++ **Example**:+ ```yaml+ export AGENTS_MATERIALIZE_PARALLEL=4+ ```+</ResponseField>++<ResponseField name="AGENTS_LOG_LEVEL">+ Log level for the Agents materialization job (`debug`, `info`, `warn`, `error`). Defaults to `info`.++ **Example**:+ ```yaml+ export AGENTS_LOG_LEVEL=info+ ```+</ResponseField>++<ResponseField name="AGENTS_LOG_STACK">+ Set to `false` to omit stack traces from Agents materialization error logs. Defaults to including them.++ **Example**:+ ```yaml+ export AGENTS_LOG_STACK=false+ ```+</ResponseField>++## Telemetry variables++<Note>+See [Anonymous Telemetry](/latest/openlit/developer-resources/anonymous-telemetry) for what OpenLIT's own usage telemetry collects and why.+</Note>++<ResponseField name="TELEMETRY_ENABLED">+ Set to `false` to disable OpenLIT's anonymous usage telemetry, including the daily instance snapshot.++ **Example**:+ ```yaml+ export TELEMETRY_ENABLED=false+ ```+</ResponseField>++<ResponseField name="TELEMETRY_SNAPSHOT_SCHEDULE">+ Cron schedule for the daily anonymous instance telemetry snapshot. Defaults to `17 3 * * *`.++ **Example**:+ ```yaml+ export TELEMETRY_SNAPSHOT_SCHEDULE="17 3 * * *"+ ```+</ResponseField>+## Security variablesOpenLIT enables stricter API protections by default, including security response headers, CSRF checks for browser session API requests, vault secret encryption, and restricted CORS for the vault secrets API.+<ResponseField name="CRON_JOB_SECRET">+ Token required to trigger internal cron-driven endpoints (Auto Evaluation, Auto Pricing, Agents materialization, telemetry snapshot). **Optional for typical self-hosted installs** - when unset, both the cron scripts and the endpoint check fall back to the same shared default, so scheduled jobs work out of the box with no configuration. Set it only if your instance is exposed such that an untrusted party could otherwise call these endpoints directly.++ Changing this takes effect on the next restart, since cron entries are re-created from the current environment on every server startup.++ **Example**:+ ```yaml+ export CRON_JOB_SECRET=your-cron-secret+ ```+</ResponseField>++<ResponseField name="OPENLIT_REQUIRE_ORG_FILTER">+ Set to `true` to enforce organisation-scoped isolation on coding-agent telemetry queries in multi-organisation deployments.++ **Example**:+ ```yaml+ export OPENLIT_REQUIRE_ORG_FILTER=true+ ```+</ResponseField>+<ResponseField name="OPENLIT_VAULT_ENCRYPTION_KEY">Secret used to encrypt Vault values at rest with AES-256-GCM. If this is not set, OpenLIT falls back to `NEXTAUTH_SECRET`.@@ -207,6 +338,12 @@ Environment variables can be configured in multiple ways depending on your deploFor production deployments, set environment variables directly in your hosting platform or container orchestration system (Kubernetes, Docker Swarm, etc.).+### Applying changes at runtime++OpenLIT reads environment variables once when the server process starts - there's no hot-reload, so a config change (editing a `.env` file, updating a Kubernetes Secret, etc.) only takes effect after you restart the container or process.++A restart is also all you need for the cron-driven features (Auto Evaluation, Auto Pricing, Agents materialization, telemetry snapshot): their scheduled jobs are re-created from the current environment on every startup, so there's no separate step to "re-save" settings after changing something like `API_URL`, `CRON_JOB_SECRET`, or `AGENTS_MATERIALIZE_SCHEDULE`.+## Sample environment file (.env)```.env.example .env@@ -228,9 +365,21 @@ GOOGLE_CLIENT_SECRET="your-google-client-secret"GITHUB_CLIENT_ID="your-github-client-id"GITHUB_CLIENT_SECRET="your-github-client-secret"+# Server Configuration (Optional)+# PORT is for Docker Compose; use DOCKER_PORT instead for Kubernetes or a raw container run+PORT="3000"+API_URL="http://localhost:3000"++# Agents Materialization (Optional)+AGENTS_MATERIALIZE_SCHEDULE="* * * * *"++# Telemetry (Optional)+TELEMETRY_ENABLED="true"+# Security Configuration (Optional)OPENLIT_VAULT_ENCRYPTION_KEY="your-vault-encryption-key"OPENLIT_ALLOWED_CORS_ORIGINS="https://app.example.com,https://admin.example.com"+CRON_JOB_SECRET="your-cron-secret"```---@@ -239,17 +388,10 @@ OPENLIT_ALLOWED_CORS_ORIGINS="https://app.example.com,https://admin.example.com"<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>Create custom visualizations with flexible widgets, queries, and real-time AI monitoring</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card>- <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground" icon='flask'>+ <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground/overview" icon='flask'>Compare cost, duration, and response tokens across different LLMs to find the most efficient model</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/openlit/costs/analytics.mdx b/docs/latest/openlit/costs/analytics.mdxnew file mode 100644index 00000000..fb84ba70--- /dev/null+++ b/docs/latest/openlit/costs/analytics.mdx@@ -0,0 +1,43 @@+---+title: "Analytics"+description: "AI cost analytics for LLM spend - cost by environment and application, Auto Pricing run history, and optimization charts by provider and model"+---++The **Analytics** tab on **Costs** (`/costs`) is OpenLIT's AI cost analytics view. Use the time range tabs at the top (**24H**, **7D**, **1M**, **3M**, or **CUSTOM**) to scope the window, then review usage, Auto Pricing activity, and optimization charts.++If Auto Pricing is off and LLM spans in the range are missing cost, a banner appears: **Some LLM traces are missing cost**, with CTA **Configure Auto Pricing** (switches to the [Configuration](/latest/openlit/costs/configuration) tab).++## Cost usage++Section title: **Cost usage**. Summary tiles:++| Tile | Meaning |+|------|---------|+| **Total AI cost** | Platform-wide spend in the selected range |+| **LLM cost** | Cost attributed to LLM/telemetry traffic |+| **Evaluations cost** | Spend from running evaluations |+| **Openground cost** | Spend from OpenGround comparisons |+| **Avg cost per request** | Average cost per request |++Charts in this section: **Cost by environment**, **Cost by application**.++## Auto pricing runs++Section title: **Auto pricing runs**. Tiles: **Cron runs**, **Successful runs**, **Costs applied**, **Spans scanned**. A run history table shows **Started**, **Status**, **Duration**, **Costs applied**, **Scanned**, **Skipped**, and **Failed**.++Empty copy: *No Auto Pricing runs applied costs in this time range.* Hint CTA: **Enable Auto Pricing** → [Configuration](/latest/openlit/costs/configuration).++## Cost optimization++Section title: **Cost optimization**. Charts: **Cost over time**, **Cost by provider**, **Cost by model**, and **Openground spend by provider**.++---++<CardGroup cols={2}>+ <Card title="Manage models" href="/latest/openlit/costs/manage-models/overview" icon="sliders">+ Edit the prices Analytics uses for LLM cost+ </Card>+ <Card title="Configuration" href="/latest/openlit/costs/configuration" icon="gear">+ Schedule Auto Pricing to fill missing costs+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/costs/configuration.mdx b/docs/latest/openlit/costs/configuration.mdxnew file mode 100644index 00000000..47cd98f7--- /dev/null+++ b/docs/latest/openlit/costs/configuration.mdx@@ -0,0 +1,69 @@+---+title: "Configuration"+sidebarTitle: "Configuration"+description: "AI cost tracking configuration - enable Auto Pricing on a schedule and recalculate LLM costs on individual traces from Monitor → Costs"+---++The **Configuration** tab on **Costs** (`/costs?tab=configuration`) controls AI cost tracking backfills. It recalculates cost on existing LLM traces using the per-model prices stored under [Manage models](/latest/openlit/costs/manage-models/overview). Use it when a trace was ingested without cost, pricing changed and you want missing costs filled in, or you added a custom model after traffic already landed.++Open **Monitor → Costs**, then select **Configuration**. (`/pricing` redirects here.)++## Auto Pricing++The **Auto Pricing** card schedules AI cost tracking for new LLM traces.++<Steps>+ <Step title="Enable Auto Pricing">+ Toggle **Enable Auto Pricing** on.+ </Step>+ <Step title="Set the Cron Schedule">+ Enter a standard cron expression in **Cron Schedule**. Examples:+ - `*/15 * * * *` - every 15 minutes+ - `0 * * * *` - every hour+ - `0 0 * * *` - once a day at midnight+ </Step>+ <Step title="Save">+ Click **Save** (or **Update** if a config already exists). OpenLIT runs Auto Pricing on that schedule for the active database configuration.+ </Step>+</Steps>++On each run, OpenLIT finds recent LLM spans that are missing cost (or have cost `0`), looks up each span's provider and model in Manage models, computes the cost from token counts, and writes it back onto the span. Spans that already have a non-zero cost are left alone - Auto Pricing is a backfill, not an overwrite of vendor-reported or previously set costs. Spans with missing provider/model/tokens, or a model that isn't in Manage models, are skipped (not treated as failures).++<Tip>+Edit pricing on the [Manage models](/latest/openlit/costs/manage-models/overview) tab first. Auto Pricing and manual recalculation both use those prices.+</Tip>++## Manual Pricing++The **Manual Pricing** card explains how to recalculate a single trace on demand:++1. Open any LLM request in [Telemetry](/latest/openlit/observability/telemetry/traces) (**Go to Traces** jumps there).+2. In the trace detail panel, find the **Cost** tile and click the refresh icon. The tooltip reads *Recalculate cost using the model's price in Manage Models* - there is no separate labeled button text.+3. A toast shows **Cost updated** with the new value.++If cost is missing or zero, a pinging dot draws attention to that icon. Unlike Auto Pricing, a manual recalculation refreshes the cost even when a value is already present. The control only appears when the span has a model and span id.++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Why is a trace's cost missing or zero?">+ Usually the SDK didn't send cost, or the model wasn't in Manage models when the span was ingested. Add or fix the model, then recalculate manually or wait for the next Auto Pricing run.+ </Accordion>+ <Accordion title="Does Auto Pricing overwrite an existing cost?">+ No. Auto Pricing only fills in missing or zero-cost spans - it never overwrites a cost your SDK or an upstream vendor already set.+ </Accordion>+ <Accordion title="What happens to Auto Pricing after a restart?">+ Active Auto Pricing schedules are restored automatically when OpenLIT starts again - you don't need to re-enable them after a redeploy.+ </Accordion>+</AccordionGroup>++---++<CardGroup cols={2}>+ <Card title="Analytics" href="/latest/openlit/costs/analytics" icon="chart-line">+ See spend and Auto Pricing run history+ </Card>+ <Card title="Manage models" href="/latest/openlit/costs/manage-models/overview" icon="sliders">+ Edit LLM model pricing used for cost calculation+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/costs/manage-models/import-and-export.mdx b/docs/latest/openlit/costs/manage-models/import-and-export.mdxnew file mode 100644index 00000000..30272079--- /dev/null+++ b/docs/latest/openlit/costs/manage-models/import-and-export.mdx@@ -0,0 +1,62 @@+---+title: "Import and export"+description: "Import Pricing JSON into Manage Models or Export Pricing as SDK-compatible pricing.json - duplicates are skipped on import"+---++Use the header actions on **Manage Models** to move pricing between environments or feed the OpenLIT SDK a file.++## Export Pricing++Click the download icon in the header (tooltip: **Export Pricing**) to open `GET /api/openground/models/export` and download all models as SDK-compatible pricing JSON (download name `custom-pricing.json`), grouped by model type (chat, embeddings, images, audio).++This requires a signed-in OpenLIT session. For a public URL your apps can call without login, use the [SDK Pricing URL](/latest/openlit/costs/manage-models/sdk-pricing) instead.++## Import Pricing JSON++<Steps>+ <Step title="Open the dialog">+ Click the upload icon in the header (tooltip: **Import Pricing JSON**).+ </Step>+ <Step title="Paste JSON">+ Paste either a structured `providers` + `models` payload, a models-only array, or an SDK-style `pricing_json` object. The dialog description: existing models with the same provider + model ID are skipped.+ </Step>+ <Step title="Import">+ Click **Import Pricing JSON**. The toast reports model `imported` / `skipped` counts (provider import counts are returned by the API but not shown in that toast).+ </Step>+</Steps>++Example structured payload (also shown as the dialog placeholder):++```json+{+ "providers": [+ {+ "providerId": "my-provider",+ "displayName": "My Provider",+ "description": "Custom LLM provider"+ }+ ],+ "models": [+ {+ "provider": "my-provider",+ "model_id": "my-model-v1",+ "displayName": "My Model v1",+ "inputPricePerMToken": 2.5,+ "outputPricePerMToken": 10+ }+ ]+}+```++SDK-format imports without a provider default to provider `unknown` - edit those models afterward if needed.++## API++<CardGroup cols={2}>+ <Card title="Export models" href="/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export" icon="download">+ Authenticated pricing.json download+ </Card>+ <Card title="Import models" href="/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import" icon="upload">+ Bulk import providers and models+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/costs/manage-models/models.mdx b/docs/latest/openlit/costs/manage-models/models.mdxnew file mode 100644index 00000000..4c201d77--- /dev/null+++ b/docs/latest/openlit/costs/manage-models/models.mdx@@ -0,0 +1,32 @@+---+title: "Models"+description: "Browse and edit LLM model pricing in Manage Models - Model ID, type, context window, and input/output price per 1M tokens"+---++The left sidebar on **Manage Models** lists providers and their models. Each row shows context window and input price. Badges mark **Default** (seeded) vs **Custom** (added by you). Click a model to open the editor on the right, or use **+** on a provider to add one.++## Edit or add a model++| Field | Notes |+|-------|--------|+| **Model ID** | Identifier your SDK sends (e.g. `gpt-4o-2024-08-06`). Normalized to lowercase; only editable when adding. |+| **Display Name** | Human-readable name in dropdowns. |+| **Model Type** | Chat, Embeddings, Images, or Audio - controls how export JSON is shaped. |+| **Context Window** | Max context length in tokens. |+| **Input Price (per 1M tokens)** / **Output Price (per 1M tokens)** | Used for AI cost tracking and recalculation. |+| **Capabilities (comma separated)** | Optional list (e.g. `function-calling, vision, streaming`). |++Click **Save Model** to persist. For an existing model, **Delete** opens a confirmation titled **Delete Model**.++Empty state copy when nothing is selected: **Select a model to view details**.++## Tips++- Model ID should match `gen_ai.request.model` on your spans so [Configuration](/latest/openlit/costs/configuration) and ingest-time pricing can resolve it.+- After changing prices, use Auto Pricing or the Cost tile refresh icon on a trace for historical spans; the [SDK Pricing URL](/latest/openlit/costs/manage-models/sdk-pricing) applies to future tracing.++## API++<Card title="Models API" href="/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models" icon="code">+ List, create/update, and delete models+</Card>diff --git a/docs/latest/openlit/costs/manage-models/overview.mdx b/docs/latest/openlit/costs/manage-models/overview.mdxnew file mode 100644index 00000000..42110b8e--- /dev/null+++ b/docs/latest/openlit/costs/manage-models/overview.mdx@@ -0,0 +1,58 @@+---+title: "Overview"+description: "Manage models on the Costs page - edit LLM providers and per-model token prices used for AI cost tracking across traces, Otter, OpenGround, and the SDK"+---++**Manage models** is the **Manage models** tab on **Costs** (`/costs?tab=models`) under Monitor. Every connected ClickHouse database is seeded with **14 providers** and a large built-in model catalog (90+ models). You can edit any price, add custom providers and models, import/export pricing JSON, and point the OpenLIT SDK at a public pricing URL so ingest-time cost tracking stays in sync with the UI.++Those prices power [Configuration](/latest/openlit/costs/configuration) (Auto Pricing and manual recalculation), [Analytics](/latest/openlit/costs/analytics), cost columns in telemetry, [OpenGround](/latest/openlit/prompts-experiments/openground/overview) comparisons, and Otter's message cost estimates.++Open **Monitor → Costs → Manage models**. (`/manage-models` redirects here.) Toolbar actions (Add Provider, SDK Usage, Import, Export) are icon-only with tooltips.++<CardGroup cols={2}>+ <Card title="Providers" href="/latest/openlit/costs/manage-models/providers" icon="tags">+ Add or edit LLM providers (Provider ID, display name, Vault requirement)+ </Card>+ <Card title="Models" href="/latest/openlit/costs/manage-models/models" icon="cube">+ Browse, edit, add, or delete models and per-1M-token pricing+ </Card>+ <Card title="Import and export" href="/latest/openlit/costs/manage-models/import-and-export" icon="file-import">+ Import Pricing JSON or Export Pricing for backup and sync+ </Card>+ <Card title="SDK pricing URL" href="/latest/openlit/costs/manage-models/sdk-pricing" icon="code">+ Copy the public SDK Pricing URL and wire it into openlit.init()+ </Card>+</CardGroup>++## Built-in providers++Out of the box you get: OpenAI, Anthropic, Google AI, Mistral AI, Groq, Perplexity, Azure OpenAI, Cohere, Together AI, Fireworks AI, DeepSeek, xAI, Hugging Face, and Replicate. Model lists change as the catalog is updated - treat Manage models itself as the source of truth for what is installed in your database.++## API reference++Authenticated Manage Models APIs and the public SDK export are documented under [API Reference → Manage Models](/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models). Feature pages link there for request shapes and examples instead of duplicating code samples.++## Frequently asked questions++<AccordionGroup>+ <Accordion title="How does OpenLIT calculate cost from these prices?">+ It multiplies each span's input and output token counts by that model's input and output price per 1M tokens from Manage models.+ </Accordion>+ <Accordion title="Can I add a custom or self-hosted model?">+ Yes. Add a provider if needed, then add a model with your own Model ID and pricing - or bulk-import via Import Pricing JSON.+ </Accordion>+ <Accordion title="Does Default vs Custom matter for pricing?">+ **Default** means the model was seeded by OpenLIT; **Custom** means you added it. Both are fully editable and used the same way for cost tracking.+ </Accordion>+</AccordionGroup>++---++<CardGroup cols={2}>+ <Card title="Configuration" href="/latest/openlit/costs/configuration" icon="gear">+ Backfill or refresh gen_ai.usage.cost from Manage models prices+ </Card>+ <Card title="OpenGround" href="/latest/openlit/prompts-experiments/openground/overview" icon="display">+ Compare models side-by-side using the same provider catalog+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/costs/manage-models/providers.mdx b/docs/latest/openlit/costs/manage-models/providers.mdxnew file mode 100644index 00000000..1c5af372--- /dev/null+++ b/docs/latest/openlit/costs/manage-models/providers.mdx@@ -0,0 +1,37 @@+---+title: "Providers"+description: "Add and edit LLM providers in Manage Models - Provider ID, display name, description, and Requires API Key (Vault)"+---++Providers are the top-level grouping for models on **Manage Models** (`/costs?tab=models`). OpenLIT ships 14 built-in providers; you can add your own for self-hosted or niche APIs.++## Add a provider++<Steps>+ <Step title="Open Add Provider">+ In the Manage Models header, click the **+** control (tooltip: **Add Provider**).+ </Step>+ <Step title="Fill in the fields">+ - **Provider ID** - lowercase identifier (e.g. `my-provider`). Spaces become hyphens; other special characters are stripped. This must match what your SDK sends in `gen_ai.system`.+ - **Display Name** - label shown in the UI.+ - **Description** - optional.+ - **Requires API Key (Vault)** - toggle on if callers need a Vault secret for this provider.+ </Step>+ <Step title="Save">+ Click **Add Provider**. The provider appears in the left sidebar.+ </Step>+</Steps>++## Edit a provider++Hover a provider in the sidebar and use the edit control to open **Edit Provider**. You can change display name, description, and the Vault requirement. **Provider ID cannot be changed** after creation.++## Delete a provider++There is no delete button in the Manage Models UI today. To remove a provider and all of its models, use the [Providers API](/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers) `DELETE` operation.++## API++<Card title="Providers API" href="/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers" icon="code">+ List, create, update, and delete providers+</Card>diff --git a/docs/latest/openlit/costs/manage-models/sdk-pricing.mdx b/docs/latest/openlit/costs/manage-models/sdk-pricing.mdxnew file mode 100644index 00000000..6aa9016f--- /dev/null+++ b/docs/latest/openlit/costs/manage-models/sdk-pricing.mdx@@ -0,0 +1,26 @@+---+title: "SDK pricing URL"+description: "Use Manage Models' public SDK Pricing URL in openlit.init() so LLM token pricing stays synced without shipping a local pricing.json"+---++Each active database configuration exposes a public **SDK Pricing URL** on Manage Models:++`/api/pricing/export/<database-config-id>`++The bar under the page header shows the full URL for the currently selected database config. Click **Copy** to copy it. The endpoint needs no authentication - it only returns pricing data (no secrets or traces). Share it only with systems that should read your model prices.++## SDK Usage dialog++Click the code icon in the header (tooltip: **SDK Usage**), or the **SDK Usage** button on the empty editor state, to open **Use this pricing in your OpenLIT SDK**. The dialog shows the URL plus ready-to-paste Python, TypeScript, and Go snippets that pass it to `pricing_json` / `pricingJson` / `PricingJson` in `openlit.init()`.++Pass your OpenLIT host and the copied path - do not hardcode a machine-local address in shared configs.++The dialog note: the SDK fetches this URL on startup, and price edits here apply to **future tracing**. Historical spans still need [Configuration](/latest/openlit/costs/configuration) if you want them updated.++Responses are cache-friendly (`Cache-Control: public, max-age=300`).++## API++<Card title="Public pricing export" href="/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing" icon="cloud-arrow-down">+ Full schema and SDK examples for GET /api/pricing/export/{dbConfigId}+</Card>diff --git a/docs/latest/openlit/costs/overview.mdx b/docs/latest/openlit/costs/overview.mdxnew file mode 100644index 00000000..221eba8d--- /dev/null+++ b/docs/latest/openlit/costs/overview.mdx@@ -0,0 +1,33 @@+---+title: "Overview"+description: "AI cost tracking and AI cost analytics in OpenLIT - monitor LLM spend, manage model pricing, and backfill missing costs from Monitor → Costs"+---++**Costs** (`/costs`) is OpenLIT's AI cost tracking and analytics page under **Monitor**. It combines spend analytics, LLM model pricing (Manage models), and Auto Pricing configuration in one place - the old Pricing and Manage Models pages redirect here.++Open it from **Monitor → Costs**. The page has three tabs:++<CardGroup cols={3}>+ <Card title="Analytics" href="/latest/openlit/costs/analytics" icon="chart-line">+ Cost usage, Auto Pricing runs, and optimization charts+ </Card>+ <Card title="Manage models" href="/latest/openlit/costs/manage-models/overview" icon="sliders">+ Providers, per-model token prices, import/export, SDK pricing URL+ </Card>+ <Card title="Configuration" href="/latest/openlit/costs/configuration" icon="gear">+ Enable Auto Pricing and recalculate cost on individual traces+ </Card>+</CardGroup>++Default tab is **Analytics** (`/costs`). Use `?tab=models` or `?tab=configuration` for the other tabs (`/pricing` redirects to configuration; `/manage-models` redirects to models).++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Where did Pricing and Manage Models go?">+ They live on the Costs page under Monitor. Pricing settings are the **Configuration** tab; model catalog and prices are the **Manage models** tab.+ </Accordion>+ <Accordion title="How does OpenLIT calculate AI cost?">+ From each span's input/output token counts times that model's input/output price per 1M tokens in Manage models. Analytics and Auto Pricing both use those prices.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/dashboards/create-dashboard.mdx b/docs/latest/openlit/dashboards/create-dashboard.mdxindex 7c9c5166..923ffb3e 100644--- a/docs/latest/openlit/dashboards/create-dashboard.mdx+++ b/docs/latest/openlit/dashboards/create-dashboard.mdx@@ -10,7 +10,7 @@ description: "Learn how to create dashboards in OpenLIT with step-by-step instru<Steps titleSize="h4"><Step title="Create a new dashboard"><Frame>- <img src="/images/create-dashboard-form.png" />+ <img src="/images/create-dashboard-form.png" alt="Create a new dashboard or folder form" /></Frame>1. Navigate to the **Dashboards** section in the main navigation@@ -26,17 +26,17 @@ description: "Learn how to create dashboards in OpenLIT with step-by-step instru</Step><Step title="Add your first widget"><Frame>- <img src="/images/dashboard-empty.png" />+ <img src="/images/dashboard-empty.png" alt="Empty dashboard canvas with a Create Your First Widget button" /></Frame>1. Click **Create Your First Widget** button on the empty dashboard2. You'll see a **"Select a Widget"** popup with two options:<Frame>- <img src="/images/dashboard-select-existing-widget-popup.png" />+ <img src="/images/dashboard-select-existing-widget-popup.png" alt="Select a Widget popup showing existing pre-built widgets" /></Frame>- **Choose from existing widgets**: Select from pre-built widgets like "Avg temperature", "Avg utilization percentage", "Requests per time", etc.- **Create new widget**: Click **+ Create New Widget** to build a custom widget<Frame>- <img src="/images/dashboard-select-new-widget-popup.png" />+ <img src="/images/dashboard-select-new-widget-popup.png" alt="Create New Widget option in the Select a Widget popup" /></Frame></Step>@@ -44,7 +44,7 @@ description: "Learn how to create dashboards in OpenLIT with step-by-step instruWhen creating a new widget, you'll see a configuration panel with three tabs:**General Tab**<Frame>- <img src="/images/dashboard-widget-configuration-general.png" />+ <img src="/images/dashboard-widget-configuration-general.png" alt="Widget configuration panel General tab" /></Frame>1. **Widget Title**: Enter a descriptive name (e.g., "Total Requests")2. **Widget Type**: Select from **Stat Card** in this example@@ -55,7 +55,7 @@ description: "Learn how to create dashboards in OpenLIT with step-by-step instru**Query Tab**<Frame>- <img src="/images/dashboard-widget-configuration-query.png" />+ <img src="/images/dashboard-widget-configuration-query.png" alt="Widget configuration panel Query tab with a SQL editor" /></Frame>1. **Write your SQL query** against the ClickHouse database. For example:```sql@@ -70,7 +70,7 @@ description: "Learn how to create dashboards in OpenLIT with step-by-step instru2. The widget will appear on your dashboard canvas3. You can immediately see your data visualization<Frame>- <img src="/images/dashboard-widget-created.png" />+ <img src="/images/dashboard-widget-created.png" alt="Dashboard canvas showing the newly created widget" /></Frame></Step><Step title="Your dashboard is ready!">diff --git a/docs/latest/openlit/dashboards/create-folder.mdx b/docs/latest/openlit/dashboards/create-folder.mdxindex d12fd2e2..3444ae6e 100644--- a/docs/latest/openlit/dashboards/create-folder.mdx+++ b/docs/latest/openlit/dashboards/create-folder.mdx@@ -10,7 +10,7 @@ description: "Create folders to organize dashboards into logical collections by<Steps titleSize="h4"><Step title="Create a new folder"><Frame>- <img src="/images/create-dashboard-form.png" />+ <img src="/images/create-dashboard-form.png" alt="Create a new dashboard or folder form" /></Frame>1. Navigate to the **Dashboards** section in the main navigationdiff --git a/docs/latest/openlit/dashboards/export-dashboard.mdx b/docs/latest/openlit/dashboards/export-dashboard.mdxindex 2d70a654..62d6f74d 100644--- a/docs/latest/openlit/dashboards/export-dashboard.mdx+++ b/docs/latest/openlit/dashboards/export-dashboard.mdx@@ -19,7 +19,7 @@ Dashboard export allows you to save your dashboard configurations as JSON files,<Step title="Access export menu"><Frame>- <img src="/images/dashboard-export.png" />+ <img src="/images/dashboard-export.png" alt="Dashboard context menu with the Export Layout option" /></Frame>1. Click the **three dots (⋯)** menu icon next to your dashboard name to open the context menu.1. From the dropdown menu, select **Export Layout**.diff --git a/docs/latest/openlit/dashboards/filters-and-dynamic-bindings.mdx b/docs/latest/openlit/dashboards/filters-and-dynamic-bindings.mdxindex efa0913c..b4c483f7 100644--- a/docs/latest/openlit/dashboards/filters-and-dynamic-bindings.mdx+++ b/docs/latest/openlit/dashboards/filters-and-dynamic-bindings.mdx@@ -22,7 +22,7 @@ At runtime, OpenLIT replaces each binding with the user-provided filter value.### Common examples:-* `{{filter.timeLimit.start}}` and `{{filter.timeLimit.end}}` — for date range filters+* `{{filter.timeLimit.start}}` and `{{filter.timeLimit.end}}` - for date range filters## How to add and use filters@@ -54,4 +54,4 @@ In this example:* Always use meaningful keys like `team_id`, `env`, or `region`* Use `coalesce({{param}}, 'default')` to provide fallback values in ClickHouse* Test your queries in the SQL editor with example values for clarity-* Avoid hardcoded constraints—use bindings to keep dashboards flexible+* Avoid hardcoded constraints - use bindings to keep dashboards flexiblediff --git a/docs/latest/openlit/dashboards/import-dashboard.mdx b/docs/latest/openlit/dashboards/import-dashboard.mdxindex e92526da..e0ad2d52 100644--- a/docs/latest/openlit/dashboards/import-dashboard.mdx+++ b/docs/latest/openlit/dashboards/import-dashboard.mdx@@ -17,7 +17,7 @@ Dashboard import lets you load pre-built dashboard configurations from JSON fileNavigate to your Dashboards Explorer and locate the **Import Layout** button in the top-right corner of the interface.<Frame>- <img src="/images/dashboard-import-cta.png" />+ <img src="/images/dashboard-import-cta.png" alt="Import Layout button in the Dashboards Explorer" /></Frame>Click the **Import Layout** button to open the import dialog.@@ -27,7 +27,7 @@ Dashboard import lets you load pre-built dashboard configurations from JSON fileThe "Import Board Layout" dialog will appear with a file upload interface.<Frame>- <img src="/images/dashboard-import-popup.png" />+ <img src="/images/dashboard-import-popup.png" alt="Import Board Layout dialog with a file upload interface" /></Frame>You can upload your dashboard configuration in two ways:diff --git a/docs/latest/openlit/dashboards/pin-dashboard.mdx b/docs/latest/openlit/dashboards/pin-dashboard.mdxindex 94f8daa0..2235dc22 100644--- a/docs/latest/openlit/dashboards/pin-dashboard.mdx+++ b/docs/latest/openlit/dashboards/pin-dashboard.mdx@@ -24,5 +24,5 @@ To unpin a dashboard:The dashboard will return to its normal position in the folder/board structure.<Tip>- You can pin multiple dashboards—perfect for surfacing daily dashboards, team-specific views, or in-progress monitoring.+ You can pin multiple dashboards - perfect for surfacing daily dashboards, team-specific views, or in-progress monitoring.</Tip>diff --git a/docs/latest/openlit/dashboards/widgets/bar-chart-widget.mdx b/docs/latest/openlit/dashboards/widgets/bar-chart-widget.mdxindex 76cfad4c..f2ee29fb 100644--- a/docs/latest/openlit/dashboards/widgets/bar-chart-widget.mdx+++ b/docs/latest/openlit/dashboards/widgets/bar-chart-widget.mdx@@ -3,7 +3,7 @@ title: Bar Chart Widgetdescription: Learn how to use the Bar Chart Widget in OpenLIT to compare grouped data using ClickHouse queries and dynamic filters.----The **Bar Chart Widget** in OpenLIT is great for comparing categorical or grouped data—like errors by type, usage by region, or revenue by plan. You can customize the chart direction, stack behavior, and bind dynamic filters directly into your ClickHouse queries.+The **Bar Chart Widget** in OpenLIT is great for comparing categorical or grouped data - like errors by type, usage by region, or revenue by plan. You can customize the chart direction, stack behavior, and bind dynamic filters directly into your ClickHouse queries.## When to usediff --git a/docs/latest/openlit/dashboards/widgets/stat-widget.mdx b/docs/latest/openlit/dashboards/widgets/stat-widget.mdxindex 01f45c9a..e96a9e43 100644--- a/docs/latest/openlit/dashboards/widgets/stat-widget.mdx+++ b/docs/latest/openlit/dashboards/widgets/stat-widget.mdx@@ -65,4 +65,4 @@ Use dynamic filters like `{{filter.timeLimit.start}}`, `{{filter.timeLimit.end}}* Combine stat cards with charts for more context* Use consistent units (e.g., %, \$, MB)-* Avoid overloading the dashboard with too many KPIs—focus on what matters\ No newline at end of file+* Avoid overloading the dashboard with too many KPIs - focus on what matters\ No newline at end of filediff --git a/docs/latest/openlit/dashboards/widgets/table-widget.mdx b/docs/latest/openlit/dashboards/widgets/table-widget.mdxindex 35145e64..9f6aca3d 100644--- a/docs/latest/openlit/dashboards/widgets/table-widget.mdx+++ b/docs/latest/openlit/dashboards/widgets/table-widget.mdx@@ -43,5 +43,5 @@ Use dynamic filters like `{{filter.timeLimit.start}}`, `{{filter.timeLimit.end}}## Best practices* Use aliases for column names for readability (`AS user_email`, `AS created_at`)-* Limit unnecessary columns—focus on actionable fields+* Limit unnecessary columns - focus on actionable fields* Use `{{filter.timeLimit.start}}`, `{{filter.timeLimit.end}}` to make tables dynamic\ No newline at end of filediff --git a/docs/latest/openlit/developer-resources/anonymous-telemetry.mdx b/docs/latest/openlit/developer-resources/anonymous-telemetry.mdxindex 2c87129c..d9073da4 100644--- a/docs/latest/openlit/developer-resources/anonymous-telemetry.mdx+++ b/docs/latest/openlit/developer-resources/anonymous-telemetry.mdx@@ -1,59 +1,67 @@----title: 'Anonymous Usage Metrics'-description: 'Details about OpenLIT anonymous usage metrics reporting'+title: "Anonymous Usage Metrics"+sidebarTitle: "Anonymous Usage Metrics"+description: "How OpenLIT's anonymous usage metrics work, what is collected, and how to opt out of telemetry with TELEMETRY_ENABLED=false"+icon: "chart-column"----## Why?-To enhance and improve the OpenLIT experience, we have introduced a telemetry feature that collects anonymous usage data. This data helps us understand how users interact with OpenLIT and identify areas for improvement. We assure you that all collected data is completely anonymous and is used solely for the purpose of enhancing our services.+OpenLIT collects anonymous usage metrics by default so we can see which features are used and where to improve the product. Nothing here is meant to identify you personally - there is no email, name, password, secret, or span/trace content in what we send.-## What identifies you?+There is **no in-app settings toggle** for this. Opt out with an environment variable on the OpenLIT deployment, then restart so the change takes effect.-Telemetry is anonymous by design:+## How to opt out of telemetry-- **User (MAU)**: an opaque, randomly generated user id via `identify` (never your email, name, or password). PostHog Active Users = DAU / WAU / MAU.-- **Organisation (MAO)**: an opaque organisation id via PostHog Groups (`organisation`). PostHog Active Groups for that type = DAO / WAO / MAO. Project scope uses a second group (`project`) the same way.-- **Install**: a stable, random `install_id` generated per OpenLIT installation so we can distinguish deployments without identifying you.-- **Environment**: standard analytics context such as browser, operating system, and IP-derived country. We do not track personal identity.+Set:-## What do we track?+```bash+TELEMETRY_ENABLED=false+```-OpenLIT collects anonymous events from both OpenLIT and the OpenLIT SDK to analyze usage. We deliberately avoid high-noise, per-request events. Events include:+That disables browser analytics, server-side usage events, and the daily install snapshot. Omit the variable (or set it to anything other than `false`) to leave collection on - Compose and `.env` examples ship with it enabled.-- **Active usage**: opaque user id for DAU/WAU/MAU; organisation and project Groups for active-org / active-project counts; login and registration; named page visits (by page name, not full URLs).-- **Organisations & projects**: create / switch / list inventory counts (`organisation_id` / `project_id` only — never names or member emails).-- **Feature adoption**: create/update/delete counts for prompts, rules, evaluations, dashboards, contexts, openground, and database configurations.-- **Inventory counts**: how many of each entity exist (for example orgs, projects, prompts, rules, evaluation types, dashboards, fleet agents connected, application and coding agents) — counts only, never names or contents.-- **Auto features**: whether auto pricing and auto evaluation are enabled, and their run outcomes.-- **SDK operations**: success/failure counts for fetching prompts and vault secrets.-- **Daily instance snapshot**: once per day, per install, we send an aggregate `INSTANCE_TELEMETRY_SNAPSHOT` with anonymous totals — including OpenLIT version, `users_total` / `organisations_total` / `projects_total`, other entity inventory counts, and world-total ingestion volume (`spans_total`, `traces_total`, and 24h deltas). This lets us understand OpenLIT usage at scale without ever sending your telemetry data itself.+## What identifies an install (anonymously)-We never send emails, names, passwords, secrets, API keys, span/trace contents, or per-span ingestion events.+| Signal | What it is |+|--------|------------|+| **User** | An opaque user id - never your email, name, or password |+| **Organisation / project** | Opaque organisation and project group ids - never names or member emails |+| **Install** | A stable random `install_id` per OpenLIT deployment |+| **Environment** | Ordinary analytics context such as browser, OS, and approximate location |-## How to disable the collection of usage metrics+## What we track-Usage metrics collection in OpenLIT is enabled by default, but you can opt-out at any time. To disable usage metrics collection (including the daily instance snapshot), set the `TELEMETRY_ENABLED=false` environment variable when starting OpenLIT.+When enabled, OpenLIT sends:-## Where are usage metrics stored?+- **Active usage** - opaque user / organisation / project activity, login and registration, and page path visits as people use the UI+- **Feature adoption** - create/update/delete-style signals for prompts, rules, evaluations, dashboards, contexts, OpenGround, and database configurations (ids and counts, not names or contents)+- **Auto features** - whether Auto Pricing and auto evaluation are turned on, and run outcomes+- **SDK-facing API use** - success/failure when applications fetch prompts or Vault secrets through OpenLIT (the OpenLIT server records this; the SDK packages do not phone home on their own)+- **Daily install snapshot** - once per day, aggregate totals for that install: OpenLIT version, counts of users / organisations / projects / database configs, whether auto pricing or auto evaluation is configured, and high-level ingestion volume counts-We use Posthog to securely store and visualize usage metrics, providing essential insights into how OpenLIT is used.+We never send emails, names, passwords, secrets, API keys, prompt or rule text, or the contents of your spans and traces.++## Where metrics are stored++Usage metrics are sent to PostHog for secure storage and product analytics.++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Is there a UI switch to disable this?">+ No. Opt out only via `TELEMETRY_ENABLED=false` on the OpenLIT process, then restart.+ </Accordion>+ <Accordion title="Does disabling this stop my own OpenTelemetry traces?">+ No. This only controls OpenLIT's anonymous product usage metrics. Your application's OTLP traces, metrics, and logs are separate.+ </Accordion>+</AccordionGroup>----<CardGroup cols={3}>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking+<CardGroup cols={2}>+ <Card title="Configuration" href="/latest/openlit/configuration" icon="gears">+ Full environment variable reference, including TELEMETRY_ENABLED</Card>- <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground" icon='flask'>- Compare cost, duration, and response tokens across different LLMs to find the most efficient model+ <Card title="Installation" href="/latest/openlit/installation" icon="download">+ Deploy OpenLIT and set environment variables for your setup</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export.mdxnew file mode 100644index 00000000..7149e572--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export.mdx@@ -0,0 +1,14 @@+---+title: "Export Pricing"+openapi: "GET /api/openground/models/export"+---++Authenticated export used by the **Export Pricing** button on Manage Models.++```bash+curl -X GET "/api/openground/models/export" \+ -H "Cookie: <signed-in-session-cookie>" \+ -o pricing.json+```++For a public URL the SDK can call without a session cookie, use [Public pricing export](/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing).diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export.yml b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export.ymlnew file mode 100644index 00000000..f11b9410--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/export.yml@@ -0,0 +1,37 @@+openapi: 3.0.0+info:+ title: Manage Models - Export Pricing+ description: Authenticated download of all model pricing as SDK-compatible pricing.json for the active database configuration.+ version: 1.0.0+servers:+ - url: /+paths:+ /api/openground/models/export:+ get:+ summary: Export pricing JSON+ description: Returns pricing grouped by model type (chat, embeddings, images, audio) in OpenLIT SDK pricing_json shape.+ operationId: exportModelsPricing+ responses:+ '200':+ description: pricing.json body+ content:+ application/json:+ schema:+ type: object+ example:+ chat:+ gpt-4o:+ promptPrice: 0.0025+ completionPrice: 0.01+ embeddings:+ text-embedding-3-small: 0.00002+ '401':+ description: Not signed in+components:+ securitySchemes:+ sessionAuth:+ type: apiKey+ in: cookie+ name: next-auth.session-token+security:+ - sessionAuth: []diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import.mdxnew file mode 100644index 00000000..9bcc6ed5--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import.mdx@@ -0,0 +1,30 @@+---+title: "Import Pricing"+openapi: "POST /api/openground/models/import"+---++Bulk import used by **Import Pricing JSON** on Manage Models. Existing provider + model ID pairs are skipped.++```bash+curl -X POST "/api/openground/models/import" \+ -H "Content-Type: application/json" \+ -H "Cookie: <signed-in-session-cookie>" \+ -d '{+ "providers": [+ {+ "providerId": "my-provider",+ "displayName": "My Provider",+ "description": "Custom LLM provider"+ }+ ],+ "models": [+ {+ "provider": "my-provider",+ "model_id": "my-model-v1",+ "displayName": "My Model v1",+ "inputPricePerMToken": 2.5,+ "outputPricePerMToken": 10+ }+ ]+ }'+```diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import.yml b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import.ymlnew file mode 100644index 00000000..b11006c3--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/import.yml@@ -0,0 +1,74 @@+openapi: 3.0.0+info:+ title: Manage Models - Import Pricing+ description: Bulk-import providers and models from structured JSON or SDK pricing_json. Skips existing provider+model_id pairs. Requires a signed-in OpenLIT session.+ version: 1.0.0+servers:+ - url: /+paths:+ /api/openground/models/import:+ post:+ summary: Import pricing JSON+ description: >-+ Accepts a structured providers and models payload, or the SDK+ pricing_json object shape keyed by model type (chat, embeddings, and so on).+ operationId: importModelsPricing+ requestBody:+ required: true+ content:+ application/json:+ schema:+ type: object+ properties:+ providers:+ type: array+ items:+ type: object+ properties:+ providerId:+ type: string+ displayName:+ type: string+ description:+ type: string+ models:+ type: array+ items:+ type: object+ properties:+ provider:+ type: string+ model_id:+ type: string+ displayName:+ type: string+ inputPricePerMToken:+ type: number+ outputPricePerMToken:+ type: number+ responses:+ '200':+ description: Import summary+ content:+ application/json:+ schema:+ type: object+ properties:+ imported:+ type: number+ skipped:+ type: number+ providersImported:+ type: number+ providersSkipped:+ type: number+ '401':+ description: Not signed in+components:+ securitySchemes:+ sessionAuth:+ type: apiKey+ in: cookie+ name: next-auth.session-token+security:+ - sessionAuth: []diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models.mdxnew file mode 100644index 00000000..b6ac9763--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models.mdx@@ -0,0 +1,55 @@+---+title: "Models"+openapi: "GET /api/openground/models"+---++Session-authenticated APIs used by the Manage Models UI for the active database configuration.++## Operations++| Method | Path | Purpose |+|--------|------|---------|+| `GET` | `/api/openground/models` | List models (optional `?provider=`) |+| `POST` | `/api/openground/models` | Create or update a model |+| `DELETE` | `/api/openground/models?model_id=&provider=` | Delete a model |++## Examples++<Tabs>+ <Tab title="List">+ ```bash+ curl -X GET "/api/openground/models" \+ -H "Cookie: <signed-in-session-cookie>"+ ```+ </Tab>+ <Tab title="Create or update">+ ```bash+ curl -X POST "/api/openground/models" \+ -H "Content-Type: application/json" \+ -H "Cookie: <signed-in-session-cookie>" \+ -d '{+ "provider": "openai",+ "model": {+ "id": "gpt-4o-custom",+ "model_id": "gpt-4o-custom",+ "displayName": "GPT-4o Custom",+ "modelType": "chat",+ "contextWindow": 128000,+ "inputPricePerMToken": 2.5,+ "outputPricePerMToken": 10,+ "capabilities": ["function-calling"]+ }+ }'+ ```+ </Tab>+ <Tab title="Delete">+ ```bash+ curl -X DELETE "/api/openground/models?model_id=gpt-4o-custom&provider=openai" \+ -H "Cookie: <signed-in-session-cookie>"+ ```+ </Tab>+</Tabs>++<Note>+ These routes require a signed-in OpenLIT browser session. They are not the same as API-key authenticated SDK endpoints such as Prompt Hub or Vault.+</Note>diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models.yml b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models.ymlnew file mode 100644index 00000000..0a241a22--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models.yml@@ -0,0 +1,129 @@+openapi: 3.0.0+info:+ title: Manage Models - Models API+ description: List, create or update, and delete LLM models and pricing for the active database configuration. Requires a signed-in OpenLIT session (same APIs the Manage Models UI calls).+ version: 1.0.0+servers:+ - url: /+paths:+ /api/openground/models:+ get:+ summary: List models+ description: Returns models grouped by provider. Optional `provider` query filters to one provider.+ operationId: listModels+ parameters:+ - name: provider+ in: query+ required: false+ schema:+ type: string+ example: openai+ responses:+ '200':+ description: Models grouped by provider id, or a list when `provider` is set.+ content:+ application/json:+ schema:+ type: object+ additionalProperties:+ type: array+ items:+ type: object+ properties:+ model_id:+ type: string+ displayName:+ type: string+ modelType:+ type: string+ contextWindow:+ type: number+ inputPricePerMToken:+ type: number+ outputPricePerMToken:+ type: number+ capabilities:+ type: array+ items:+ type: string+ isDefault:+ type: boolean+ '401':+ description: Not signed in+ post:+ summary: Create or update a model+ description: Upserts a model under a provider for the active database configuration.+ operationId: upsertModel+ requestBody:+ required: true+ content:+ application/json:+ schema:+ type: object+ required:+ - provider+ - model+ properties:+ provider:+ type: string+ example: openai+ customId:+ type: string+ description: Existing row id when updating+ model:+ type: object+ properties:+ id:+ type: string+ example: gpt-4o-custom+ model_id:+ type: string+ example: gpt-4o-custom+ displayName:+ type: string+ modelType:+ type: string+ enum: [chat, embeddings, images, audio]+ contextWindow:+ type: number+ inputPricePerMToken:+ type: number+ outputPricePerMToken:+ type: number+ capabilities:+ type: array+ items:+ type: string+ responses:+ '200':+ description: Model saved+ '401':+ description: Not signed in+ delete:+ summary: Delete a model+ description: Deletes a model by `model_id` and `provider` query parameters.+ operationId: deleteModel+ parameters:+ - name: model_id+ in: query+ required: true+ schema:+ type: string+ - name: provider+ in: query+ required: true+ schema:+ type: string+ responses:+ '200':+ description: Model deleted+ '401':+ description: Not signed in+components:+ securitySchemes:+ sessionAuth:+ type: apiKey+ in: cookie+ name: next-auth.session-token+security:+ - sessionAuth: []diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers.mdxnew file mode 100644index 00000000..03921476--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers.mdx@@ -0,0 +1,43 @@+---+title: "Providers"+openapi: "GET /api/openground/providers"+---++Session-authenticated provider APIs for Manage Models.++## Operations++| Method | Path | Purpose |+|--------|------|---------|+| `GET` | `/api/openground/providers` | List providers (optional `?provider=` or `?search=`) |+| `POST` | `/api/openground/providers` | Create a provider |+| `PUT` | `/api/openground/providers` | Update provider metadata |+| `DELETE` | `/api/openground/providers?provider=` | Delete a provider and all its models |++## Examples++<Tabs>+ <Tab title="Create">+ ```bash+ curl -X POST "/api/openground/providers" \+ -H "Content-Type: application/json" \+ -H "Cookie: <signed-in-session-cookie>" \+ -d '{+ "providerId": "my-provider",+ "displayName": "My Provider",+ "description": "Custom LLM provider",+ "requiresVault": true+ }'+ ```+ </Tab>+ <Tab title="Delete">+ ```bash+ curl -X DELETE "/api/openground/providers?provider=my-provider" \+ -H "Cookie: <signed-in-session-cookie>"+ ```+ </Tab>+</Tabs>++<Note>+ Provider delete is available on this API only - the Manage Models UI does not expose a delete button for providers.+</Note>diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers.yml b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers.ymlnew file mode 100644index 00000000..8412badc--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/providers.yml@@ -0,0 +1,105 @@+openapi: 3.0.0+info:+ title: Manage Models - Providers API+ description: List, create, update, and delete LLM providers for the active database configuration. Requires a signed-in OpenLIT session.+ version: 1.0.0+servers:+ - url: /+paths:+ /api/openground/providers:+ get:+ summary: List providers+ description: Returns providers with supported models. Optional `provider` or `search` query params.+ operationId: listProviders+ parameters:+ - name: provider+ in: query+ required: false+ schema:+ type: string+ - name: search+ in: query+ required: false+ schema:+ type: string+ responses:+ '200':+ description: Provider list or a single provider+ '401':+ description: Not signed in+ post:+ summary: Create a provider+ operationId: createProvider+ requestBody:+ required: true+ content:+ application/json:+ schema:+ type: object+ required:+ - providerId+ - displayName+ properties:+ providerId:+ type: string+ example: my-provider+ displayName:+ type: string+ example: My Provider+ description:+ type: string+ requiresVault:+ type: boolean+ default: true+ responses:+ '200':+ description: Provider created+ '401':+ description: Not signed in+ put:+ summary: Update a provider+ operationId: updateProvider+ requestBody:+ required: true+ content:+ application/json:+ schema:+ type: object+ required:+ - providerId+ properties:+ providerId:+ type: string+ displayName:+ type: string+ description:+ type: string+ requiresVault:+ type: boolean+ responses:+ '200':+ description: Provider updated+ '401':+ description: Not signed in+ delete:+ summary: Delete a provider and its models+ operationId: deleteProvider+ parameters:+ - name: provider+ in: query+ required: true+ schema:+ type: string+ responses:+ '200':+ description: Provider deleted+ '401':+ description: Not signed in+components:+ securitySchemes:+ sessionAuth:+ type: apiKey+ in: cookie+ name: next-auth.session-token+security:+ - sessionAuth: []diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing.mdxnew file mode 100644index 00000000..fae8a1fc--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing.mdx@@ -0,0 +1,57 @@+---+title: "Public pricing export"+openapi: "GET /api/pricing/export/{dbConfigId}"+---++Public endpoint behind the **SDK Pricing URL** on Manage Models. No API key or session required.++Copy the URL from Manage Models (it includes your database config id), or call:++```bash+curl -X GET "/api/pricing/export/<your-db-config-id>"+```++## SDK usage++Replace the host and database config id with values from your OpenLIT deployment and the Manage Models **SDK Pricing URL** bar.++<Tabs>+ <Tab title="Python">+ ```python+ import openlit++ openlit.init(+ otlp_endpoint="http://<otel-collector-host>:4318",+ pricing_json="https://<your-openlit-host>/api/pricing/export/<your-db-config-id>",+ )+ ```+ </Tab>+ <Tab title="TypeScript">+ ```typescript+ import openlit from "openlit";++ openlit.init({+ otlpEndpoint: "http://<otel-collector-host>:4318",+ pricingJson: "https://<your-openlit-host>/api/pricing/export/<your-db-config-id>",+ });+ ```+ </Tab>+ <Tab title="Go">+ ```go+ package main++ import openlit "github.com/openlit/openlit/sdk/go"++ func main() {+ openlit.Init(openlit.Config{+ OtlpEndpoint: "http://<otel-collector-host>:4318",+ PricingJson: "https://<your-openlit-host>/api/pricing/export/<your-db-config-id>",+ })+ }+ ```+ </Tab>+</Tabs>++<Warning>+ The path contains your database config id. The response is pricing only (no secrets or traces), but share the URL only with systems that should read your model prices.+</Warning>diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing.yml b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing.ymlnew file mode 100644index 00000000..46b64771--- /dev/null+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing.yml@@ -0,0 +1,34 @@+openapi: 3.0.0+info:+ title: Public SDK Pricing Export+ description: Unauthenticated pricing.json export for a database configuration id. Safe to pass to OpenLIT SDK pricing_json / pricingJson.+ version: 1.0.0+servers:+ - url: /+paths:+ /api/pricing/export/{dbConfigId}:+ get:+ summary: Public pricing export+ description: Returns SDK-compatible pricing for all models in that database config. No auth. Cached for 5 minutes.+ operationId: publicPricingExport+ parameters:+ - name: dbConfigId+ in: path+ required: true+ schema:+ type: string+ format: uuid+ responses:+ '200':+ description: pricing.json body+ content:+ application/json:+ schema:+ type: object+ example:+ chat:+ gpt-4o:+ promptPrice: 0.0025+ completionPrice: 0.01+ '404':+ description: Database config not founddiff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/prompt-hub/get.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/prompt-hub/get.mdxindex 0a4f5815..b093fa5a 100644--- a/docs/latest/openlit/developer-resources/api-reference/endpoint/prompt-hub/get.mdx+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/prompt-hub/get.mdx@@ -1,4 +1,75 @@---title: 'Get Prompt'openapi: 'POST /api/prompt/get-compiled'----\ No newline at end of file+---++## SDK usage++`url` and `api_key` (or `apiKey`) default to the `OPENLIT_URL` and `OPENLIT_API_KEY` environment variables if you don't pass them explicitly.++<Tabs>+ <Tab title="Python">+ ```python+ import openlit++ response = openlit.get_prompt(+ name="prompt_name",+ should_compile=True,+ variables={+ "name": "John",+ },+ )++ print(response)+ ```++ ```shell Output+ {+ err: null,+ res: {+ promptId: '88c4cbcd-87f9-4957-a37b-b41066e17471',+ name: 'prompt_name',+ version: '3.3.3',+ tags: [ 'user', 'greeting' ],+ metaProperties: { model: 'gpt4' },+ prompt: 'Hello {{name}}, how are you today?',+ compiledPrompt: 'Hello John, how are you today?'+ }+ }+ ```++ `url`, `api_key`, `name`, `prompt_id`, `version`, `should_compile`, `variables`, and `meta_properties` are all available parameters, matching the request body above.+ </Tab>+ <Tab title="TypeScript">+ ```typescript+ import Openlit from 'openlit';++ const response = await Openlit.getPrompts({+ name: "prompt_name",+ shouldCompile: true,+ variables: {+ name: "John",+ }+ });++ console.log(response);+ ```++ ```shell Output+ {+ err: null,+ res: {+ promptId: '88c4cbcd-87f9-4957-a37b-b41066e17471',+ name: 'prompt_name',+ version: '3.3.3',+ tags: [ 'user', 'greeting' ],+ metaProperties: { model: 'gpt4' },+ prompt: 'Hello {{name}}, how are you today?',+ compiledPrompt: 'Hello John, how are you today?'+ }+ }+ ```++ `url`, `apiKey`, `name`, `promptId`, `version`, `shouldCompile`, `variables`, and `metaProperties` are all available parameters, matching the request body above.+ </Tab>+</Tabs>\ No newline at end of filediff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/rule-engine.yml b/docs/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/rule-engine.ymlindex ad80ad05..648c63da 100644--- a/docs/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/rule-engine.yml+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/rule-engine.yml@@ -26,8 +26,7 @@ paths:enum:- context- prompt- - dataset- - meta_config+ - evaluationexample: contextdescription: Filter results to entities of this type.fields:@@ -99,12 +98,19 @@ paths:entity_data:type: objectadditionalProperties: true+ description: Full entity records, keyed as "entity_type:entity_id". Only populated for context and prompt entity types today - evaluation entities never populate entity_data.example:"context:b2c3d4e5-2345-6789-bcde-f01234567890":id: b2c3d4e5-2345-6789-bcde-f01234567890name: Premium System Prompt+ description: System prompt used for premium-tier users on gpt-4content: You are a helpful AI assistant for premium users...+ tags: ["premium", "system"]+ meta_properties: { model: gpt-4, region: us }status: ACTIVE+ created_by: [email protected]+ created_at: "2026-01-15T10:32:00Z"+ updated_at: "2026-02-01T08:10:00Z""400":description: Invalid request body or missing required fields.content:@@ -114,7 +120,7 @@ paths:properties:err:type: string- example: "entity_type is required and must be one of: context, prompt, dataset, meta_config"+ example: "entity_type is required and must be one of: context, prompt, evaluation""401":description: Missing or invalid API key.content:diff --git a/docs/latest/openlit/developer-resources/api-reference/endpoint/vault/get.mdx b/docs/latest/openlit/developer-resources/api-reference/endpoint/vault/get.mdxindex f367fbd3..6f954faa 100644--- a/docs/latest/openlit/developer-resources/api-reference/endpoint/vault/get.mdx+++ b/docs/latest/openlit/developer-resources/api-reference/endpoint/vault/get.mdx@@ -3,6 +3,71 @@ title: 'Get Secret(s)'openapi: 'POST /api/vault/get-secrets'---+## SDK usage++`url` and `api_key` (or `apiKey`) default to the `OPENLIT_URL` and `OPENLIT_API_KEY` environment variables if you don't pass them explicitly.++<Tabs>+ <Tab title="Python">+ ```python+ import openlit++ response = openlit.get_secrets(+ should_set_env=True+ )++ print(response)+ ```++ ```shell Output+ {+ err: null,+ res: { ANTHROPIC_API_KEY: 'ANTHROPIC_API_VALUE', OPENAI_API_KEY: 'OPENAI_API_VALUE' }+ }+ ```++ | Parameter | Description |+ |------------------|-------------------------------------------------------------------------------------------------------------------|+ | `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable. |+ | `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+ | `key` | Sets the key to fetch a specific secret. Optional |+ | `should_set_env` | Boolean value that sets all the secrets as environment variables for the application. Optional |+ | `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |++ `should_set_env` is client-side only: it doesn't change what's sent to OpenLIT, or what comes back in the response. It only controls whether the SDK, after receiving the secrets, also sets them directly as environment variables in your running process.+ </Tab>+ <Tab title="TypeScript">+ ```typescript+ import Openlit from 'openlit';++ const response = await Openlit.getSecrets({+ shouldSetEnv: true,+ });++ console.log(response);+ ```++ ```shell Output+ {+ err: null,+ res: { ANTHROPIC_API_KEY: 'ANTHROPIC_API_VALUE', OPENAI_API_KEY: 'OPENAI_API_VALUE' }+ }+ ```++ | Parameter | Description |+ |-----------------|---------------------------------------------------------------------------------------------------------------------|+ | `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable. |+ | `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |+ | `key` | Sets the key to fetch a specific secret. Optional |+ | `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |+ | `shouldSetEnv` | Boolean value that sets all the secrets as environment variables for the application. Optional |+ </Tab>+</Tabs>++<Note>+ This endpoint always responds successfully (HTTP 200), even for an invalid API key. Check the `err` field in the response body rather than relying on the HTTP status code alone to know whether the request actually succeeded.+</Note>+## Cross-origin browser requests`POST /api/vault/get-secrets` is an API-key authenticated endpoint for retrieving Vault secrets. Browser requests from a different origin are blocked unless the calling origin is explicitly allowed.diff --git a/docs/latest/openlit/developer-resources/api-reference/introduction.mdx b/docs/latest/openlit/developer-resources/api-reference/introduction.mdxindex 213fc877..b39d0f9b 100644--- a/docs/latest/openlit/developer-resources/api-reference/introduction.mdx+++ b/docs/latest/openlit/developer-resources/api-reference/introduction.mdx@@ -21,12 +21,20 @@ description: 'OpenAPI specification for API Endpoints in OpenLIT'href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate"></Card>+ <Card+ title="Manage Models API"+ icon="cube"+ href="/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models"+ >+ </Card></CardGroup>## Authentication-All API endpoints are authenticated using Authetication Header.+Most SDK-facing endpoints (Prompt Hub, Vault, Rule Engine) use an Authorization header:```yamlAuthorization: Bearer <api-key>-```\ No newline at end of file+```++[Manage Models](/latest/openlit/developer-resources/api-reference/endpoint/manage-models/models) UI APIs use a signed-in OpenLIT session instead. The [public pricing export](/latest/openlit/developer-resources/api-reference/endpoint/manage-models/public-pricing) requires no auth.\ No newline at end of filediff --git a/docs/latest/openlit/developer-resources/multiple-db.mdx b/docs/latest/openlit/developer-resources/multiple-db.mdxindex 8e8e86a5..0b16652a 100644--- a/docs/latest/openlit/developer-resources/multiple-db.mdx+++ b/docs/latest/openlit/developer-resources/multiple-db.mdx@@ -1,45 +1,66 @@----title: 'Connect Multiple Databases'-description: 'Connect and switch between multiple ClickHouse databases in OpenLIT'-sidebarTitle: 'Connect Multiple Databases'+title: "Connect Multiple Databases"+sidebarTitle: "Connect Multiple Databases"+description: "Connect multiple ClickHouse database configurations in one OpenLIT project and switch which one powers traces, dashboards, Prompt Hub, and Vault"+icon: "database"----OpenLIT allows you to add and switch between multiple ClickHouse databases, which helps manage observability data, prompts, and secrets separately for various environments. This feature enables users to maintain distinct databases tailored to specific environments or tenants, all while using a single OpenLIT deployment for efficient data analysis. The active database you select will power the data displayed across different sections of the application.+OpenLIT lets a single project hold **multiple database configurations** - each one is a ClickHouse connection (host, port, database, credentials) that you can select as the active source for observability and platform data. The active configuration powers traces, dashboards, Prompt Hub, Vault, Manage Models, and the rest of the app for your session.-## When to use this feature+Managing those connections lives under **Organisation → Project → Database configurations**. There is no standalone “database configuration” settings page - `/settings/database-config` redirects to Organisation.-This feature is particularly useful in the following scenarios:+## When to use multiple databases-- **Different Environments**: If you're running multiple environments, such as development, testing, and production, you can keep their data separate and easily switch between them as needed.+- **Environments** - keep development, staging, and production data on separate ClickHouse databases and switch between them without changing projects.+- **Isolation** - give different tenants or teams their own database configurations inside the same organisation structure.+- **One OpenLIT, many stores** - analyze several ClickHouse backends from a single OpenLIT deployment.-- **Multi-Tenancy**: When supporting multiple tenants, this feature allows you to maintain isolated datasets for each tenant, ensuring data privacy and organization.+Search interest for niche phrases like "multi-tenant observability platform" is too thin and noisy to lean on - the product job is simply: connect and switch multiple databases from one place.-- **Centralized Management**: Manage all your ClickHouse databases from a single OpenLIT deployment, simplifying your AI Engineering data architecture and reducing overhead.+## Add and switch configurations-## Get started+<Steps>+ <Step title="Open Organisation">+ Select **Organisation** (or **Projects**) from the sidebar under Settings.+ </Step>+ <Step title="Open a project">+ Open the project that should own the configuration, then switch to the **Database configurations** tab. Configurations are project-scoped - one created under project A is never selectable from project B.+ </Step>+ <Step title="Add New Config">+ Click **Add New Config** (or **Add database config** from the empty state). Fill in **Config Name**, **Environment**, **Username**, **Password**, **Host**, **Port**, **Database**, and optional **Query params**, then **Save**.+ </Step>+ <Step title="Mark it active">+ Use the checkbox on the list row (tooltip: **Mark … as the active database config**) or pick the config from the header **Databases** switcher. Only one is active for your user in that project at a time - teammates can select a different one.+ </Step>+</Steps>-<Frame>- <img src="/images/multiple-db.png" />-</Frame>+There is no **Test connection** button on the form - OpenLIT checks connectivity after a config is set active. The active database configuration drives what you see across the product, including dashboards, Telemetry, Prompt Hub, and Vault.-1. **Log in** to the OpenLIT.-2. Go to the **database configuration** page.-3. Click on the **Add new** option.-4. Enter the ClickHouse database configuration details.-5. Click **Save** to confirm.-6. Simply **enable** the desired database to view its data.-7. The selected active database will power data shown on the dashboards, requests page, Prompt Hub, and Vault secrets.+## Sharing and project boundaries++You can share a database configuration with specific teammates by email, with separate edit / delete / re-share permissions. For the full field list, selection behavior, and FAQs, see [Database Config](/latest/openlit/organisation/database-config) under Organisation.++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Can two projects share one database configuration?">+ No. A configuration belongs to the project it was created in. Point another project at the same ClickHouse instance by creating a new configuration there.+ </Accordion>+ <Accordion title="Do I need a new ClickHouse cluster for every configuration?">+ No. Multiple configurations can target the same cluster with different database names, or entirely different clusters.+ </Accordion>+ <Accordion title="Where did the Database Config settings page go?">+ `/settings/database-config` redirects to Organisation - open a project and use the **Database configurations** tab.+ </Accordion>+</AccordionGroup>----<CardGroup cols={3}>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking+<CardGroup cols={2}>+ <Card title="Database Config" href="/latest/openlit/organisation/database-config" icon="database">+ Field-level guide for adding and selecting ClickHouse configs</Card>- <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground" icon='flask'>- Compare cost, duration, and response tokens across different LLMs to find the most efficient model+ <Card title="Organisation" href="/latest/openlit/organisation/overview" icon="building">+ Organisations, projects, and how database configs fit the hierarchy</Card></CardGroup>diff --git a/docs/latest/openlit/developer-resources/vault.mdx b/docs/latest/openlit/developer-resources/vault.mdxdeleted file mode 100644index 75a2cf35..00000000--- a/docs/latest/openlit/developer-resources/vault.mdx+++ /dev/null@@ -1,155 +0,0 @@-----title: 'Vault'-description: 'Store and access sensitive information like LLM API keys securely'------Vault is designed to securely store and access sensitive information, such as API keys (e.g., `OPENAI_API_TOKEN`), eliminating the need to directly embed them in your code or environment variables. This centralized approach ensures that all developers can securely share and use the same API key without the necessity of distributing individual keys.--## Key features--- **Secrets Management**: Seamlessly create, edit, and monitor the secrets associated with your applications.-- **Secure Access**: Retrieve secrets based on keys or tags, and safely integrate them into your Node.js or Python environments.-- **Encrypted Storage**: Vault values are encrypted at rest with AES-256-GCM before they are stored.--## Security and cross-origin access--Vault secret retrieval uses API-key authentication and can be called from SDKs, REST clients, and server-side applications. Browser-based cross-origin calls are additionally restricted by CORS.--If your application is hosted on a different domain from OpenLIT and calls `POST /api/vault/get-secrets` from the browser, add the application origin to `OPENLIT_ALLOWED_CORS_ORIGINS`:--```bash-OPENLIT_ALLOWED_CORS_ORIGINS="https://app.example.com"-```--Multiple origins can be comma-separated:--```bash-OPENLIT_ALLOWED_CORS_ORIGINS="https://app.example.com,https://admin.example.com"-```--Server-to-server SDK and REST calls usually do not require CORS configuration because CORS is enforced by browsers. Do not use wildcard origins for secret retrieval endpoints.--<Note>- Vault values are encrypted with `OPENLIT_VAULT_ENCRYPTION_KEY`, or `NEXTAUTH_SECRET` if no dedicated Vault key is set. Keep the encryption key stable so existing secrets remain decryptable.-</Note>--## Get started--<Steps>- <Step title="List of secrets">- Get a quick overview of all secrets created.- 1. Navigate to the Vault in OpenLIT.- 2. Explore the available secrets listed.- <Frame>- <img src="/images/docs-vault-list.png" />- </Frame>- </Step>- <Step title="Create or edit a secret">- Build new secret with ease.- 1. Click on **Create new** button to create a new secret- 2. In the secret form, add the secret key, secret value and add tags to easily manage and access the secrets via sdk.- <Frame>- <img src="/images/docs-vault-create.png" />- </Frame>- </Step>- <Step title="Retrieve the secrets">- <Steps>- <Step title="Create an API Key">- To authenticate your requests, you need an API key. Here's how you can create one:-- - Go to the OpenLIT.- - Navigate to the **API Keys** page.- - Click on **Create API Key**.- - Enter a name for your API key.- - Save the API key displayed. Ensure you store it securely as it will be used for authentication in the SDK.- </Step>- <Step title="Get secrets using the SDK">- <Tabs>- <Tab title="Python">- Here's how you can fetch and set environment secret(s) in Python:-- ```python- import openlit-- response = openlit.get_secrets(- url="http://127.0.0.1:3000",- api_key="_OPENLIT_API_KEY",- should_set_env=True- )-- print(response)- ```-- ```shell Output- {- err: null,- res: { ANTHROPIC_API_KEY: 'ANTHROPIC_API_VALUE', OPENAI_API_KEY: 'OPENAI_API_VALUE' }- }- ```- ### SDK parameters-- Below are the parameters for use with the SDK, formatted to indicate whether each is required or optional:-- | Parameter | Description |- |-------------------|------------------------------------------------------------------------------------------------------------------------------------|- | `url` | Sets the Openlit URL. Defaults to the `OPENLIT_URL` environment variable. |- | `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |- | `key` | Sets the key to fetch a specific secret. Optional |- | `should_set_env` | Boolean value that sets all the secrets as environment variables for the application. Optional |- | `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |-- </Tab>- <Tab title="Typescript">- Here's how you can fetch and compile a prompt in Typescript/Javascript:-- ```typescript- import Openlit from 'openlit'; // Import the Openlit SDK-- const response = await Openlit.getSecrets({- shouldSetEnv: true, // Set the secrets as the environment variables for the application- });-- console.log(response); // Print or process the fetched and compiled prompt- ```-- ```shell Output- {- err: null,- res: { ANTHROPIC_API_KEY: 'ANTHROPIC_API_VALUE', OPENAI_API_KEY: 'OPENAI_API_VALUE' }- }- ```- ### SDK parameters-- Below are the parameters for use with the SDK, formatted to indicate whether each is required or optional:-- | Parameter | Description |- |------------------|------------------------------------------------------------------------------------------------------------------------------------|- | `url` | Sets the Openlit URL. Defaults to the `OPENLIT_URL` environment variable or `http://127.0.0.1:3000` if not set. |- | `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |- | `key` | Sets the key to fetch a specific secret. Optional |- | `tags` | Sets the tags for fetching only the secrets that have the mentioned tags assigned. Optional |- | `shouldSetEnv` | Boolean value that sets all the secrets as environment variables for the application. Optional |-- </Tab>- <Tab title="API">- To retrieve the secret via an API request, Refer to the [**GET Secret(s)**](/latest/openlit/developer-resources/api-reference/endpoint/vault/get) API Reference Documentation.- </Tab>- </Tabs>- </Step>- </Steps>- </Step>-</Steps>-------<CardGroup cols={3}>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking- </Card>- <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground" icon='flask'>- Compare cost, duration, and response tokens across different LLMs to find the most efficient model- </Card>-</CardGroup>diff --git a/docs/latest/openlit/developer-resources/vault/manage-secrets.mdx b/docs/latest/openlit/developer-resources/vault/manage-secrets.mdxnew file mode 100644index 00000000..9a1fbc45--- /dev/null+++ b/docs/latest/openlit/developer-resources/vault/manage-secrets.mdx@@ -0,0 +1,48 @@+---+title: "Manage Secrets"+description: "Create, edit, tag, and delete secrets from the Vault list page"+---++Vault has a single page - `/vault` - a list of every secret you've created. There's no separate create, edit, or detail page: creating and editing both happen through the same pop-up dialog on the list page.++## The Vault list++The list shows four columns:++- **Key** - the secret's identifier.+- **Created By** - who created the secret.+- **Last Updated On** - the formatted date it was last saved, or `-` if it hasn't been updated.+- **Actions** - an edit (pencil) icon that opens the edit dialog, and a delete (trash) icon.++Before any secrets exist, the page shows a "Get Started with Vault" empty state describing Vault as a centralized secret management system to store, retrieve, and manage your secrets, track secret usage, manage variables, and easily retrieve secrets across your applications - alongside four feature highlights: Secure Storage, API Access, User Tracking, and Update History.++The page header has a **Learn more** button that links to this documentation, and a **Create new** button. **Create new** only appears once your database connection is healthy.++## Creating a secret++1. Click **Create new**. This opens the **Create a new secret** dialog.+2. Fill in the fields:+ - **Key** - only capital letters, digits, and underscores are allowed. As you type, OpenLIT auto-formats the value: spaces become underscores, letters are uppercased, and any other character is stripped. This is the only required field.+ - **Value** - your secret value, shown as a password-style input. You can leave this blank and save a secret with no value at all - useful as a placeholder you fill in later.+ - **Tags** - optional free-form tags to help you find and filter the secret later.+3. Click **Create secret**.++While the request is in flight, OpenLIT shows a "Creating secret..." toast. On success you'll see "Created secret successfully!"; on failure, "Creation of secret failed!". If you reuse a Key that already exists, creation fails with "Secret name is taken!".++## Editing a secret++Click the edit (pencil) icon on a row to open the **Update a secret** dialog.++- **Key** is read-only once a secret exists, with the hint "Cannot update the key" - you can't rename a secret after creating it.+- **Value** is always shown blank when you open the dialog, since OpenLIT never sends a stored value back to the browser. Leave it blank to keep the existing value unchanged, or type a new value to overwrite it. There's no way to view a secret's current value in the UI once it's saved - this is by design, see [Security](/latest/openlit/developer-resources/vault/security).+- **Tags** can be changed freely.++Click **Update secret** to save. You'll see the same "Creating secret..." loading toast, followed by "Updated secret successfully!" or "Updation of secret failed!".++## Deleting a secret++Click the delete (trash) icon on a row. You'll be asked to confirm: "Are you sure you want to delete this secret?", with the warning "Deleting secrets might result in breaking application if they are getting used. Please confirm before deleting it." Deleting is permanent - any application still fetching that Key will start failing to find it.++## Using Vault secrets elsewhere in OpenLIT++You don't always need to visit the Vault page first. The Evaluation configuration screen, the Chat settings screen, and OpenGround's provider configuration each have an "API Key (from Vault)" field that lets you pick an existing Vault secret, or create a new one on the spot, without leaving that screen.diff --git a/docs/latest/openlit/developer-resources/vault/overview.mdx b/docs/latest/openlit/developer-resources/vault/overview.mdxnew file mode 100644index 00000000..f1b73d05--- /dev/null+++ b/docs/latest/openlit/developer-resources/vault/overview.mdx@@ -0,0 +1,57 @@+---+title: "Overview"+description: "Securely manage LLM API keys and other secrets in one place, then retrieve them at runtime from any application via SDK or API"+---++Vault is OpenLIT's built-in secrets manager for LLM API keys and other sensitive values. Instead of pasting an API key into every codebase, `.env` file, or teammate's laptop, you store it once in Vault and let any application fetch it at runtime - decoupling key rotation from redeploys.++## What you get++<CardGroup cols={3}>+ <Card title="Manage Secrets" href="/latest/openlit/developer-resources/vault/manage-secrets" icon='key'>+ Create, edit, tag, and delete secrets from the Vault list page+ </Card>+ <Card title="Security" href="/latest/openlit/developer-resources/vault/security" icon='shield-halved'>+ Encryption at rest, per-user access, and CORS for browser calls+ </Card>+ <Card title="Retrieve Secrets" href="/latest/openlit/developer-resources/vault/retrieve-secrets" icon='cloud-arrow-down'>+ Fetch secrets into your application with the Python or TypeScript SDK, or the REST API+ </Card>+</CardGroup>++## Why Vault++Vault has no direct built-in equivalent in most LLM observability tools - typically you're expected to manage API keys yourself, through your own environment variables or a general-purpose secrets manager. OpenLIT centralizes secrets so that any application (not just OpenLIT itself) can fetch them at runtime through the SDK or API. That means rotating a key in Vault takes effect the next time an application fetches it - no restarts or redeploys required.++Vault secrets are also usable directly inside OpenLIT: the Evaluation configuration screen, the Chat settings screen, and OpenGround's provider configuration all let you pick - or create on the spot - a Vault secret as their API key, instead of requiring you to visit the Vault page first.++---++<CardGroup cols={3}>+ <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>+ Create custom visualizations with flexible widgets, queries, and real-time AI monitoring+ </Card>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>+ Version, deploy, and collaborate on prompts with centralized management and tracking+ </Card>+ <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground/overview" icon='flask'>+ Compare cost, duration, and response tokens across different LLMs to find the most efficient model+ </Card>+</CardGroup>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is Vault used for?">+ Vault centrally stores secrets like LLM API keys, encrypted at rest with AES-256-GCM, so applications can retrieve them remotely via SDK or API instead of embedding keys in code or environment files.+ </Accordion>+ <Accordion title="Can secrets be rotated without redeploying?">+ Yes. Since applications fetch secrets from Vault at runtime, rotating a key in Vault takes effect on the next fetch - no restarts or redeploys needed.+ </Accordion>+ <Accordion title="Is Vault safe to call from a browser?">+ Vault enforces API-key authentication and CORS. Browser-based cross-origin calls require adding your app's origin to `OPENLIT_ALLOWED_CORS_ORIGINS`; server-to-server calls are unaffected.+ </Accordion>+ <Accordion title="Can I change a secret's Key after creating it, or view its Value later?">+ No to both, by design. A secret's Key becomes read-only once it's created - you'd need to create a new secret under a different Key instead. The Value is write-only: it's never sent back to the browser, so the Value field is always blank when you open a secret to edit it. Leave it blank to keep the existing value, or type a new one to overwrite it.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/developer-resources/vault/retrieve-secrets.mdx b/docs/latest/openlit/developer-resources/vault/retrieve-secrets.mdxnew file mode 100644index 00000000..abdd5577--- /dev/null+++ b/docs/latest/openlit/developer-resources/vault/retrieve-secrets.mdx@@ -0,0 +1,34 @@+---+title: "Retrieve Secrets"+description: "Fetch Vault secrets into your application with the Python or TypeScript SDK, or the REST API"+---++Once a secret exists in Vault, any application can fetch it at runtime - it doesn't have to be OpenLIT itself. Retrieval requires an OpenLIT API key, and you can filter what comes back by an exact Key or by one or more Tags.++## Create an API key++<Steps>+ <Step title="Create an API Key">+ To authenticate your requests, you need an API key:+ - Go to OpenLIT.+ - Navigate to the **API Keys** page.+ - Click on **Create API Key**.+ - Enter a name for your API key.+ - Save the API key displayed. Store it securely - it's used for authentication in the SDK or API calls below.+ </Step>+</Steps>++## Filtering++You can narrow down which secrets you get back:++- **By Key** - pass the exact Key of a single secret.+- **By Tags** - pass one or more tags. OpenLIT returns any secret matching at least one of the given tags.++Leave both out to retrieve all secrets you have access to.++## Get secrets using the SDK or API++<Card title="GET Secret(s) API reference" href="/latest/openlit/developer-resources/api-reference/endpoint/vault/get" icon='cloud-arrow-down'>+ Full request/response schema, plus Python and TypeScript SDK examples, parameter tables, and the CORS/error-handling notes+</Card>diff --git a/docs/latest/openlit/developer-resources/vault/security.mdx b/docs/latest/openlit/developer-resources/vault/security.mdxnew file mode 100644index 00000000..894a1bdd--- /dev/null+++ b/docs/latest/openlit/developer-resources/vault/security.mdx@@ -0,0 +1,41 @@+---+title: "Security"+description: "Encryption at rest, per-user access, and CORS for browser-based secret retrieval"+---++## Encryption at rest++Secret values are encrypted with AES-256-GCM before they're stored - they're never written in plaintext.++The encryption key comes from the `OPENLIT_VAULT_ENCRYPTION_KEY` environment variable. If that's not set, OpenLIT falls back to `NEXTAUTH_SECRET`.++<Warning>+ Keep whichever key you use stable. If the encryption key changes after secrets have been saved, those secrets can no longer be decrypted - you'll see an error asking you to either re-save the secret or restore the original key.+</Warning>++## Who can see a secret++Access to a secret depends on how it's being accessed:++- **From the Vault page itself**, secrets are per-user: only the person who created a secret can view, edit, or delete it there.+- **Via API key** (SDK or REST calls, see [Retrieve Secrets](/latest/openlit/developer-resources/vault/retrieve-secrets)), access isn't scoped to a single creator - any valid API key for your OpenLIT instance can retrieve secrets by Key or Tag.++## CORS for browser-based retrieval++CORS only matters if your application calls the secret-retrieval endpoint directly from browser JavaScript, from a different origin than your OpenLIT instance. It does not apply to server-to-server SDK or REST calls, since CORS is a browser-only mechanism enforced through the `Origin` header - which server-to-server requests don't send.++If you do need browser-based cross-origin retrieval:++- Add your application's origin to `OPENLIT_ALLOWED_CORS_ORIGINS`. Comma-separate multiple origins:++ ```bash+ OPENLIT_ALLOWED_CORS_ORIGINS="https://app.example.com,https://admin.example.com"+ ```++- Two other environment variables are also checked and merged into the same allow-list, so an origin listed in any of the three is allowed: `OPENLIT_ALLOWED_ORIGINS` and `NEXTAUTH_URL`.+- Calls made from the exact same origin as your OpenLIT instance are always allowed automatically, regardless of the allow-list.+- Do not use wildcard origins for secret retrieval.++<Note>+ Server-to-server calls made from the SDK or a backend REST client are not subject to CORS at all - you don't need to add anything to the allow-list for those.+</Note>diff --git a/docs/latest/openlit/evaluations/analytics.mdx b/docs/latest/openlit/evaluations/analytics.mdxnew file mode 100644index 00000000..ce0da131--- /dev/null+++ b/docs/latest/openlit/evaluations/analytics.mdx@@ -0,0 +1,40 @@+---+title: "Analytics"+description: "LLM evaluation analytics - pass rates, executions, total cost, and per-evaluator results under Monitor → Evaluations"+---++The **Analytics** tab on **Evaluations** (`/evaluations`) shows how your AI evals are performing over the selected time range. Use the filter bar to change the window.++If no evaluation engine is configured yet, you see **Set up Evaluations** with steps to open Configuration, enable evaluators, and turn on Auto Evaluation or run evals from Telemetry. CTA: **Open Configuration**.++## Summary tiles++| Tile | Meaning |+|------|---------|+| **Evaluations** / **Active** | Activity in the selected range |+| **Traces evaluated** | How many traces received evaluation results |+| **Executions** | Total evaluator runs |+| **Total cost** | Spend attributed to evaluations |+| **Avg pass rate** | Average pass rate across evaluators |+| **Scores marked as failed** | Failed scores in the range |+| **Auto executions** | Runs tagged from Auto Evaluation |+| **Evaluators** | Count of evaluators in play |++Charts include **Pass rate** and **Executions** over time (via the embedded evaluations dashboard).++## Evaluators table++Section title: **Evaluators**. Columns include **Evaluation**, **Matcher**, **Executions**, **Pass rate**, and **Pass rate trend**. Matcher shows linked Rule Engine rules when present, or `*` for default.++Empty copy: *No evaluation results in this time range.*++---++<CardGroup cols={2}>+ <Card title="Evaluators" href="/latest/openlit/evaluations/evaluators" icon="list-check">+ Enable or create the types behind these rows+ </Card>+ <Card title="Configuration" href="/latest/openlit/evaluations/configuration" icon="gear">+ Set the judge model and Auto Evaluation schedule+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/evaluations/configuration.mdx b/docs/latest/openlit/evaluations/configuration.mdxnew file mode 100644index 00000000..5f3a2438--- /dev/null+++ b/docs/latest/openlit/evaluations/configuration.mdx@@ -0,0 +1,71 @@+---+title: "Configuration"+description: "Configure the LLM evaluation engine, Vault API key, Auto Evaluation schedule, and sample rate under Monitor → Evaluations"+---++The **Configuration** tab on **Evaluations** (`/evaluations?tab=configuration`) sets the judge model and Auto Evaluation schedule used for online LLM evaluation. The page opens with a **Manual & Auto** info card explaining both paths. (`/evaluations/settings` and `/settings/evaluation` redirect here.)++## Evaluation engine++Under the **Configuration** card, choose:++| Field | Label | Notes |+|-------|-------|--------|+| Provider | **Provider** | From [Costs → Manage models](/latest/openlit/costs/manage-models/overview) |+| Model | **Model** | List from the selected provider, or a custom model name (`e.g. gpt-4o-mini or custom model name`) |+| API key | **API Key (Vault)** | Select a Vault secret, or **Create new** |++Save with **Save Changes** or **Create Config** depending on whether a config already exists.++## Auto Evaluation++**Auto Evaluation** runs enabled [evaluators](/latest/openlit/evaluations/evaluators) against newly ingested LLM traces on a schedule - no manual click or SDK call required.++| | Manual | Auto |+|---|---|---|+| Trigger | **Run Evaluation** on a trace, or SDK | Cron schedule |+| Scope | One trace (or one SDK call) | New LLM traces since the last run |+| Sampling | None | Configurable sample rate |+| Result tagging | `source: manual` | `source: auto` (or skipped when sampled out) |++<Steps>+ <Step title="Open Configuration">+ Go to **Monitor → Evaluations → Configuration**.+ </Step>+ <Step title="Choose the judge model">+ Select **Provider**, **Model**, and **API Key (Vault)**.+ </Step>+ <Step title="Enable auto evaluation">+ Toggle **Enable auto evaluation**.+ </Step>+ <Step title="Set sample rate">+ Enter **Sample rate (%)** - the percentage of eligible traces to evaluate (e.g. `10` for 10%). Sampling is deterministic for a given trace.+ </Step>+ <Step title="Set Cron schedule">+ Enter a standard cron expression (e.g. `*/15 * * * *` every 15 minutes).+ </Step>+ <Step title="Save">+ Click **Save Changes** / **Create Config**.+ </Step>+</Steps>++Which types actually run is controlled on the **Evaluators** tab (and via Rule Engine links). Manual evaluation from a trace still uses this same configured judge.++### After a restart++Active Auto Evaluation schedules are restored when OpenLIT starts again - no manual re-setup after a redeploy.++## Manual evaluation reminder++The **Manual Evaluation** card points you to Telemetry: open a trace and run evaluation from its Evaluation tab, or use **Go to Traces**. See [LLM-as-a-Judge](/latest/openlit/evaluations/llm-as-a-judge).++---++<CardGroup cols={2}>+ <Card title="Evaluators" href="/latest/openlit/evaluations/evaluators" icon="list-check">+ Choose which types Auto Evaluation runs+ </Card>+ <Card title="Analytics" href="/latest/openlit/evaluations/analytics" icon="chart-line">+ Review pass rates and auto execution volume+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/evaluations/evaluators.mdx b/docs/latest/openlit/evaluations/evaluators.mdxnew file mode 100644index 00000000..ee6cc261--- /dev/null+++ b/docs/latest/openlit/evaluations/evaluators.mdx@@ -0,0 +1,55 @@+---+title: "Evaluators"+description: "Enable built-in LLM evaluation types or create custom evaluators for hallucination, bias, toxicity, and your own criteria"+---++The **Evaluators** tab on **Evaluations** (`/evaluations?tab=evaluators`) lists evaluation types. Section title: **Evaluation types**. Description: *Enable built-in evaluators or create custom ones for your use case.*++## Built-in types++OpenLIT ships 11 built-in types. Each can be enabled or disabled independently and linked to [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) rules for conditional runs.++| Evaluation Type | Description | Default |+|----------------|-------------|---------|+| **Hallucination** | Factual inaccuracies, contradictions, fabricated information | Enabled |+| **Bias** | Discriminatory patterns across gender, ethnicity, age, religion | Enabled |+| **Toxicity** | Harmful, offensive, threatening, or hateful language | Enabled |+| **Relevance** | How well the response addresses the prompt | Disabled |+| **Coherence** | Logical flow and internal consistency | Disabled |+| **Faithfulness** | Alignment with provided context or source material | Disabled |+| **Safety** | Jailbreak, prompt injection, unsafe content | Disabled |+| **Instruction Following** | Adherence to instructions and formatting | Disabled |+| **Completeness** | Whether all parts of the query are addressed | Disabled |+| **Conciseness** | Avoids unnecessary verbosity | Disabled |+| **Sensitivity** | PII, credentials, confidential data exposure | Disabled |++Badges you may see: **Enabled** / **Disabled**, **Custom**, **Rule engine**. Rows can show **Linked from rules:** when Rule Engine entities point at that type.++<Info>+When context is provided via the Rule Engine, evaluations judge the LLM response against that context - not against real-world knowledge alone.+</Info>++## Custom evaluators++Click **Create custom evaluator** to open the create form (Type ID, Label, Description, Evaluation Prompt). Custom types appear under **Custom evaluation types**, run alongside built-in types in auto and manual evaluation, and can be deleted when you no longer need them.++Click a built-in or custom card to open its detail page (`/evaluations/types/[id]`) - enable/disable, edit the prompt, and link Rule Engine rules. From Analytics, an evaluator row opens `/evaluations/evaluators/[id]` for pass-rate and recent results (**Back to Analytics**, **Configure**).++Legacy URLs `/evaluations/types` and `/evaluations/settings` redirect to the Evaluators and Configuration tabs.++## Online vs offline++- **Online** - Auto Evaluation or **Run Evaluation** on a live trace (see [Configuration](/latest/openlit/evaluations/configuration) and [LLM-as-a-Judge](/latest/openlit/evaluations/llm-as-a-judge)).+- **Offline** - [Programmatic evaluations](/latest/openlit/evaluations/programmatic-evals) via the SDK, typically in CI/CD.+- **Human** - [Manual Feedback](/latest/openlit/evaluations/manual-feedback) on a trace (Good / Bad / Neutral).++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is LLM evaluation?">+ Scoring LLM outputs for quality, safety, and correctness - automatically with an LLM-as-a-judge, with custom types, or with human feedback - so you can catch issues in production or before you ship.+ </Accordion>+ <Accordion title="Can I create custom evaluation types?">+ Yes. Use **Create custom evaluator** on the Evaluators tab with your own prompt and criteria.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/evaluations/manual-feedback.mdx b/docs/latest/openlit/evaluations/manual-feedback.mdxnew file mode 100644index 00000000..aeb5c4e5--- /dev/null+++ b/docs/latest/openlit/evaluations/manual-feedback.mdx@@ -0,0 +1,39 @@+---+title: "Manual Feedback"+description: "Add human feedback - good, bad, or neutral, with an optional comment - directly on a trace"+---++Not every quality signal needs an LLM judge. Manual Feedback lets a human rate a specific trace directly, alongside the automated evaluations already running on it.++## How it works++1. Open any LLM trace in [Telemetry](/latest/openlit/observability/telemetry/traces) and click its **Evaluation** tab.+2. In the **Feedback** panel, rate the response **Good**, **Bad**, or **Neutral**.+3. Optionally add a comment explaining the rating.+4. Click **Submit Feedback**.++Your feedback is stored as its own evaluation entry (`manual_feedback`) in the same evaluation pipeline as the automated evaluators - so it appears in the trace's evaluation list right alongside Hallucination, Bias, and any other evaluator results, and rolls up into [Evaluations → Analytics](/latest/openlit/evaluations/analytics).++<Info>+Score follows the same convention as the automated evaluators (low score = no issue): **Good** stores `0`, **Neutral** stores `0.5`, and **Bad** stores `1`. Your comment is saved as the entry's explanation.+</Info>++## When to use it++- Spot-checking responses an automated evaluator didn't flag, or disagreeing with one that did+- Collecting reviewer or subject-matter-expert sign-off on specific traces+- Capturing qualitative notes that don't fit a built-in or custom evaluation type++---++<CardGroup cols={3}>+ <Card title="Evaluators" href="/latest/openlit/evaluations/evaluators" icon='book'>+ Built-in and custom evaluation types+ </Card>+ <Card title="LLM-as-a-Judge" href="/latest/openlit/evaluations/llm-as-a-judge" icon='gavel'>+ Automated scoring for hallucination, bias, toxicity, and more+ </Card>+ <Card title="Configuration" href="/latest/openlit/evaluations/configuration" icon='gear'>+ Auto Evaluation schedule and judge model+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/evaluations/overview.mdx b/docs/latest/openlit/evaluations/overview.mdxindex 2895cf13..643781bc 100644--- a/docs/latest/openlit/evaluations/overview.mdx+++ b/docs/latest/openlit/evaluations/overview.mdx@@ -1,88 +1,62 @@---title: "Overview"-description: "Automated Eval scoring with 11 evaluation types — hallucination, bias, toxicity, safety, and more"+description: "LLM evaluation in OpenLIT - AI evals for hallucination, bias, toxicity, and custom criteria under Monitor → Evaluations"----OpenLIT provides automated evaluation that helps you assess and monitor the quality, safety, and performance of your LLM outputs across development and production environments.+**Evaluations** (`/evaluations`) is OpenLIT's LLM evaluation surface under **Monitor**. Score production traces with an LLM-as-a-judge, enable built-in or custom evaluators, schedule Auto Evaluation, and review pass-rate analytics - or run the same criteria [programmatically via the SDK](/latest/sdk/quickstart-programmatic-evals) for offline CI/CD gates.-<video- autoPlay- muted- loop- controls- className="w-full aspect-video rounded-xl"- src="/images/evaluations.mp4"- >- </video>+Open it from **Monitor → Evaluations**. The page has three tabs:-## Why evaluations?--Evaluation is crucial for improving the accuracy and robustness of language models, ultimately enhancing user experience and trust in your AI applications. Here are the key benefits:--- **Quality & Safety Assurance**: Detect hallucinations, bias, toxicity, safety issues, and ensure consistent, reliable AI outputs-- **Performance Monitoring**: Track model performance degradation and measure response quality across different scenarios-- **Risk Mitigation**: Catch potential issues before they reach users and ensure compliance with safety standards-- **Cost Optimization**: Monitor cost-effectiveness and ROI of different AI configurations and model choices-- **Continuous Improvement**: Build data-driven insights for A/B testing, optimization, and iterative development--## 11 built-in evaluation types--OpenLIT includes 11 evaluation types out of the box. Each can be independently enabled, customized with custom prompts, and linked to Rule Engine rules for conditional evaluation.--| Evaluation Type | Description | Default |-|----------------|-------------|---------|-| **Hallucination** | Detects factual inaccuracies, contradictions, and fabricated information | Enabled |-| **Bias** | Monitors for discriminatory patterns across gender, ethnicity, age, religion | Enabled |-| **Toxicity** | Screens for harmful, offensive, threatening, or hateful language | Enabled |-| **Relevance** | Evaluates how well the response addresses the prompt | Disabled |-| **Coherence** | Assesses logical flow and internal consistency | Disabled |-| **Faithfulness** | Measures alignment with provided context or source material | Disabled |-| **Safety** | Detects jailbreak attempts, prompt injection, and unsafe content generation | Disabled |-| **Instruction Following** | Evaluates adherence to instructions, constraints, and formatting | Disabled |-| **Completeness** | Assesses whether all parts of the query are fully addressed | Disabled |-| **Conciseness** | Evaluates whether the response avoids unnecessary verbosity | Disabled |-| **Sensitivity** | Detects PII leakage, credentials, and confidential data exposure | Disabled |--<Info>-**Context is the source of truth**: When context is provided (via the Rule Engine), evaluations judge the LLM response against the provided context — not against real-world knowledge. This enables domain-specific evaluation where your custom knowledge is authoritative.-</Info>--### Custom evaluation types--Beyond the 11 built-in types, you can create custom evaluation types with your own prompts. Navigate to **Evaluations > Settings > Evaluation Types** and click **Create Custom Type**. Each custom type requires an id (lowercase with underscores), a label, a description, and an evaluation prompt. Custom types run alongside built-in types in both auto and manual evaluations, and can be linked to Rule Engine rules just like built-in types. Unlike built-in types, custom types can be deleted when no longer needed.--## AI evaluation methods--OpenLIT provides automated LLM evaluation and testing capabilities for production AI applications:--### Automated LLM-as-a-Judge--Zero-setup AI quality monitoring that automatically evaluates your LLM responses:--- **Production Monitoring**: Auto-evaluate every LLM response for quality and safety issues-- **Smart Scheduling**: Configure evaluation frequency with cron schedules for cost optimization-- **Real-time Scoring**: Instant evaluation results visible in trace details and dashboards-- **Rule Engine Integration**: Link evaluation types to rules for conditional evaluation based on trace attributes (model, provider, service, environment)-- **Context-Aware**: Evaluations use context from the Rule Engine as ground truth for judging responses--### Programmatic evaluations--Programmatic AI evaluation tools for custom testing workflows and development pipelines:--- **Code Integration**: Call LLM evaluators directly in your application code via the SDK-- **CI/CD Quality Gates**: Automated testing for model improvements and regression detection-- **Rule Engine SDK**: Use `evaluate_rule()` to fetch matching contexts and evaluation configs at runtime+<CardGroup cols={3}>+ <Card title="Analytics" href="/latest/openlit/evaluations/analytics" icon="chart-line">+ Pass rates, executions, cost, and evaluator results over time+ </Card>+ <Card title="Evaluators" href="/latest/openlit/evaluations/evaluators" icon="list-check">+ Enable built-in types or create custom evaluators+ </Card>+ <Card title="Configuration" href="/latest/openlit/evaluations/configuration" icon="gear">+ Judge model, Vault API key, Auto Evaluation schedule and sampling+ </Card>+</CardGroup>----+Default tab is **Analytics**. Use `?tab=evaluators` or `?tab=configuration` for the others.++## Find the right feature++| If you want to... | Use this |+|---|---|+| See pass rates and spend for evals | [Analytics](/latest/openlit/evaluations/analytics) |+| Turn on hallucination, bias, toxicity, or a custom type | [Evaluators](/latest/openlit/evaluations/evaluators) |+| Schedule Auto Evaluation or set the judge model | [Configuration](/latest/openlit/evaluations/configuration) |+| Score one trace on demand | [LLM-as-a-Judge](/latest/openlit/evaluations/llm-as-a-judge) from the trace Evaluation tab |+| Rate a trace yourself | [Manual Feedback](/latest/openlit/evaluations/manual-feedback) |+| Gate CI/CD on quality | [Programmatic evaluations](/latest/openlit/evaluations/programmatic-evals) |++```mermaid+flowchart TB;+ subgraph Online["Online evaluation - production traces"]+ direction LR;+ A[LLM / Agent call] -->|creates| B[Trace in OpenLIT];+ B -->|Auto Eval schedule or manual Run Evaluation| C[LLM Judge];+ C -->|score + classification + reasoning| D[Analytics and trace detail];+ end+ subgraph Offline["Offline evaluation - CI/CD"]+ direction LR;+ E[Dataset] -->|SDK evaluate call| F[LLM Judge];+ F -->|score| G[Pass / fail gate];+ end+```++## Methods<CardGroup cols={3}>- <Card title="LLM-as-a-Judge" href="/latest/openlit/evaluations/llm-as-a-judge" icon='gavel'>- Use advanced LLMs to evaluate AI application quality with automated scoring+ <Card title="LLM-as-a-Judge" href="/latest/openlit/evaluations/llm-as-a-judge" icon="gavel">+ Automated scoring on live traces</Card>- <Card title="Programmatic evaluations" href="/latest/sdk/quickstart-programmatic-evals" icon='bolt'>- Quick start guide for implementing custom evaluations in your code+ <Card title="Programmatic evals" href="/latest/openlit/evaluations/programmatic-evals" icon="bolt">+ SDK / CI offline evaluation</Card>- <Card title="Rule Engine" href="/latest/openlit/prompts-experiments/rule-engine" icon='gears'>- Define conditional rules to provide context and control which evaluations run+ <Card title="Manual Feedback" href="/latest/openlit/evaluations/manual-feedback" icon="thumbs-up">+ Good / Bad / Neutral human ratings</Card></CardGroup>diff --git a/docs/latest/openlit/evaluations/programmatic-evals.mdx b/docs/latest/openlit/evaluations/programmatic-evals.mdxindex d1134caf..5bf3829e 100644--- a/docs/latest/openlit/evaluations/programmatic-evals.mdx+++ b/docs/latest/openlit/evaluations/programmatic-evals.mdx@@ -10,6 +10,12 @@ import EvalsQuickstart from '/snippets/quickstart-programmatic-evals.mdx';---<CardGroup cols={3}>+ <Card title="Full SDK reference" href="/latest/sdk/features/evaluations" icon='book'>+ Parameters, result objects, batch evaluation, and Rule Engine attribute resolution in depth+ </Card>+ <Card title="CI/CD integration" href="/latest/sdk/features/evaluations#ci-cd-integration" icon='code-branch'>+ Use evaluations as quality gates in your test suite or pipeline+ </Card><Card title="LLM-as-a-judge" href="/latest/openlit/evaluations/llm-as-a-judge" icon='gavel'>Automatically add evaluation scoring to production traces</Card>@@ -17,13 +23,6 @@ import EvalsQuickstart from '/snippets/quickstart-programmatic-evals.mdx';60+ AI integrations with automatic instrumentation and performance tracking</Card><Card title="Destinations" href="/latest/sdk/destinations/overview" icon='link'>- Send elemetry to Datadog, Grafana, New Relic, and other observability stacks+ Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/openlit/installation.mdx b/docs/latest/openlit/installation.mdxindex 6794edb0..4e03bb60 100644--- a/docs/latest/openlit/installation.mdx+++ b/docs/latest/openlit/installation.mdx@@ -1,10 +1,11 @@----title: 'Self-hosting'-description: 'Guide to help you deploy your own instance of OpenLIT'+title: "Self-Host OpenLIT"+sidebarTitle: Self Host+description: "Self-host OpenLIT with Docker or Kubernetes. Deploy OpenLIT, ClickHouse, and an OpenTelemetry Collector on your own infrastructure."icon: 'circle-down'----OpenLIT is open source and is very easy and lightweight to self-host as it needs only 3 componets to run:+OpenLIT is open source and is very easy and lightweight to self-host as it needs only 3 components to run:1. OpenLIT itself2. ClickHouse for storage3. OpenTelemetry Collector for telemetry collection.@@ -71,26 +72,30 @@ For a quick and straightforward setup, Docker Compose can be used to deploy Open</Step></Steps>+## Frequently asked questions++<AccordionGroup>+ <Accordion title="What do I need to self-host OpenLIT?">+ Just Docker and Docker Compose for the quickest path, or a Kubernetes cluster with Helm if you're deploying at scale. Both methods deploy the same 3 components: OpenLIT, ClickHouse, and an OpenTelemetry Collector.+ </Accordion>+ <Accordion title="Can I reuse my existing ClickHouse or OpenTelemetry Collector?">+ Yes. OpenLIT can connect to infrastructure you already run instead of deploying its own ClickHouse or OTel Collector. See [Connect Multiple Databases](/latest/openlit/developer-resources/multiple-db) for setup.+ </Accordion>+ <Accordion title="Is OpenLIT free to self-host?">+ Yes. OpenLIT is licensed under Apache-2.0 - every feature in this documentation is free to self-host, with no license key or usage limits.+ </Accordion>+</AccordionGroup>+----<CardGroup cols={2}>- <Card title="Quickstart: LLM Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>+<CardGroup cols={3}>+ <Card title="Start tracing" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>Production-ready AI monitoring setup in 2 simple steps with zero code changes</Card>- <Card title="Configuration" href="/latest/openlit/configuration" icon='sliders'>- Configure OpenLIT deployment settings, environment variables, and database connections- </Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Use prompt management" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='shapes'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card>-</CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.+ <Card title="Set up evals" href="/latest/openlit/evaluations/llm-as-a-judge" icon='display'>+ Score live traces automatically with prebuilt LLM-as-a-judge evaluators</Card>+</CardGroup>diff --git a/docs/latest/openlit/oauth.mdx b/docs/latest/openlit/oauth.mdxindex a15f0ebf..f0bc367b 100644--- a/docs/latest/openlit/oauth.mdx+++ b/docs/latest/openlit/oauth.mdx@@ -222,17 +222,10 @@ NextAuth.js handles all session management including:<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>Create custom visualizations with flexible widgets, queries, and real-time AI monitoring</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='shapes'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card>- <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground" icon='flask'>+ <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground/overview" icon='display'>Compare cost, duration, and response tokens across different LLMs to find the most efficient model</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/openlit/observability/agents/analytics.mdx b/docs/latest/openlit/observability/agents/analytics.mdxnew file mode 100644index 00000000..5e8348b4--- /dev/null+++ b/docs/latest/openlit/observability/agents/analytics.mdx@@ -0,0 +1,16 @@+---+title: "Analytics"+description: "The Dashboard tab - an agent's own latency, cost, token usage, and error rate over time"+---++The **Dashboard** tab on an agent's detail page reuses OpenLIT's standard LLM dashboard, scoped down to just this agent: latency, cost, token usage, and error rate over time, with its own time-range filter.++## Version scoping++Unlike the [Overview tab's DAG](/latest/openlit/observability/agents/overview#overview-and-the-agent-graph), the Dashboard tab shows **all-versions, aggregate data by default** - it only narrows to a single version's traffic once you explicitly pick one from the [version chooser or drawer](/latest/openlit/observability/agents/versioning). This is intentional: cost and latency trends are usually most useful viewed across the agent's whole history, with the option to isolate a specific version when you're comparing before/after a prompt or model change.++---++<Card title="Monitoring" href="/latest/openlit/observability/agents/monitoring" icon='activity'>+ Drill from an aggregate trend down to the individual traces behind it+</Card>diff --git a/docs/latest/openlit/observability/agents/configuration.mdx b/docs/latest/openlit/observability/agents/configuration.mdxnew file mode 100644index 00000000..4818e8bd--- /dev/null+++ b/docs/latest/openlit/observability/agents/configuration.mdx@@ -0,0 +1,18 @@+---+title: "Configuration"+description: "Toggle LLM Observability and Agent Observability for an already-discovered agent"+---++The **Configuration** tab shows two toggles - **LLM Observability** and **Agent Observability** - that turn data capture for this agent on or off. The tab shows an attention indicator when an agent has been discovered but hasn't started sending telemetry yet, so you know instrumentation is still needed.++Toggling is blocked, with an explanatory tooltip, while the agent is stopped or mid-transition - there's no live process to attach or detach instrumentation from in that state.++## Version scoping++Configuration is **never version-scoped**. Selecting a different version in the header has no effect on this tab - the toggles reflect the agent's current, live state, not a historical snapshot. See [Versioning](/latest/openlit/observability/agents/versioning) for how the other tabs behave differently.++---++<Card title="Setup" href="/latest/openlit/observability/agents/setup" icon='rocket'>+ What it takes for an agent to be discovered in the first place+</Card>diff --git a/docs/latest/openlit/observability/agents/definition.mdx b/docs/latest/openlit/observability/agents/definition.mdxnew file mode 100644index 00000000..543bea4a--- /dev/null+++ b/docs/latest/openlit/observability/agents/definition.mdx@@ -0,0 +1,18 @@+---+title: "Definition"+description: "The system prompt and tools captured for an agent's currently selected version"+---++The **Definition** tab shows the **System Prompt** captured for the currently selected [version](/latest/openlit/observability/agents/versioning), with raw/formatted view toggles, a copy button, and a diff view against the previous version. Alongside it, the **Tools** card lists every tool the agent called during that version's traces - name, description, and JSON schema, captured directly from `gen_ai.tool.*` / function-call attributes, or a "schema not captured" placeholder if the schema wasn't emitted.++Because both cards are captured from live trace data rather than a static registry, the system prompt and tool list always match what the agent actually ran - not a hand-maintained description of it. If an agent hasn't emitted `gen_ai.system_instructions` or tool definitions yet, both cards show an empty state explaining what's missing.++## Version scoping++Definition always shows one specific version's snapshot - whichever is selected in the header, or the latest version if you haven't picked one. It never shows aggregate or all-versions data, since a system prompt and tool list only make sense as a single point-in-time capture.++---++<Card title="Versioning" href="/latest/openlit/observability/agents/versioning" icon='code-branch'>+ How versions are fingerprinted, and how selecting one scopes every tab+</Card>diff --git a/docs/latest/openlit/observability/agents/monitoring.mdx b/docs/latest/openlit/observability/agents/monitoring.mdxnew file mode 100644index 00000000..3dc5397e--- /dev/null+++ b/docs/latest/openlit/observability/agents/monitoring.mdx@@ -0,0 +1,16 @@+---+title: "Monitoring"+description: "The Monitoring tab - this agent's own traces and requests, filtered down from the Telemetry page's trace list"+---++The **Monitoring** tab on an agent's detail page is the same trace/request list used on the [Telemetry page's Traces tab](/latest/openlit/observability/telemetry/traces), scoped down to just this agent's traffic. It's a live view of the agent's own requests rather than a bespoke component - the same filters, columns, and trace detail drill-in apply here.++## Version scoping++Like the [Dashboard tab](/latest/openlit/observability/agents/analytics), Monitoring shows **all-versions traffic by default** and only narrows to a single version once you explicitly select one from the [version chooser or drawer](/latest/openlit/observability/agents/versioning).++---++<Card title="Telemetry: Traces" href="/latest/openlit/observability/telemetry/traces" icon='diagram-project'>+ The same trace list and detail view, unscoped to any one agent+</Card>diff --git a/docs/latest/openlit/observability/agents/overview.mdx b/docs/latest/openlit/observability/agents/overview.mdxnew file mode 100644index 00000000..a0fcac07--- /dev/null+++ b/docs/latest/openlit/observability/agents/overview.mdx@@ -0,0 +1,55 @@+---+title: "Overview"+description: "AI agent observability, built entirely from OpenTelemetry trace data - discover every agent and its live call graph with zero registration"+---++The **Agents** page (`/agents`) is OpenLIT's AI agent observability view: a single place to see every agent OpenLIT has observed, built entirely from your OpenTelemetry trace data - there's nothing to declare or register up front. As soon as an agent's spans (including tool calls, model calls, and system prompts) show up in your traces, OpenLIT surfaces it here. See [Setup](/latest/openlit/observability/agents/setup) for what it takes to get an agent showing up.++## Agents list++The **Applications** tab lists every agent and service instrumented with the OpenLIT SDK, discovered from `service.name` and `gen_ai.*` span attributes. Each row shows the service name, host system, runtime, providers in use, and when it was last seen. You can filter by System, Provider, or discovery Status.++Click any row to open its detail view, which has five tabs:++<CardGroup cols={3}>+ <Card title="Overview" href="/latest/openlit/observability/agents/overview#overview-and-the-agent-graph" icon='diagram-project'>+ Metadata grid and the trace-derived call graph+ </Card>+ <Card title="Analytics" href="/latest/openlit/observability/agents/analytics" icon='chart-line'>+ Latency, cost, tokens, and errors over time+ </Card>+ <Card title="Monitoring" href="/latest/openlit/observability/agents/monitoring" icon='activity'>+ The agent's own traces and requests+ </Card>+ <Card title="Definition" href="/latest/openlit/observability/agents/definition" icon='file-code'>+ Captured system prompt and tools+ </Card>+ <Card title="Configuration" href="/latest/openlit/observability/agents/configuration" icon='sliders'>+ LLM / Agent Observability toggles+ </Card>+</CardGroup>++Every tab can be scoped to a specific [version](/latest/openlit/observability/agents/versioning) of the agent - a trace-derived config snapshot that changes whenever the observed system prompt, tools, models, or runtime config change.++<Note>+Coding-assistant clients (Claude Code, Cursor, Codex, Windsurf) have their own tab and detail layout, since they're IDE/CLI clients rather than deployed services - see the dedicated [Coding Agents](/latest/openlit/coding-agents/overview) section.+</Note>++## Overview and the agent graph++The Overview tab starts with a metadata grid - Requests (24h), Primary Model, Models, Tools, Age, and Last Seen - followed by a **DAG (directed acyclic graph)** of the agent's call flow: nodes for span names and tool calls (derived from `gen_ai.tool.name` and function-call attributes on LLM spans), edges annotated with call count, p50 latency, and error rate.++This graph is computed live from ClickHouse: OpenLIT samples up to 500 recent traces for the agent and self-joins the spans to build an aggregate call graph. It is **entirely trace-derived** - there's no static or declared workflow definition behind it, so the graph always reflects what the agent actually did, not what it was designed to do. The graph always shows one specific version's traffic - the version selected in the header, or the latest version if you haven't picked one - see [Versioning](/latest/openlit/observability/agents/versioning) for how that differs from the other tabs.++---++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Do I need to register or declare my agents?">+ No. Every agent, its call graph, tools, and versions are discovered automatically from OpenTelemetry trace data - there's nothing to register up front.+ </Accordion>+ <Accordion title="How is the agent graph built?">+ OpenLIT samples up to 500 recent traces for the agent and self-joins the spans into an aggregate call graph, with nodes for span names and tool calls and edges annotated with call count, latency, and error rate.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/observability/agents/setup.mdx b/docs/latest/openlit/observability/agents/setup.mdxnew file mode 100644index 00000000..1d524129--- /dev/null+++ b/docs/latest/openlit/observability/agents/setup.mdx@@ -0,0 +1,36 @@+---+title: "Setup"+description: "What it takes for an agent to show up on the Agents page - zero code by default, explicit identity when you want it"+---++<Steps>+ <Step title="Instrument your application">+ The [OpenLIT SDK](/latest/openlit/quickstart-ai-observability) auto-instruments 90+ LLMs, agent frameworks, and vector databases with zero code changes. Any process emitting `gen_ai.*` spans under a given `service.name` shows up as its own row on the Applications tab - there's no separate agent registration step.+ </Step>+ <Step title="(Optional) Tag explicit agent identity and version">+ By default, OpenLIT infers agent identity from `service.name`. If you want explicit control - for example, multiple agents running in the same service - wrap the relevant code in `agent_context()` and `agent_version_context()`:++ ```python+ import openlit++ with openlit.agent_context(name="support-agent"):+ with openlit.agent_version_context(version="v3"):+ completion = client.chat.completions.create(...)+ ```++ These stamp `gen_ai.agent.name` and `gen_ai.agent.version` on every span inside the block, and are what power per-agent grouping and version fingerprints when you don't want to rely on inferred grouping from `service.name` alone. See [Custom Attributes](/latest/sdk/features/custom-attributes#tag-spans-with-agent-identity) for the full API.+ </Step>+ <Step title="Check the Agents page">+ Once spans land, the agent appears automatically on the Applications tab - refresh the [Agents page](/latest/openlit/observability/agents/overview) after a request or two. No restart or manual sync is needed.+ </Step>+</Steps>++## Enabling or disabling observability for an agent++Once an agent is discovered, whether LLM Observability and Agent Observability are actively capturing data for it is controlled per-agent from its own [Configuration tab](/latest/openlit/observability/agents/configuration) - not from a setup step here.++---++<Card title="Configuration" href="/latest/openlit/observability/agents/configuration" icon='sliders'>+ Toggle LLM / Agent Observability for an already-discovered agent+</Card>diff --git a/docs/latest/openlit/observability/agents/versioning.mdx b/docs/latest/openlit/observability/agents/versioning.mdxnew file mode 100644index 00000000..3868af0a--- /dev/null+++ b/docs/latest/openlit/observability/agents/versioning.mdx@@ -0,0 +1,28 @@+---+title: "Versioning"+description: "Trace-derived config snapshots, and exactly how selecting one scopes each tab on the agent detail page"+---++Every agent has a **version chooser** in the header and a **Version History** drawer listing all versions observed so far. Unlike a code or deployment version, an OpenLIT agent version is a **trace-derived config snapshot**: whenever the observed combination of system prompt, tools, models/providers, or runtime config (temperature, top_p, max_tokens) changes in incoming traces, OpenLIT fingerprints it into a new version and captures a snapshot of that configuration.++The version drawer lists each version's number, fingerprint, first seen / last seen timestamps, and request count, alongside a version timeline chart showing traffic per version over the current time window.++## How selecting a version scopes each tab++Selecting a version doesn't affect every tab the same way - some tabs default to showing everything and only narrow down once you pick a version; others always show a single version's data:++| Tab | Without an explicit selection | With a version selected |+|---|---|---|+| [Overview](/latest/openlit/observability/agents/overview) (the DAG) | Shows the **latest** version's call graph | Shows that version's call graph |+| [Analytics](/latest/openlit/observability/agents/analytics) (Dashboard) | Shows **all-versions** aggregate trends | Narrows to that version's traffic |+| [Monitoring](/latest/openlit/observability/agents/monitoring) | Shows **all-versions** traces | Narrows to that version's traces |+| [Definition](/latest/openlit/observability/agents/definition) | Shows the **latest** version's system prompt + tools | Shows that version's system prompt + tools |+| [Configuration](/latest/openlit/observability/agents/configuration) | Not version-scoped | Not version-scoped - always shows live state |++In other words: the Overview graph and Definition tab always default to *some* version (the latest one) even before you touch the chooser, while Analytics and Monitoring default to the *whole history* and only get version-scoped once you deliberately select one. This lets you compare how the agent behaved before and after a prompt or tool change without instrumenting version tracking yourself - pick a version to isolate its Dashboard/Monitoring data, or leave it on "latest" to inspect its current graph and definition without losing the aggregate trend view on the other tabs.++---++<Card title="Definition" href="/latest/openlit/observability/agents/definition" icon='file-code'>+ See exactly what a given version's system prompt and tools looked like+</Card>diff --git a/docs/latest/openlit/observability/error.mdx b/docs/latest/openlit/observability/error.mdxdeleted file mode 100644index 4dffcc31..00000000--- a/docs/latest/openlit/observability/error.mdx+++ /dev/null@@ -1,52 +0,0 @@-----title: 'Errors'-description: "View all error traces in one place with filtering and detailed exception information."------The Exceptions page shows all traces that contain errors or exceptions from your AI applications. You can filter by time range and exception type to quickly find and debug issues.--<Frame>- <img src="/images/errors-page.png" />-</Frame>--## Error traces--All error traces are automatically captured and displayed here, including:--- **LLM API errors**: Authentication, rate limits, model issues-- **Framework errors**: LangChain, LlamaIndex execution failures-- **Vector database errors**: Connection and query issues-- **Application errors**: Custom exceptions and validation errors--## Filtering--Use the time range selector to filter errors by period (24H, 7D, 1M, 3M, or custom range). Each error shows the trace ID, timestamp, span name, deployment type, and specific exception type.--## Exception details--Click any error to view the complete trace context with detailed exception information, stack traces, and execution flow.--<Frame>- <img src="/images/errors-trace.png" />-</Frame>-------<CardGroup cols={3}>- <Card title="Quickstart: LLM Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>- Production-ready AI monitoring setup in 2 simple steps with zero code changes- </Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Integrations" href="/latest/sdk/integrations/overview" icon='circle-nodes'>- 60+ AI integrations with automatic instrumentation and performance tracking- </Card>-</CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/openlit/observability/fleet-hub.mdx b/docs/latest/openlit/observability/fleet-hub.mdxdeleted file mode 100644index 11c52ad1..00000000--- a/docs/latest/openlit/observability/fleet-hub.mdx+++ /dev/null@@ -1,310 +0,0 @@-----title: "Fleet Hub"-description: "Centralized management and monitoring of OpenTelemetry collectors using OpAMP (Open Agent Management Protocol)"------<video- autoPlay- muted- loop- controls- className="w-full rounded-xl"- src="/images/fleet-hub.mp4"->-</video>--## Upgrade Information--<Warning>-**Important Upgrade Notice**: When upgrading from OpenLIT versions prior to 1.15.0 (which introduced Fleet Hub), special attention is required for Docker Compose deployments.-</Warning>--### Docker Compose Upgrade--If you're running OpenLIT using Docker Compose, you **must** use the `--remove-orphans` flag when upgrading:--```bash-# Upgrade with orphan removal-docker-compose up -d --remove-orphans-```--**Why this is necessary:**-- The OpenTelemetry Collector has been integrated directly into the OpenLIT container-- The standalone `otel-collector` container is no longer needed-- Without `--remove-orphans`, the old collector container will conflict with the new integrated collector-- Both containers attempt to bind to the same ports (4317, 4318), causing startup failures--### Migration Steps--1. **Stop Current Deployment**- ```bash- docker-compose down- ```--2. **Pull Latest Images**- ```bash- docker-compose pull- ```--3. **Start with Orphan Removal**- ```bash- docker-compose up -d --remove-orphans- ```--4. **Verify Fleet Hub Access**- - Navigate to Fleet Hub in the OpenLIT UI- - Confirm the integrated collector is running- - Update any existing collector configurations to connect to the new OpAMP endpoint--### What Changed in Version 1.15.0--- **Integrated Collector**: OpenTelemetry Collector now runs inside the OpenLIT container-- **OpAMP Server**: New OpAMP server for Fleet Hub functionality-- **Port Consolidation**: OTLP ports (4317, 4318) now served by the main OpenLIT container-- **Configuration Management**: Centralized configuration through Fleet Hub UI--<Note>-After upgrading, existing OpenTelemetry Collectors in your infrastructure can be connected to Fleet Hub by configuring them with OpAMP supervisors pointing to your OpenLIT instance.-</Note>--## Overview--Fleet Hub is OpenLIT's centralized management system for OpenTelemetry collectors across your infrastructure. Built on the **OpAMP (Open Agent Management Protocol)** standard, Fleet Hub provides real-time monitoring, configuration management, and health tracking for all your collectors from a unified dashboard.--<Note>-Fleet Hub uses the industry-standard OpAMP protocol, ensuring compatibility with OpenTelemetry ecosystem tools and future-proofing your observability infrastructure.-</Note>--## Key Features--### 🔍 **Real-time Monitoring**-- Live health status of all connected collectors-- Component-level health monitoring-- Performance metrics and resource usage-- Connection status and uptime tracking--### ⚙️ **Configuration Management**-- Centralized configuration updates-- Real-time configuration deployment-- Configuration validation and rollback-- Custom instance-specific configurations--### 📊 **Comprehensive Dashboard**-- Unified view of all collectors-- Detailed collector information (OS, architecture, version)-- Health status indicators with error details-- Searchable and filterable collector list--### 🔧 **OpAMP Integration**-- Standards-compliant OpAMP implementation-- Secure WebSocket connections with TLS support-- Automatic agent discovery and registration-- Bi-directional communication for configuration and status--## How It Works--Fleet Hub operates through three main components:--1. **OpAMP Server**: Built into OpenLIT, handles agent connections and configuration distribution-2. **OpAMP Supervisor**: Manages the OpenTelemetry Collector lifecycle on each host-3. **Fleet Hub UI**: Web interface for monitoring and managing collectors--```mermaid-graph TB- A[Fleet Hub UI] --> B[OpAMP Server]- B --> C[OpAMP Supervisor 1]- B --> D[OpAMP Supervisor 2]- B --> E[OpAMP Supervisor N]- C --> F[OTel Collector 1]- D --> G[OTel Collector 2]- E --> H[OTel Collector N]-- F --> I[Your Applications]- G --> J[Your Applications]- H --> K[Your Applications]-```--## Getting Started--### Prerequisites--- OpenLIT platform deployed and running-- OpenTelemetry Collector with OpAMP supervisor support-- Network connectivity between collectors and OpenLIT--### Step 1: Access Fleet Hub--Navigate to Fleet Hub in your OpenLIT dashboard:--1. Open your OpenLIT instance-2. Go to **Fleet Hub** from the main navigation-3. You'll see the Fleet Hub dashboard with any connected collectors--### Step 2: Configure OpAMP Supervisor--On each host where you want to monitor collectors, set up the OpAMP supervisor:--<CodeGroup>--```yaml supervisor.yaml-server:- # Replace with your OpenLIT instance URL- endpoint: wss://your-openlit-instance:4320/v1/opamp- tls:- insecure_skip_verify: false # Set to true for development--agent:- # Path to your OpenTelemetry Collector binary- executable: /usr/local/bin/otelcol-contrib-- # Configuration files for the collector- config_files:- - /etc/otel/otel-collector-config.yaml--capabilities:- accepts_remote_config: true- reports_effective_config: true- reports_own_metrics: false- reports_own_logs: true- reports_own_traces: false- reports_health: true- reports_remote_config: true--storage:- directory: ./storage-```--```bash Start Supervisor-# Download OpAMP supervisor-curl -LO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/opampsupervisor_linux_amd64--# Make executable-chmod +x opampsupervisor_linux_amd64--# Start with configuration-./opampsupervisor_linux_amd64 --config supervisor.yaml-```--</CodeGroup>--### Step 3: Monitor Your Fleet--Once connected, you can:--- **View all collectors** in the Fleet Hub dashboard-- **Monitor health status** with real-time updates-- **Inspect configurations** both custom and effective-- **Update configurations** remotely through the UI-- **Track performance** and connection metrics--## Using Fleet Hub--### Dashboard Overview--The Fleet Hub dashboard displays:--- **Collector List**: All connected collectors with key information-- **Health Status**: Visual indicators for healthy/unhealthy collectors-- **System Information**: OS type, architecture, version details-- **Connection Details**: Start time, uptime, and connection status--### Managing Collectors--#### Viewing Collector Details--Click on any collector to view:-- Detailed system information-- Component health status-- Current configuration (read-only)-- Custom configuration editor-- Connection and performance metrics--#### Updating Configurations--1. Select a collector from the list-2. Navigate to the **Configuration** section-3. Edit the custom configuration in the YAML editor-4. Click **Save** to deploy changes-5. Monitor the **Effective Configuration** for applied changes--<Warning>-Configuration changes are applied immediately. Always validate your YAML syntax before saving to avoid collector disruption.-</Warning>--#### Health Monitoring--Fleet Hub provides multi-level health monitoring:--- **Overall Health**: Collector-wide health status-- **Component Health**: Individual component status (receivers, processors, exporters)-- **Error Details**: Specific error messages for troubleshooting--## Architecture Details--### OpAMP Protocol--Fleet Hub implements the OpenTelemetry Agent Management Protocol (OpAMP) which provides:--- **Standardized Communication**: Industry-standard protocol for agent management-- **Secure Connections**: WebSocket with TLS encryption-- **Bi-directional Messaging**: Real-time status reporting and configuration updates-- **Capability Negotiation**: Flexible feature support based on agent capabilities--### Security Features--- **TLS Encryption**: All communications encrypted in transit-- **Certificate Management**: Automatic certificate handling-- **Authentication**: Secure agent identification and verification-- **Access Control**: Role-based access to configuration management--### Scalability--Fleet Hub is designed to handle:-- **Multiple Collectors**: Hundreds of collectors per instance-- **Real-time Updates**: Low-latency status and configuration synchronization-- **High Availability**: Resilient to network interruptions and reconnections--## Troubleshooting--### Common Issues--#### Collector Not Appearing in Fleet Hub--1. **Check Network Connectivity**- ```bash- # Test connection to OpAMP endpoint- telnet your-openlit-instance 4320- ```--2. **Verify Supervisor Configuration**- ```bash- # Check supervisor logs- ./opampsupervisor --config supervisor.yaml --verbose- ```--3. **Validate TLS Settings**- - Ensure TLS configuration matches between supervisor and server- - Check certificate validity and trust chains--#### Configuration Updates Not Applied--1. **Check Supervisor Logs**: Look for configuration parsing errors-2. **Validate YAML Syntax**: Ensure configuration is valid YAML-3. **Verify Permissions**: Ensure supervisor has write access to config files-4. **Monitor Effective Config**: Check if changes appear in effective configuration--#### Health Status Issues--1. **Component Failures**: Check individual component health in the dashboard-2. **Resource Constraints**: Monitor system resources (CPU, memory, disk)-3. **Network Issues**: Verify connectivity to downstream services-4. **Configuration Errors**: Review collector logs for configuration issues--### Getting Help--If you encounter issues with Fleet Hub:--1. **Check Logs**: Review both OpenLIT and supervisor logs-2. **Community Support**: Join our [Slack community](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ)-3. **GitHub Issues**: Report bugs on our [GitHub repository](https://github.com/openlit/openlit)-4. **Documentation**: Check the [OpAMP specification](https://github.com/open-telemetry/opamp-spec) for protocol detailsdiff --git a/docs/latest/openlit/observability/metrics.mdx b/docs/latest/openlit/observability/metrics.mdxdeleted file mode 100644index fa3a30be..00000000--- a/docs/latest/openlit/observability/metrics.mdx+++ /dev/null@@ -1,39 +0,0 @@-----title: 'Metrics'-description: "View and analyze OpenTelemetry metrics in OpenLIT"------<video- autoPlay- muted- loop- controls- className="w-full aspect-video rounded-xl"- src="/images/dashboards.mp4"- >- </video>--The OpenLIT platform allows you to view OpenTelemetry metrics sent from your AI applications through **Dashboard Widgets**. While there's no dedicated metrics explorer page, you can create comprehensive dashboards with customizable widgets to visualize the metrics data stored in ClickHouse, including LLM performance, cost analysis, token usage, and system health metrics.--Navigate to the **Dashboards** section to build personalized metric widgets with various chart types, filters, and real-time visualization of your OpenTelemetry metrics data.-------<CardGroup cols={3}>- <Card title="Quickstart: LLM Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>- Production-ready AI monitoring setup in 2 simple steps with zero code changes- </Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Integrations" href="/latest/sdk/integrations/overview" icon='circle-nodes'>- 60+ AI integrations with automatic instrumentation and performance tracking- </Card>-</CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/openlit/observability/telemetry/ai-analysis.mdx b/docs/latest/openlit/observability/telemetry/ai-analysis.mdxnew file mode 100644index 00000000..7a252cbf--- /dev/null+++ b/docs/latest/openlit/observability/telemetry/ai-analysis.mdx@@ -0,0 +1,55 @@+---+title: "AI Analysis"+description: "Turn a trace or span into a structured, AI-generated review of performance, reliability, cost, and token efficiency"+---++Every trace and span in the [Traces detail view](/latest/openlit/observability/telemetry/traces) has an **AI Analysis** tab. Use it to turn raw OpenTelemetry data into a structured review instead of reading spans by hand.++Analysis runs at two scopes:++- **Trace hierarchy analysis** (the AI Analysis tab in the left-hand view-mode row): reviews the root span and every child span together, so you can understand the full request - agent path, tool calls, model calls, retries, errors, and cost drivers.+- **Individual span analysis** (the AI Analysis tab in the right-hand attributes panel): focuses on the single span currently selected, for when you need to inspect one model call, tool execution, retrieval step, error, or latency hotspot.++## Analysis dimensions++Each run is broken into dimension tabs:++- **Strengths**: what the trace did well - efficient paths, good cache usage, appropriate model choice.+- **Improvements**: concrete, actionable changes, each with a severity, an explanation, and a suggested fix shown as a word-diff preview.+- **Wrong turns**: retries, repeated span names, and other signs the agent backtracked or looped.+- **Cost**: cost per call, the most expensive span, and where spend concentrates.+- **Token efficiency**: input/output/cache/reasoning token usage, duplicate tool or retrieval inputs, and repeated context.+- **Path**: the request's execution path across services, tools, databases, and HTTP calls.++## Run analysis from trace details++1. Open a trace from the Traces page or from a trace widget.+2. Select the **AI Analysis** tab - in the left panel for the whole trace, or in the right panel for the currently selected span.+3. Click **Analyze**. Progress streams in as the analysis runs.+4. When the run completes, it's saved to a run history tab so you can revisit it without re-running. Use **Re-run** only when the trace data or your investigation question has changed.++Each finding card includes a **Try in Chat** button that hands the finding off to Otter with context pre-filled, so you can keep investigating conversationally instead of staying in the panel.++## Run analysis from Otter++You can also ask [Otter](/latest/openlit/chat/overview) to analyze traces or spans by natural language - Otter has dedicated tools for running or retrieving AI Analysis runs, including by session, user, or any custom span attribute:++```+"Analyze this trace hierarchy"+"Review span <span_id> for token waste"+"Find cost optimization opportunities for traces where session.id is dc7b1ba1-..."+"Analyze the slowest traces from the last 24 hours"+```++When Otter references a trace or span in its answer, OpenLIT renders clickable trace/span pills so you can jump back into the trace detail view.++---++<CardGroup cols={2}>+ <Card title="Traces & Exceptions" href="/latest/openlit/observability/telemetry/traces" icon='diagram-project'>+ Where the AI Analysis tab lives inside the trace detail view+ </Card>+ <Card title="Chat with Otter" href="/latest/openlit/chat/conversations" icon='comment'>+ Ask Otter to run analysis, find traces, or explain findings in natural language+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/observability/telemetry/logs.mdx b/docs/latest/openlit/observability/telemetry/logs.mdxnew file mode 100644index 00000000..597d564a--- /dev/null+++ b/docs/latest/openlit/observability/telemetry/logs.mdx@@ -0,0 +1,20 @@+---+title: "Logs"+description: "A searchable, filterable explorer for OTel log records ingested through OpenLIT's built-in OpenTelemetry Collector"+---++Navigate to **Telemetry > Logs** for a dedicated logs explorer - a searchable, filterable list of ingested log records with a detail view per log entry.++OpenLIT ingests OTel logs the same way it ingests traces and metrics: through the built-in OpenTelemetry Collector's `otlp` receiver (gRPC `:4317`, HTTP `:4318`), which exports them into the `otel_logs` table in ClickHouse. Point your application's log exporter at the same OTLP endpoint you use for traces and metrics to have logs show up here automatically - no separate pipeline is required.++The logs list shows, per entry: Time, Severity, Service, Body, Trace ID, and Span ID. Like traces and metrics, logs support attribute-based filtering - Log Attributes, Resource Attributes, Scope Attributes, or top-level Field - so you can narrow down to a specific service, severity, or custom field before inspecting individual entries.++## Log detail view++Opening a log entry shows a severity badge, meta tiles (Time, Service, Event, Cost, Tokens), a log summary grid (body, duration, input/output/cache tokens, request ID, model, source, session, prompt ID, terminal, user - with links out to the parent trace and span where available), and a tabbed attributes panel for Log Attributes, Resource Attributes, and Scope Attributes.++---++<Card title="Traces & Exceptions" href="/latest/openlit/observability/telemetry/traces" icon='diagram-project'>+ Jump from a log's linked trace/span IDs into the full trace detail view+</Card>diff --git a/docs/latest/openlit/observability/telemetry/metrics.mdx b/docs/latest/openlit/observability/telemetry/metrics.mdxnew file mode 100644index 00000000..37f72608--- /dev/null+++ b/docs/latest/openlit/observability/telemetry/metrics.mdx@@ -0,0 +1,27 @@+---+title: "Metrics"+description: "A dedicated explorer for every OTel metric OpenLIT has received, with a trend chart and data-point detail per metric"+---++Navigate to **Telemetry > Metrics** for a dedicated metrics explorer - a list of every metric OpenLIT has received, with a detail view per metric (`/telemetry/metrics/<name>`).++The metrics list shows, per metric: Metric name, Type, Service, Unit, Latest value, number of Points, and Last Seen. Each metric can be filtered by:++- **Attributes** - the metric's own data-point attributes.+- **Resource attributes** - service, environment, and other resource-level attributes.+- **Scope attributes** - instrumentation scope metadata.+- **Field** - top-level metric fields (name, unit, type).++Metrics cover gauge, sum, histogram, summary, and exponential-histogram types, matching the OpenTelemetry metrics data model.++## Metric detail view++Opening a metric shows summary tiles (Latest, Total, Average, Loaded Points, Last Seen), a **Metric Trend** area chart of its data points over time, and a tabbed attributes panel: Metric Attributes, Resource Attributes, Scope Attributes, plus raw **Points** and **Series** tabs for the underlying data.++You can still build custom **Dashboard Widgets** on top of the same metrics data for cross-signal visualizations, trend charts, and alert-style stat tiles - the Metrics explorer and Dashboards read from the same underlying data and are complementary, not either/or.++---++<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>+ Build cross-signal visualizations and trend charts on top of the same metrics data+</Card>diff --git a/docs/latest/openlit/observability/telemetry/overview.mdx b/docs/latest/openlit/observability/telemetry/overview.mdxnew file mode 100644index 00000000..01775d43--- /dev/null+++ b/docs/latest/openlit/observability/telemetry/overview.mdx@@ -0,0 +1,62 @@+---+title: "Overview"+description: "LLM observability for traces, metrics, logs, and exceptions - all in one Telemetry page, built on OpenTelemetry"+---++<video+ autoPlay+ muted+ loop+ controls+ className="w-full aspect-video rounded-xl"+ src="/images/trace-details.mp4"+ alt="Demo of viewing trace details in OpenLIT"+>+</video>++The **Telemetry** page is the single home for every OpenTelemetry signal OpenLIT ingests from your AI applications. Navigate to it at `/telemetry`. It has four tabs, all backed by the same ClickHouse tables (`otel_traces`, `otel_metrics_*`, `otel_logs`) populated by the OpenTelemetry Collector built into OpenLIT:++<CardGroup cols={2}>+ <Card title="Traces" href="/latest/openlit/observability/telemetry/traces" icon="diagram-project">+ Distributed traces with grouping, a five-view trace detail explorer, and span/resource attributes+ </Card>+ <Card title="AI Analysis" href="/latest/openlit/observability/telemetry/ai-analysis" icon="sparkles">+ AI-generated review of a trace or span across strengths, improvements, cost, and token efficiency+ </Card>+ <Card title="Metrics" href="/latest/openlit/observability/telemetry/metrics" icon="chart-line">+ Every OTel metric OpenLIT has received, with a trend chart and data-point detail per metric+ </Card>+ <Card title="Logs" href="/latest/openlit/observability/telemetry/logs" icon="file-lines">+ Searchable, filterable OTel log records with a detail view per entry+ </Card>+</CardGroup>++The **Exceptions** tab is documented alongside Traces since it reuses the same trace detail explorer - see [Traces & Exceptions](/latest/openlit/observability/telemetry/traces).++---++<CardGroup cols={3}>+ <Card title="Quickstart: LLM Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>+ Production-ready AI monitoring setup in 2 simple steps with zero code changes+ </Card>+ <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>+ Create custom visualizations with flexible widgets, queries, and real-time AI monitoring+ </Card>+ <Card title="Integrations" href="/latest/sdk/integrations/overview" icon='circle-nodes'>+ 60+ AI integrations with automatic instrumentation and performance tracking+ </Card>+</CardGroup>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is AI observability?">+ AI observability is the practice of collecting traces, metrics, and logs from LLM applications and agents so you can understand behavior, cost, latency, and errors in production - OpenLIT does this via standard OpenTelemetry data.+ </Accordion>+ <Accordion title="Does OpenLIT support log monitoring, not just traces?">+ Yes. Logs have their own dedicated explorer under Telemetry, ingested the same way as traces and metrics through the built-in OpenTelemetry Collector's OTLP receiver.+ </Accordion>+ <Accordion title="Can I still use dashboards alongside the Metrics explorer?">+ Yes. The Metrics explorer and Dashboard Widgets read from the same underlying data - use the explorer to inspect a single metric, and dashboards for cross-signal, trend-style visualizations.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/observability/telemetry/traces.mdx b/docs/latest/openlit/observability/telemetry/traces.mdxnew file mode 100644index 00000000..06ebb0db--- /dev/null+++ b/docs/latest/openlit/observability/telemetry/traces.mdx@@ -0,0 +1,90 @@+---+title: "Traces & Exceptions"+description: "Filter and group traces, then drill into a five-view trace detail explorer with span attributes, chat view, timeline, and DAG"+---++Navigate to **Telemetry > Traces** (`/telemetry`) to view all distributed traces from your AI applications with detailed span analysis and execution flow. You can also create custom trace widgets in your dashboards to monitor specific trace metrics, latency trends, and performance insights alongside other observability data.++## Filtering++The Traces list supports filtering by Models, Providers, Max Cost, Span Names, Environments, and Services, plus a time range and result limit. You can also filter on custom attributes - Span Attributes, Resource Attributes, or any top-level trace Field.++The trace table shows, per row: Span/Trace ID, Time, Span Name, Duration, Service Name, Application Name, Cost, Total Tokens, Model, Provider, and Vector Count. Clicking a row opens the trace detail view in a resizable side sheet, with a **Full Screen** button that opens the same view at `/telemetry/traces/<id>`.++## Group traces++Use **Group By** on the Traces page to roll up large trace lists into meaningful groups before drilling into individual spans. Grouping works with the selected time range and any active filters, so you can narrow the dataset first and then compare trace segments.++You can group traces by:++- **Model**: Compare requests by `gen_ai.request.model`.+- **Provider**: Compare requests by `gen_ai.system`.+- **Span Names**: Group repeated operations or framework steps.+- **Application**: Compare services using the `service.name` resource attribute.++For anything else, use **Custom** to group by any span attribute or resource attribute key present in your trace data - pick the attribute type (Span Attributes or Resource Attributes) and the key, separately from the four preset options above.++Grouped rows show the number of spans in each group, total cost, token usage, and average duration. Click a group row to drill into the matching traces. The breadcrumb above the table shows the current grouping path and lets you return to all groups or remove grouping.++<Note>+ Grouping is best for finding high-volume models, expensive providers, slow span types, or application-level hotspots before opening an individual trace.+</Note>++Grouping can be combined with the existing trace filters. For example, you can filter to a single environment, apply a maximum cost threshold, and then group by model to find which models dominate that filtered slice.++## Trace detail view++Opening a trace splits the screen into two panels.++### Left panel: five view modes++The left panel is the **Span Hierarchy Explorer**, with five tabs:++- **Tree**: The classic nested span hierarchy, one row per span with its children indented underneath.+- **Chat**: A conversational rendering of the trace - messages, tool calls, and thinking steps laid out like a chat transcript instead of a span tree. Traces from coding agents (Claude Code, Cursor, Codex, Windsurf) open on this view by default, since a conversational read is more useful than a span tree for those traces.+- **AI Analysis**: Runs the [AI Analysis](/latest/openlit/observability/telemetry/ai-analysis) panel scoped to the whole trace hierarchy.+- **Timeline**: A waterfall view of every span plotted against elapsed time, useful for spotting sequential bottlenecks and overlapping work.+- **Graph**: A DAG (directed acyclic graph) view of the trace's call flow - nodes for spans, edges for parent/child call relationships.++### Right panel: attributes, per-span analysis, and evaluations++The right panel shows raw data for the span currently selected in the left panel:++- **Span Attributes**: every OTel attribute recorded on the selected span.+- **Resource Attributes**: service- and environment-level resource attributes.+- **AI Analysis**: a second, span-scoped AI Analysis tab - lets you analyze just the currently selected span instead of the whole trace.+- **Evaluations**: shown only when evaluation scores exist for the trace, with per-check results.++Header stats above both panels show Status, Duration, total Tokens, Cost (with a recalculate button), and Model, plus meta pills for Trace ID, Span ID, Service, and Environment. Use the **Prev / Next** controls to step through the traces matching your current filters without going back to the list.++## Exceptions++The **Exceptions** tab shows all traces that contain errors or exceptions from your AI applications. It uses the same trace list shell as Traces, but the table columns are Time, Span Name, Duration, Service Name, Deployment Type, and Exception Type, and you can filter by exception type in addition to time range.++<Frame>+ <img src="/images/errors-page.png" alt="Exceptions page listing traces with errors" />+</Frame>++All error traces are automatically captured and displayed here, including:++- **LLM API errors**: Authentication, rate limits, model issues+- **Framework errors**: LangChain, LlamaIndex execution failures+- **Vector database errors**: Connection and query issues+- **Application errors**: Custom exceptions and validation errors++Clicking an exception opens the same Tree / Chat / AI Analysis / Timeline / Graph trace detail view described above, so you get the full execution context - including the stack trace - around the error.++<Frame>+ <img src="/images/errors-trace.png" alt="Trace detail view with exception information and stack trace" />+</Frame>++---++<CardGroup cols={2}>+ <Card title="AI Analysis" href="/latest/openlit/observability/telemetry/ai-analysis" icon='sparkles'>+ Turn a trace or span into a structured review of performance, cost, and reliability+ </Card>+ <Card title="Metrics" href="/latest/openlit/observability/telemetry/metrics" icon='chart-line'>+ Explore the metrics captured alongside your traces+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/observability/tracing.mdx b/docs/latest/openlit/observability/tracing.mdxdeleted file mode 100644index 3ae74ff1..00000000--- a/docs/latest/openlit/observability/tracing.mdx+++ /dev/null@@ -1,104 +0,0 @@-----title: "Tracing"-description: "View and analyze distributed traces in OpenLIT"------<video- autoPlay- muted- loop- controls- className="w-full aspect-video rounded-xl"- src="/images/trace-details.mp4"->-</video>--The OpenLIT platform provides comprehensive trace visualization capabilities. You can view traces in two ways:--1. **Traces Page**: Navigate to the **Traces** section at `127.0.0.1:3000/traces` to view all distributed traces from your AI applications with detailed span analysis and execution flow.--2. **Dashboard Widgets**: Create custom trace widgets in your dashboards to monitor specific trace metrics, latency trends, and performance insights alongside other observability data.--## AI analysis for traces and spans--Trace details include an **AI Analysis** tab for both the full trace hierarchy and individual spans. Use it to turn raw OpenTelemetry data into a structured review of performance, reliability, cost, token usage, and execution flow.--The analysis works at two scopes:--- **Trace hierarchy analysis**: Reviews the root span and child spans together so you can understand the full request, agent path, tool calls, model calls, retries, errors, and cost drivers.-- **Individual span analysis**: Focuses on one selected span when you need to inspect a specific model call, tool execution, retrieval step, error, or latency hotspot.--Each analysis run evaluates:--- **Performance**: latency, slow spans, blocking operations, duration hotspots, and inefficient paths.-- **Reliability**: errors, failed spans, exception signals, retries, and unstable dependencies.-- **Cost**: model usage, estimated spend, high-cost spans, and cost optimization opportunities.-- **Token efficiency**: input tokens, output tokens, cache usage, repeated context, and verbose responses.-- **Prompt and model behavior**: prompt structure, model choice, output quality, and response efficiency.-- **Telemetry quality**: missing attributes, incomplete trace context, inconsistent service metadata, and weak observability signals.-- **Actionability**: a grader pass reviews each section so recommendations are clearer and easier to apply.--### Run analysis from trace details--1. Open a trace from the Traces page or from a trace widget.-2. Select **AI Analysis** in the trace detail view.-3. Choose whether to analyze the full trace hierarchy or the currently selected span.-4. Review the streamed progress. When the run completes, the analysis collapses into a reusable result.--Existing runs are shown in the analysis panel, so you can revisit prior findings without rerunning the model. Use rerun only when the trace data or investigation question has changed.--### Run analysis from Otter--You can also ask Otter to analyze traces or spans by natural language:--```-"Analyze this trace hierarchy"-"Review span <span_id> for token waste"-"Find cost optimization opportunities for traces where session.id is dc7b1ba1-..."-"Analyze the slowest traces from the last 24 hours"-```--When Otter references a trace or span in its answer, OpenLIT renders clickable trace/span pills so you can jump back into the trace detail experience.--## Group traces--Use **Group By** on the Traces page to roll up large trace lists into meaningful groups before drilling into individual spans. Grouping works with the selected time range and any active filters, so you can narrow the dataset first and then compare trace segments.--You can group traces by:--- **Model**: Compare requests by `gen_ai.request.model`.-- **Provider**: Compare requests by `gen_ai.system`.-- **Span Name**: Group repeated operations or framework steps.-- **Application**: Compare services using the `service.name` resource attribute.-- **Custom attribute**: Group by any span attribute, resource attribute, or top-level trace field available in your trace data.--Grouped rows show the number of spans in each group, total cost, token usage, and average duration. Click a group row to drill into the matching traces. The breadcrumb above the table shows the current grouping path and lets you return to all groups or remove grouping.--<Note>- Grouping is best for finding high-volume models, expensive providers, slow span types, or application-level hotspots before opening an individual trace.-</Note>--## Filter and group together--Grouping can be combined with the existing trace filters. For example, you can filter to a single environment, apply a maximum cost threshold, and then group by model to find which models dominate that filtered slice. Custom attribute filters and custom group-by attributes can be used together when you need to inspect application-specific metadata.-------<CardGroup cols={3}>- <Card title="Quickstart: LLM Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>- Production-ready AI monitoring setup in 2 simple steps with zero code changes- </Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Integrations" href="/latest/sdk/integrations/overview" icon='circle-nodes'>- 60+ AI integrations with automatic instrumentation and performance tracking- </Card>-</CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/openlit/organisation.mdx b/docs/latest/openlit/organisation.mdxindex bb7a923f..dafda6ee 100644--- a/docs/latest/openlit/organisation.mdx+++ b/docs/latest/openlit/organisation.mdx@@ -106,10 +106,10 @@ Every organisation has a default project. Existing organisations are mapped to a---<CardGroup cols={3}>- <Card title="Manage LLM secrets" href="/latest/openlit/developer-resources/vault" icon='vault'>+ <Card title="Manage LLM secrets" href="/latest/openlit/developer-resources/vault/overview" icon='book'>Centrally store LLM API keys that applications can retrieve remotely without restarts</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='shapes'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card><Card title="Multiple databases" href="/latest/openlit/developer-resources/multiple-db" icon='database'>diff --git a/docs/latest/openlit/organisation/database-config.mdx b/docs/latest/openlit/organisation/database-config.mdxindex 3ffd27b8..15db27ba 100644--- a/docs/latest/openlit/organisation/database-config.mdx+++ b/docs/latest/openlit/organisation/database-config.mdx@@ -4,36 +4,56 @@ description: 'Add and manage ClickHouse database configurations for an OpenLIT picon: 'database'----Database configurations connect OpenLIT to ClickHouse. They are scoped to the active project, so each project can use its own database connection.+A database configuration is the ClickHouse connection - host, port, database, and credentials - that a project reads and writes data through. Every project needs at least one before it can show traces, dashboards, or any other data.## Add a database configuration+Configs live on the project page under the **Database configurations** tab (`/organisation/project/[projectId]?tab=database`). `/settings/database-config` redirects to Organisation.+<Steps><Step title="Open Organisation">- Select **Organisation** from the sidebar.+ Select **Organisation** or **Projects** from the sidebar under Settings.</Step><Step title="Open a project">- Choose the project that should own the database configuration.- </Step>- <Step title="Add config">- Click **Add database config** and enter the ClickHouse connection details.+ Choose the project that should own the database configuration, then open the **Database configurations** tab.</Step>- <Step title="Test and save">- Test the connection, then save the configuration.+ <Step title="Add New Config">+ Click **Add New Config** and enter **Config Name**, **Environment**, **Username**, **Password**, **Host**, **Port**, **Database**, and optional **Query params**. Click **Save** (or **Update** when editing).</Step>- <Step title="Select config">- Select the database configuration from the project page or the top navigation selector.+ <Step title="Mark it active">+ Check the list row to **Mark … as the active database config**, or choose it from the header **Databases** switcher. Connectivity is checked after the config becomes active - there is no separate test-connection button on the form.</Step></Steps>## Selection behavior-The active database configuration is scoped to the current user and project. Switching projects updates the available database configurations. If a project does not have a database configuration, OpenLIT guides you through onboarding before showing data pages.+The active database configuration is scoped to the current user and project - it's not a global setting, so two people can be looking at different database configurations inside the same project at the same time. Switching projects updates the list of available database configurations to only those that belong to the newly selected project; a configuration created under one project is never selectable from another. If a project has no database configuration yet, OpenLIT guides you through onboarding before showing any data pages.## Multiple databases-A project can have multiple database configurations. Use multiple configurations when you need to switch between environments or ClickHouse clusters without changing the organisation.+A project can have multiple database configurations - for example, one per ClickHouse cluster or one per data retention tier. Use multiple configurations when you need to switch between them without changing projects or organisations.<Card title="Multiple database guide" href="/latest/openlit/developer-resources/multiple-db" icon="database">Learn more about using multiple ClickHouse database configurations in OpenLIT.</Card>++## Sharing a database configuration++A database configuration can be shared directly with specific teammates by email, independent of broader project membership - each share grants its own **edit**, **delete**, and **re-share** permissions, so you can give someone read-only-equivalent access to a connection without letting them modify or share it further.++## Frequently asked questions++<AccordionGroup>+ <Accordion title="Can I use the same database configuration across two projects?">+ No. A database configuration belongs to the project it was created in and can't be selected from a different project - create a new configuration (pointing at the same or a different ClickHouse instance) in each project that needs one.+ </Accordion>+ <Accordion title="Do I need a new ClickHouse instance for every database configuration?">+ No. Multiple configurations can point at the same ClickHouse cluster using different database names, or at entirely different clusters - whichever fits your infrastructure.+ </Accordion>+ <Accordion title="What happens if I open a project with no database configuration?">+ OpenLIT detects the missing configuration and walks you through adding one before showing any data pages for that project.+ </Accordion>+ <Accordion title="Is my ClickHouse password visible after I save a configuration?">+ No. The password is masked in the API and UI after saving - only entering a new value overwrites it.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/organisation/overview.mdx b/docs/latest/openlit/organisation/overview.mdxindex fc7e76d7..a66b2cd3 100644--- a/docs/latest/openlit/organisation/overview.mdx+++ b/docs/latest/openlit/organisation/overview.mdx@@ -4,24 +4,46 @@ description: 'Understand how organisations, projects, and database configurationicon: 'building'----Organisations in OpenLIT are shared workspaces for teams and environments. An organisation contains projects, and each project contains one or more ClickHouse database configurations.+An organisation in OpenLIT is a shared workspace for a team. It owns projects, and each project owns one or more ClickHouse database configurations - this three-level hierarchy is how OpenLIT separates teams, environments, and data without running separate installations.-```text-Organisation- Project- Database configuration+```mermaid+flowchart TB;+ Org[Organisation] -->|owns| Proj1[Project: production];+ Org -->|owns| Proj2[Project: staging];+ Proj1 -->|connects to| DB1[ClickHouse database config];+ Proj2 -->|connects to| DB2[ClickHouse database config];```-Use organisations for workspace and membership boundaries. Use projects when you need separate database boundaries inside the same organisation, such as production, staging, customer-specific environments, or different teams.+Use organisations as the membership boundary - who can log in and see this workspace at all. Use projects inside an organisation when you need separate database boundaries for the same team, such as production, staging, or per-customer environments.++## Creating an organisation++New users are prompted to either create an organisation (just a name) or accept a pending invitation during onboarding. Skipping this step creates a personal organisation automatically, so every user always lands in exactly one organisation - onboarding never leaves you without a workspace.++Every organisation gets a default project the moment it's created, and existing installations are migrated into a default project automatically, so upgrading never breaks an existing database configuration.++## Membership and roles++Every organisation member has one of three roles:++| Role | Description |+|---|---|+| **Owner** | The user who created the organisation. Owner status can't be reassigned or changed. |+| **Admin** | Can invite members, manage projects, and change other members' roles (except the owner's). |+| **Member** | Can use the organisation's projects and data; can't manage membership. |++Invite teammates by email from the Organisation page - each invite is scoped to that organisation, and the invited user accepts or declines it from their own onboarding screen. Removing a member revokes their access to every project in that organisation immediately.## What belongs to an organisation- Organisation identity, such as the workspace name.-- Members and pending invitations.-- Projects owned by the organisation.+- Members, their roles, and pending invitations.+- Projects owned by the organisation, including the default project.- Billing and licensing state where applicable.-Every organisation has a default project. Existing installations are mapped to a default project so existing database configurations continue to work after upgrading.+## Switching and deleting organisations++If you belong to multiple organisations, use the organisation selector in the top navigation to change the active workspace - project and database configuration selectors refresh to match the newly selected organisation. Deleting an organisation removes every project and database configuration that belongs to it, so it's only available to the owner.## Common tasks@@ -34,7 +56,19 @@ Every organisation has a default project. Existing installations are mapped to a</Card></CardGroup>-## Workspace switching--If you belong to multiple organisations, use the organisation selector in the top navigation to change the active workspace. Project and database configuration selectors update based on the selected organisation.+## Frequently asked questions+<AccordionGroup>+ <Accordion title="What's the difference between an organisation and a project?">+ An organisation is the membership boundary - it controls who can access a workspace at all. A project is a database boundary inside an organisation - it controls which ClickHouse connection a given environment or team uses. One organisation can have many projects.+ </Accordion>+ <Accordion title="What happens if I skip creating an organisation during onboarding?">+ OpenLIT creates a personal organisation for you automatically, with a default project already set up, so you're never left without a workspace.+ </Accordion>+ <Accordion title="Can I change who owns an organisation?">+ No. The owner is permanently the user who created the organisation. Admins can manage members and projects, but ownership itself can't be reassigned.+ </Accordion>+ <Accordion title="What happens to my data if I delete an organisation?">+ Deleting an organisation deletes every project and database configuration that belongs to it. Only the owner can delete an organisation.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/organisation/projects.mdx b/docs/latest/openlit/organisation/projects.mdxindex edc59fc5..364302c8 100644--- a/docs/latest/openlit/organisation/projects.mdx+++ b/docs/latest/openlit/organisation/projects.mdx@@ -4,16 +4,18 @@ description: 'Create and manage project boundaries inside an OpenLIT organisatioicon: 'folder'----Projects group database configurations and resources inside an organisation. They are useful when one organisation needs multiple isolated environments without creating separate workspaces.+A project is a database boundary inside an organisation. Every project connects to its own ClickHouse database configuration, so one organisation can run several isolated environments - production, staging, per-customer, or per-team - without needing separate organisations or separate OpenLIT installations.## When to create a projectCreate a project for:-- Production, staging, and development environments.-- Separate teams that share the same organisation.-- Customer-specific deployments.-- Workloads that should use different ClickHouse database configurations.+| Scenario | Why a project helps |+|---|---|+| Production and staging | Keep live traces and test traces in separate ClickHouse databases. |+| Multiple teams, one organisation | Each team gets its own project without splitting membership across organisations. |+| Customer-specific deployments | Isolate a customer's data in its own database configuration while keeping shared org-level billing and membership. |+| Different ClickHouse clusters | Route one workload to a different ClickHouse host or cluster than another, under the same organisation. |## Create a project@@ -32,11 +34,33 @@ Create a project for:</Step></Steps>+Every organisation always has exactly one **default project**, created automatically the moment the organisation itself is created. If an organisation has no other database configuration set up yet, OpenLIT falls back to the default project so there's always a working project to land in.+## Project detailsProject details show the project name, status, and database configurations attached to that project. Open a project from the Organisation page to review or update its configuration.+## Switching projects++The project selector in the top navigation scopes everything you see - traces, dashboards, prompts, evaluations - to the currently active project's database configuration. Switching projects re-fetches the active database configuration for the new project; if that project has none configured yet, OpenLIT walks you through adding one before showing data pages.+## Member access-Project access controls which organisation members can use a project. Existing organisation members are assigned to existing projects during migration so current installations continue to work. New members can be invited to the organisation first and then granted access to specific projects as needed.+Project access controls which organisation members can use a project. Existing organisation members are assigned to existing projects during migration so current installations continue to work. New members can be invited to the organisation first and then granted access to specific projects as needed. A member without access to a project can't select it or see its database configurations, even if they're a member of the parent organisation.++## Frequently asked questions+<AccordionGroup>+ <Accordion title="Do I need multiple projects if I only have one environment?">+ No. Every organisation ships with a default project, and most single-environment setups never need to create another one.+ </Accordion>+ <Accordion title="Can a project use more than one database configuration?">+ Yes. See [Database Config](/latest/openlit/organisation/database-config) for adding and switching between multiple ClickHouse configurations inside the same project.+ </Accordion>+ <Accordion title="What happens if I switch to a project with no database configured?">+ OpenLIT detects the missing configuration and guides you through adding one before showing any data pages for that project.+ </Accordion>+ <Accordion title="Can every organisation member see every project?">+ No. Project access is assigned per member. A member only sees and can select the projects they've been granted access to.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/pricing/cost-recalculation.mdx b/docs/latest/openlit/pricing/cost-recalculation.mdxdeleted file mode 100644index aca1b110..00000000--- a/docs/latest/openlit/pricing/cost-recalculation.mdx+++ /dev/null@@ -1,121 +0,0 @@-----title: 'Cost Recalculation'-description: 'Recalculate the cost of LLM traces using current model pricing — manually or on a schedule'-sidebarTitle: 'Cost Recalculation'------The Pricing feature recalculates the `gen_ai.usage.cost` attribute on existing LLM traces using the per-model pricing stored in [Manage Models](/latest/openlit/pricing/manage-models). This is useful when:--- Your SDK didn't send cost data (or sent `0`).-- Model pricing has changed and you want historical traces to reflect the new prices.-- You added a custom model after traces were already ingested.--## How it works--For each trace, OpenLIT:--1. Reads `gen_ai.system` (provider) and `gen_ai.request.model` from the span attributes.-2. Looks up the matching model in the `openlit_provider_models` ClickHouse table.-3. Computes the cost:- ```- cost = (input_tokens / 1,000,000) × input_price_per_m_token- + (output_tokens / 1,000,000) × output_price_per_m_token- ```-4. Writes the result back to the trace:- ```sql- ALTER TABLE otel_traces- UPDATE SpanAttributes = mapUpdate(SpanAttributes, map('gen_ai.usage.cost', '<cost>'))- WHERE SpanId = '<span_id>'- ```--<Info>-Traces with missing provider, model, or zero tokens are **skipped** — they are not counted as errors.-If the model is not found in Manage Models, the trace is also skipped. Add the model first, then recalculate.-</Info>--## Manual recalculation--Recalculate the cost for a single trace directly from the trace detail panel.--<Steps>- <Step title="Open a trace">- Navigate to **Requests** and click on any LLM trace to open the detail panel.- </Step>- <Step title="Click the recalculate icon">- In the **Cost** metric card at the top of the panel, click the refresh icon (↻).- - If the cost is missing or zero, a **pinging dot** draws attention to the icon.- - If the cost already exists, clicking recalculates it using the latest pricing.- </Step>- <Step title="See the result">- A toast notification shows the updated cost. The panel refreshes automatically to display the new value.- </Step>-</Steps>--**API**: `POST /api/pricing/{spanId}` — returns `{ success, data: { spanId, cost } }` or `{ success: false, err: "..." }`.--## Auto recalculation (cron-based)--Automatically recompute pricing on a schedule for all newly ingested LLM traces.--<Steps>- <Step title="Open the Pricing page">- Navigate to **Pricing** in the sidebar under the Configuration section.- </Step>- <Step title="Enable Auto Pricing">- Toggle the **Enable Auto Pricing** switch.- </Step>- <Step title="Set the cron schedule">- Enter a standard cron expression. For example:- - `*/15 * * * *` — every 15 minutes- - `0 * * * *` — every hour- - `0 0 * * *` — once a day at midnight- </Step>- <Step title="Save">- Click **Save** (or **Update** if editing). OpenLIT installs a cron job that runs the pricing recalculation on the schedule you defined.- </Step>-</Steps>--### How auto pricing selects traces--On each cron tick:--| Step | What happens |-|------|-------------|-| **1. Select LLM spans** | Queries `otel_traces` for spans whose `gen_ai.operation.name` matches supported LLM operations (e.g. `chat`). |-| **2. Incremental window** | Only spans with `Timestamp >= last successful cron run` are processed. The first run processes all historical LLM spans. |-| **3. Look up model** | For each span, reads `gen_ai.system` (provider) and `gen_ai.request.model`, then finds the matching row in `openlit_provider_models`. |-| **4. Compute cost** | `(input_tokens / 1M) × input_price + (output_tokens / 1M) × output_price`. |-| **5. Write back** | Updates `SpanAttributes['gen_ai.usage.cost']` on the trace via `ALTER TABLE otel_traces UPDATE`. |-| **6. Log run** | Records `totalSpans`, `totalUpdated`, `totalFailed`, `totalSkipped` in the cron log with a status of `SUCCESS`, `PARTIAL_SUCCESS`, or `FAILURE`. |--<Note>-Spans with missing provider/model/tokens or an unknown model are **skipped** (not counted as errors). Add the model to Manage Models and the next cron run will pick it up.-</Note>--### Cron restoration on restart--When the OpenLIT server restarts (e.g. after a container redeployment), all active auto-pricing cron jobs are automatically restored from the database. No manual re-setup is needed.--## API reference--| Endpoint | Method | Description |-|----------|--------|-------------|-| `/api/pricing/config` | `GET` | Get the current pricing config (auto flag, cron schedule) |-| `/api/pricing/config` | `POST` | Create or update pricing config |-| `/api/pricing/{spanId}` | `POST` | Manually recalculate cost for a single trace |-| `/api/pricing/auto` | `POST` | Trigger auto pricing run (called by cron; allowlisted in middleware) |-| `/api/openground/models/export` | `GET` | Export all model pricing as downloadable JSON (authenticated) |-| `/api/pricing/export/{dbConfigId}` | `GET` | Public pricing URL for SDK `pricing_json` (no auth) |-| `/api/openground/models/import` | `POST` | Bulk-import models from JSON, skipping duplicates |--## Configuration--The pricing config is stored in the SQLite `PricingConfigs` table (managed by Prisma):--| Field | Type | Description |-|-------|------|-------------|-| `id` | String | Config ID (auto-generated) |-| `databaseConfigId` | String | Links to the ClickHouse database config |-| `auto` | Boolean | Whether auto pricing is enabled |-| `recurringTime` | String | Cron schedule (e.g. `*/15 * * * *`) |-| `meta` | String (JSON) | Internal metadata including `cronJobId` |diff --git a/docs/latest/openlit/pricing/manage-models.mdx b/docs/latest/openlit/pricing/manage-models.mdxdeleted file mode 100644index 9aa8791b..00000000--- a/docs/latest/openlit/pricing/manage-models.mdx+++ /dev/null@@ -1,251 +0,0 @@-----title: 'Manage Models'-description: 'View, edit, and add LLM models with per-model pricing across all providers'-sidebarTitle: 'Manage Models'------Manage Models is an independent feature for managing LLM provider metadata and per-model pricing. It is used across OpenLIT — including Chat, Evaluations, OpenGround, and Pricing — to look up model details and compute costs.--## Overview--Every ClickHouse database connected to OpenLIT automatically gets a pre-populated set of **14 providers and 45+ models** with default pricing. All models are fully editable — you can update pricing, add new models, or delete ones you don't use.--### Supported providers (out of the box)--| Provider | Example Models |-|----------|---------------|-| OpenAI | GPT-4o, GPT-4o Mini, GPT-4 Turbo, GPT-3.5 Turbo |-| Anthropic | Claude 3.5 Sonnet, Claude 3 Haiku, Claude 3 Sonnet |-| Google AI | Gemini 1.5 Pro, Gemini 1.5 Flash, Gemini 1.0 Pro |-| Mistral AI | Mistral Large, Mistral Medium, Mistral Small |-| Groq | Llama 3.3 70B, Llama 3.1 8B, Mixtral 8x7B |-| Perplexity | Sonar Pro, Sonar, Sonar Reasoning |-| Azure OpenAI | GPT-4o, GPT-4o Mini, GPT-4 Turbo, GPT-3.5 Turbo |-| Cohere | Command R+, Command R, Command, Command Light |-| Together AI | Llama 3.1 70B Turbo, Llama 3.1 8B Turbo, Mixtral 8x7B |-| Fireworks AI | Llama 3.1 70B, Llama 3.1 8B, Mixtral 8x7B |-| DeepSeek | DeepSeek Chat, DeepSeek Coder |-| xAI | Grok Beta, Grok 2, Grok 3, Grok 3 Mini |-| Hugging Face | Llama 3 70B, Mistral 7B, Gemma 2 9B |-| Replicate | Llama 3 70B, Llama 3 8B, Mixtral 8x7B |--## Managing providers--Providers (e.g. OpenAI, Anthropic) are the top-level grouping for models. OpenLIT ships with 14 built-in providers, but you can add your own or edit any existing one.--<Steps>- <Step title="Add a new provider">- Click the **Add Provider** button (`+` icon) in the header toolbar. Fill in:- - **Provider ID** — a lowercase slug (e.g. `my-llm`). As you type, special characters are automatically replaced with hyphens. This must match the value your SDK sends in `gen_ai.system`.- - **Display Name** — the label shown in the UI (e.g. "My LLM").- - **Description** — optional, shown in provider lists.- - **Requires API Key** — toggle on if the provider needs a Vault secret.- </Step>- <Step title="Edit an existing provider">- Hover over a provider name in the sidebar and click the pencil icon. You can change the display name, description, and vault requirement. The Provider ID cannot be changed after creation.- </Step>- <Step title="Delete a provider">- Use the `DELETE /api/openground/providers?provider=<id>` API endpoint. This removes the provider **and all its models**.- </Step>-</Steps>--## Managing models--<Steps>- <Step title="Open Manage Models">- Navigate to **Manage Models** in the sidebar under the Configuration section.- </Step>- <Step title="Browse providers and models">- The left sidebar lists all providers. Click a provider to expand its models. Each model shows its context window and input price.- - **Default** badge — a model that was pre-populated by OpenLIT.- - **Custom** badge — a model you added manually.- </Step>- <Step title="Edit a model">- Click any model to open the editor panel on the right. You can change:- - **Display Name** — the human-readable name shown in dropdowns.- - **Model Type** — chat, embeddings, images, or audio.- - **Context Window** — the maximum token context length.- - **Input / Output Price per 1M Tokens** — used by the Pricing feature to compute costs.- - **Capabilities** — comma-separated list (e.g. `function-calling, vision, streaming`).- </Step>- <Step title="Add a new model">- Click the **+** button next to any provider header to add a new model. Fill in the Model ID (the identifier your SDK sends, e.g. `gpt-4o-2024-08-06`), display name, and pricing.- </Step>- <Step title="Delete a model">- Open a model in the editor and click the **Delete** button. Deleted models can be re-added at any time.- </Step>-</Steps>--## Export pricing JSON--`GET /api/openground/models/export` returns all models in an SDK-compatible `pricing.json` format, grouped by model type.--This is useful for feeding custom pricing into the OpenLIT Python or TypeScript SDKs when the default pricing data doesn't include your models.--```json Example response-{- "chat": {- "gpt-4o": {- "promptPrice": 0.0025,- "completionPrice": 0.01- },- "claude-3-5-sonnet-20240620": {- "promptPrice": 0.003,- "completionPrice": 0.015- }- },- "embeddings": {- "text-embedding-3-small": 0.00002- }-}-```--<Tip>-Click the **Export Pricing** button in the top-right corner of the Manage Models page to download this file directly.-</Tip>--## Public SDK Pricing URL--Each database config has a **public, unauthenticated** pricing URL that you can pass directly to the OpenLIT SDK. This lets your SDK load pricing from the same models you manage in the UI — no manual JSON files needed.--The URL is shown in a bar below the header on the Manage Models page. Click **Copy** to copy it to your clipboard.--<Tip>-Click the **SDK Usage** button in the header to open a dialog with ready-to-paste code snippets for Python, TypeScript, and Go. Each snippet pre-fills the public URL and includes a copy button.-</Tip>--<CodeGroup>-```python Python-import openlit--openlit.init(- otlp_endpoint="http://127.0.0.1:4318",- pricing_json="http://your-openlit-host/api/pricing/export/<your-db-config-id>",-)-```--```typescript TypeScript-import openlit from "openlit";--openlit.init({- otlpEndpoint: "http://127.0.0.1:4318",- pricingJson: "http://your-openlit-host/api/pricing/export/<your-db-config-id>",-});-```--```go Go-package main--import openlit "github.com/openlit/openlit/sdk/go"--func main() {- openlit.Init(openlit.Config{- OtlpEndpoint: "http://127.0.0.1:4318",- PricingJson: "http://your-openlit-host/api/pricing/export/<your-db-config-id>",- })-}-```-</CodeGroup>--**Endpoint:** `GET /api/pricing/export/{dbConfigId}`--- No authentication required — safe to embed in application configs.-- Returns the same SDK-compatible pricing JSON format as the export button.-- Responses are cached for 5 minutes (`Cache-Control: public, max-age=300`).--<Warning>-The URL contains your database config ID. While the endpoint only returns pricing data (no secrets or traces), share it only with systems that should access your model pricing.-</Warning>--<Note>-The SDK fetches this URL on startup. If you update a model's price in Manage Models, restart your app (or wait for the SDK's refresh interval) to pick up the new pricing.-</Note>--## Import providers and models--You can bulk-import providers and models from a JSON file. This is useful for syncing across environments, sharing setups with teammates, or adding a custom provider with all its models in one step.--<Steps>- <Step title="Open the import dialog">- Click the **Import Pricing JSON** button in the header of the Manage Models page.- </Step>- <Step title="Paste JSON">- Paste either the **structured format** or the **SDK pricing_json format**.-- **Structured format with providers + models** (recommended):- ```json- {- "providers": [- {- "providerId": "my-provider",- "displayName": "My Provider",- "description": "Custom LLM provider"- }- ],- "models": [- {- "provider": "my-provider",- "model_id": "my-model-v1",- "displayName": "My Model v1",- "inputPricePerMToken": 2.5,- "outputPricePerMToken": 10- }- ]- }- ```-- **Models only** (provider must already exist):- ```json- {- "models": [- {- "provider": "openai",- "model_id": "gpt-4o-2024-08-06",- "displayName": "GPT-4o (Aug 2024)",- "inputPricePerMToken": 2.5,- "outputPricePerMToken": 10- }- ]- }- ```-- **SDK pricing_json format** (provider defaults to `"unknown"` — edit after import):- ```json- {- "chat": {- "gpt-4o": { "promptPrice": 0.0025, "completionPrice": 0.01 }- }- }- ```- </Step>- <Step title="Import">- Click **Import Pricing JSON**. Providers and models that already exist are automatically **skipped** — no duplicates are created. The response shows counts for both:- - `imported` / `skipped` — models- - `providersImported` / `providersSkipped` — providers- </Step>-</Steps>--**Endpoint:** `POST /api/openground/models/import` (authenticated)--## API reference--| Endpoint | Method | Description |-|----------|--------|-------------|-| `/api/openground/providers` | `GET` | List all providers with models |-| `/api/openground/providers` | `POST` | Create a new provider |-| `/api/openground/providers` | `PUT` | Update a provider's metadata |-| `/api/openground/providers?provider=<id>` | `DELETE` | Delete a provider and all its models |-| `/api/openground/models` | `GET` | List all models |-| `/api/openground/models` | `POST` | Create or update a model |-| `/api/openground/models` | `DELETE` | Delete a model |-| `/api/openground/models/import` | `POST` | Bulk import providers + models from JSON |-| `/api/openground/models/export` | `GET` | Export pricing as downloadable JSON |-| `/api/pricing/export/{dbConfigId}` | `GET` | Public pricing URL for SDK (no auth) |--## Architecture--- **ClickHouse tables**: `openlit_provider_metadata` (provider definitions), `openlit_providers` (API key configs), and `openlit_provider_models` (all models with pricing).-- **No `database_config_id` column** — each ClickHouse instance is its own scope. When you connect a new ClickHouse database, the migration automatically creates the tables and seeds all default providers and models.-- **Fully dynamic** — both providers and models are stored in ClickHouse and editable via the UI or API. No hardcoded provider list.-- **Platform library**: `src/lib/platform/providers/` — `provider-registry.ts`, `models-service.ts`, `config.ts`, `default-models.ts`.-- **Seeding**: `default-models.ts` contains the initial 14 providers and 45+ models used only by the migration for first-run seeding.diff --git a/docs/latest/openlit/prompts-experiments/context.mdx b/docs/latest/openlit/prompts-experiments/context.mdxdeleted file mode 100644index 66ab3efb..00000000--- a/docs/latest/openlit/prompts-experiments/context.mdx+++ /dev/null@@ -1,189 +0,0 @@-----title: 'Context'-description: 'Store and manage reusable contextual content that can be linked to rules and retrieved at runtime'------Context lets you store rich, reusable content — such as system prompts, knowledge snippets, policy documents, or any markdown text — and associate it with rules in the Rule Engine. When a rule matches at evaluation time, the linked context data is returned alongside the result, ready to be injected into your AI pipeline.--## Key features--- **Markdown content**: Write and preview context content in a full Write/Preview markdown editor.-- **Tags**: Categorise contexts with free-form tags for easy filtering and discovery.-- **Meta properties**: Attach arbitrary key-value metadata to contexts (e.g. `model: gpt-4`, `region: us`).-- **Status control**: Mark contexts as **Active** or **Inactive** to enable or disable them without deletion.-- **Rule linking**: Associate contexts with one or more rules. When a rule matches, linked context content is returned in the evaluate API response.--## Get started--<Steps>- <Step title="List contexts">- <Frame>- <img src="/images/context-list.png" />- </Frame>-- Get an overview of all contexts created.- 1. Navigate to **Context** in the OpenLIT sidebar.- 2. Browse the list of existing contexts with their status and description.- </Step>- <Step title="Create a context">- 1. Click **Create Context** in the top-right corner.- 2. Enter a **Name** (required) and an optional **Description**.- 3. Write your content in the **Content** editor. Markdown is fully supported — switch to **Preview** to see the rendered output.- 4. On the right panel:- - Set the **Status** to *Active* or *Inactive*.- - Add **Tags** by typing and pressing Enter.- - Add **Meta Properties** as key-value pairs for additional metadata.- 5. Click **Save Context**. You will be redirected to the context detail page.- </Step>- <Step title="Edit a context">- <Frame>- <img src="/images/context-detail.png" />- </Frame>-- 1. Click on any context in the list to open its detail page.- 2. Click **Edit** to switch to edit mode.- 3. Modify any fields — name, description, content, status, tags, or meta properties.- 4. Click **Save** to apply changes.- </Step>- <Step title="Link a context to a rule">- Contexts become actionable when linked to rules.- 1. Open the context detail page.- 2. In the **Linked Rules** panel on the right, click **New Rule** to create and link a new rule, or use **Link existing rule** to associate an existing rule.- 3. Linked rules are listed with their name and status. Click a rule name to navigate to its detail page.-- Alternatively, from the Context list page, click the **rule icon** (⧉) on any row to quickly create a rule linked to that context.- </Step>- <Step title="Retrieve context data at runtime">- Use the Rule Engine evaluate API to fetch matching context data for your AI pipeline.-- <Steps>- <Step title="Create an API Key">- - Navigate to **Settings → API Keys** in OpenLIT.- - Click **Create API Key**, enter a name, and save the key securely.- </Step>- <Step title="Call the evaluate API">- Send a `POST` request to `/api/rule-engine/evaluate` with your input fields and `entity_type: "context"` using the SDK or curl.-- <Tabs>- <Tab title="Python">- ```python- import openlit-- result = openlit.evaluate_rule(- entity_type="context",- fields={"model": "gpt-4", "user_tier": "premium"},- include_entity_data=True,- )-- if result:- for entity in result.get("entities", []):- key = f"context:{entity['entity_id']}"- data = result.get("entity_data", {}).get(key, {})- print(f"Context: {data.get('name')}")- print(f"Content: {data.get('content')}")- ```- </Tab>- <Tab title="TypeScript">- ```typescript- import Openlit from 'openlit';-- const result = await Openlit.evaluateRule({- entityType: 'context',- fields: { model: 'gpt-4', user_tier: 'premium' },- includeEntityData: true,- });-- if (!('err' in result)) {- for (const entity of result.entities) {- const key = `context:${entity.entity_id}`;- const data = result.entity_data?.[key];- console.log('Context:', data?.name);- console.log('Content:', data?.content);- }- }- ```- </Tab>- <Tab title="Go">- ```go- result, err := openlit.EvaluateRule(ctx, openlit.EvaluateRuleOptions{- EntityType: openlit.RuleEntityContext,- Fields: map[string]interface{}{"model": "gpt-4", "user_tier": "premium"},- IncludeEntityData: true,- })- if err != nil {- log.Fatal(err)- }- for _, entity := range result.Entities {- key := fmt.Sprintf("context:%s", entity.EntityID)- fmt.Printf("Context data: %v\n", result.EntityData[key])- }- ```- </Tab>- <Tab title="curl">- ```bash- curl -X POST https://your-openlit-instance/api/rule-engine/evaluate \- -H "Authorization: Bearer YOUR_API_KEY" \- -H "Content-Type: application/json" \- -d '{- "entity_type": "context",- "fields": {- "model": "gpt-4",- "user_tier": "premium"- },- "include_entity_data": true- }'- ```- </Tab>- </Tabs>-- ```json Example Response- {- "matchingRuleIds": ["rule-uuid-1"],- "entities": [- {- "rule_id": "rule-uuid-1",- "entity_type": "context",- "entity_id": "context-uuid-1"- }- ],- "entity_data": {- "context:context-uuid-1": {- "id": "context-uuid-1",- "name": "Premium System Prompt",- "content": "You are a helpful AI assistant for premium users...",- "tags": "[\"premium\",\"system\"]",- "status": "ACTIVE"- }- }- }- ```-- Set `include_entity_data: true` to receive the full context record. When omitted, only matching rule IDs and entity references are returned.-- <Info>For detailed SDK parameters and error handling, see the [SDK Rule Engine docs](/latest/sdk/features/rule-engine).</Info>- </Step>- </Steps>- </Step>- <Step title="Delete a context">- 1. Open the context detail page.- 2. Click the **Delete** button.- 3. Confirm deletion. This action is permanent.- </Step>-</Steps>-------<CardGroup cols={2}>- <Card title="Rule Engine" href="/latest/openlit/prompts-experiments/rule-engine" icon='sliders'>- Create conditional rules to match inputs and retrieve linked contexts, prompts, and other resources- </Card>- <Card title="Prompt Hub" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking- </Card>- <Card title="SDK Rule Engine" href="/latest/sdk/features/rule-engine" icon='puzzle-piece'>- Use evaluate_rule() from Python, TypeScript, or Go with full parameter reference- </Card>- <Card title="API Reference: Evaluate" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon='code'>- Full reference for the Rule Engine evaluate endpoint with request and response schemas- </Card>-</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/context/link-rules.mdx b/docs/latest/openlit/prompts-experiments/context/link-rules.mdxnew file mode 100644index 00000000..05c80ea4--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/context/link-rules.mdx@@ -0,0 +1,41 @@+---+title: "Link Rules"+description: "Connect a context to Rule Engine rules from either the context's detail page or the rule's detail page"+---++A context only becomes retrievable at runtime once it's linked to at least one rule in the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview). Linking can be done from either side - from the context's detail page, or from the rule's detail page.++## From the context detail page++Open a context's detail page. The **Rules** panel on the right shows every rule currently linked to this context. Each linked rule appears as a clickable row with its name, description, and status badge - clicking a row opens that rule's own detail page. If no rules are linked yet, the panel shows "No rules linked yet."++You have two ways to add a link from here:++### Create a new rule and link it++Click **New Rule** to open a dialog that creates a brand-new rule already linked to this context. The dialog asks for:++- **Name**+- **Description**+- **Group Operator** - AND or OR+- **Status**++Once saved, the new rule appears in the Rules panel and is linked to this context.++### Link an existing rule++Toggle **Link existing rule** to open a dropdown of rules that aren't yet linked to this context (placeholder "Select a rule...", or "All rules already linked" if there are none left to add). Pick a rule, then click **Associate** to confirm the link.++### Unlinking++There is no unlink control on the context detail page - once a rule is linked from this side, you can't remove that link from the Context page. To remove the link between a context and a rule, open the rule's own detail page instead and remove the association from its **Associated Entities** panel, where each linked entity has a remove control. See the Rule Engine's [Linked Entities](/latest/openlit/prompts-experiments/rule-engine/linked-entities) doc for details.++## From the rule detail page++You can also start from the other direction. Open a rule's detail page, and in its **Associated Entities** panel, choose entity type **Context** and pick an existing context to link. This creates the same link you'd get from the context side, and is also where you'd go to remove a link. See [Linked Entities](/latest/openlit/prompts-experiments/rule-engine/linked-entities) for the full flow.++---++<Card title="Retrieve Context" href="/latest/openlit/prompts-experiments/context/retrieve-context" icon='cloud-arrow-down'>+ Call the evaluate API at runtime to fetch matching context content+</Card>diff --git a/docs/latest/openlit/prompts-experiments/context/manage-contexts.mdx b/docs/latest/openlit/prompts-experiments/context/manage-contexts.mdxnew file mode 100644index 00000000..8119fb21--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/context/manage-contexts.mdx@@ -0,0 +1,70 @@+---+title: "Manage Contexts"+description: "Create, edit, and delete contexts - name, description, markdown content, status, tags, and meta properties"+---++## List contexts++<Frame>+ <img src="/images/context-list.png" alt="List of contexts with their status and description" />+</Frame>++Navigate to **Contexts** in the OpenLIT sidebar to see every context you've created. The table shows:++- **Name**+- **Description** - or `-` if none was set+- **Status** - an *Active* or *Inactive* badge+- **Created By**+- **Created At**+- **Actions** - a delete (trash) icon++Clicking anywhere on a row opens that context's detail page. The delete icon is the only row action - there's no shortcut on the list page to create a rule from a context row.++## Create a context++1. From the list page, open the create page at `/context/new`.+2. Enter a **Name** (required, placeholder "My Context").+3. Optionally enter a **Description** (placeholder "Optional description").+4. Write your **Content** in the markdown editor:+ - The **Write** tab is a plain textarea with the hint "Write your context content here. Markdown is supported."+ - The **Preview** tab renders your content as Markdown, or shows "Nothing to preview yet." if the content is empty.+5. Set **Status** to *Active* or *Inactive* (defaults to Active).+6. Add **Tags** by typing and pressing Enter (placeholder "Add a tag, press Enter").+7. Add **Meta Properties** as key-value pairs using the "Add property" button.+8. Click **Save Context**.++If **Name** is empty, you'll see "Context name is required!". If **Content** is empty, you'll see "Context content is required!". While saving, a "Creating context..." toast appears, followed by "Created context successfully!" on success, and you're taken to the new context's detail page.++## View and edit a context++<Frame>+ <img src="/images/context-detail.png" alt="Context detail page in edit mode" />+</Frame>++The detail page (`/context/[id]`) shows the context's name, a Status badge, and "Created By {name} · {date}".++Outside edit mode:+- **Content** renders read-only as Markdown, or shows "No content yet. Click Edit to add content." if empty.+- **Tags** and **Meta Properties** show "None" if empty.++Click **Edit** to switch the whole card into edit mode. Name, Description, Content (with its Write/Preview tabs), Status, Tags, and Meta Properties all become editable in place. From here you can:++- Click **Cancel** to discard your changes and return to the read-only view.+- Click **Save** to apply your changes. A "Updated context successfully!" toast confirms the update.++There is no separate edit page or URL - editing always happens inline on the detail page.++## Delete a context++Deleting a context is only available from the list page - there is no delete control on the detail page.++1. On the **Contexts** list page, click the delete (trash) icon on the row for the context you want to remove.+2. Confirm the prompt: "Are you sure you want to delete this context? Deleting context might break applications using it. Please confirm before deleting it."++This action is permanent.++---++<Card title="Link Rules" href="/latest/openlit/prompts-experiments/context/link-rules" icon='link'>+ Connect a context to one or more Rule Engine rules+</Card>diff --git a/docs/latest/openlit/prompts-experiments/context/overview.mdx b/docs/latest/openlit/prompts-experiments/context/overview.mdxnew file mode 100644index 00000000..931b180a--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/context/overview.mdx@@ -0,0 +1,65 @@+---+title: "Overview"+description: "RAG context management for OpenLIT - store reusable content once and have the Rule Engine return the right piece automatically"+---++Context is OpenLIT's answer to RAG context management: a place to store reusable content for your RAG pipelines and AI applications, and have it returned automatically at runtime instead of hardcoding lookup logic in your application code.++A context is simply a named, versionable piece of markdown - a system prompt, a knowledge snippet, a policy document, or any other reusable text - that you write once in OpenLIT and link to one or more rules in the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview). When a linked rule matches at runtime (for example, `model` equals `gpt-4` and `user_tier` equals `premium`), the evaluate API returns that context's content alongside the match, ready to inject into your prompt or pipeline.++## What you get++<CardGroup cols={3}>+ <Card title="Manage Contexts" href="/latest/openlit/prompts-experiments/context/manage-contexts" icon='file-lines'>+ Create, edit, and delete contexts - name, description, markdown content, status, tags, and meta properties+ </Card>+ <Card title="Link Rules" href="/latest/openlit/prompts-experiments/context/link-rules" icon='link'>+ Connect a context to one or more Rule Engine rules, from either the context or the rule side+ </Card>+ <Card title="Retrieve Context" href="/latest/openlit/prompts-experiments/context/retrieve-context" icon='cloud-arrow-down'>+ Call the evaluate API at runtime to fetch matching context content for your application+ </Card>+</CardGroup>++## Why Context++Most prompt-management tools stop at versioning and deploying prompts. OpenLIT's Context goes a step further: paired with the Rule Engine, it lets you store reusable knowledge snippets or system-prompt fragments once, and have your application fetch the right one automatically based on runtime conditions - like the model in use or a user's plan tier - rather than writing that branching logic yourself. This combination of a reusable content store plus a conditional rule engine has no direct equivalent in comparable open-source LLM observability tools.++---++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is Context used for?">+ Context stores reusable markdown content - system prompts, knowledge snippets, policy text, or any other text your AI application needs - so you can write it once and reuse it across rules instead of duplicating it in code.+ </Accordion>+ <Accordion title="How is a context different from a prompt in Prompt Hub?">+ Prompt Hub is for versioning and deploying structured prompts with variables. Context is a simpler, freeform markdown store meant to be paired with the Rule Engine so the right content is returned automatically based on runtime conditions, rather than fetched by name.+ </Accordion>+ <Accordion title="Do I need a rule to use a context?">+ A context can be created and edited on its own, but it only becomes retrievable at runtime once it's linked to at least one Active rule in the Rule Engine.+ </Accordion>+ <Accordion title="Can I delete a context from its detail page?">+ No. There is no delete control on the context detail page. Contexts are deleted from the list page, using the delete icon on that context's row.+ </Accordion>+ <Accordion title="Can I unlink a rule from the context detail page?">+ Not from that side. Once a rule is linked to a context, removing the link is done from the rule's own detail page, in its Associated Entities panel. See [Link Rules](/latest/openlit/prompts-experiments/context/link-rules) for details.+ </Accordion>+</AccordionGroup>++---++<CardGroup cols={2}>+ <Card title="Rule Engine" href="/latest/openlit/prompts-experiments/rule-engine/overview" icon='sliders'>+ Create conditional rules to match inputs and retrieve linked contexts, prompts, and other resources+ </Card>+ <Card title="Prompt Hub" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='shapes'>+ Version, deploy, and collaborate on prompts with centralized management and tracking+ </Card>+ <Card title="SDK Rule Engine" href="/latest/sdk/features/rule-engine" icon='puzzle-piece'>+ Use evaluate_rule() from Python, TypeScript, or Go with full parameter reference+ </Card>+ <Card title="API Reference: Evaluate" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon='code'>+ Full reference for the Rule Engine evaluate endpoint with request and response schemas+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/context/retrieve-context.mdx b/docs/latest/openlit/prompts-experiments/context/retrieve-context.mdxnew file mode 100644index 00000000..1beff656--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/context/retrieve-context.mdx@@ -0,0 +1,28 @@+---+title: "Retrieve Context"+description: "Fetch matching context content at runtime using the Rule Engine's evaluate API"+---++Contexts become useful once linked to at least one Active rule. Your application calls the Rule Engine's evaluate endpoint with `entity_type: "context"` and a set of input fields (like `model` or `user_tier`); OpenLIT checks all Active rules and returns which rules matched, plus references to their linked contexts.++<Note>+Creating, editing, and deleting contexts requires being logged into OpenLIT (session-based). Only this evaluate/retrieval step supports the Bearer-token API key flow described below.+</Note>++## Create an API Key++- Navigate to **Settings → API Keys** in OpenLIT.+- Click **Create API Key**, enter a name, and save the key securely.++## Call the evaluate API++Send a `POST` request to `/api/rule-engine/evaluate` with your input fields and `entity_type: "context"`. Set `include_entity_data: true` to receive the full context record (`id`, `name`, `description`, `content`, `tags`, `meta_properties`, `status`, `created_by`, `created_at`, `updated_at`) instead of just matching rule IDs and entity references.++<CardGroup cols={2}>+ <Card title="SDK Rule Engine" href="/latest/sdk/features/rule-engine" icon='puzzle-piece'>+ Python, TypeScript, and Go examples for retrieving contexts, including full parameter tables and error handling+ </Card>+ <Card title="API Reference: Evaluate" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon='code'>+ Full request/response schema and a curl example for the evaluate endpoint+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/openground.mdx b/docs/latest/openlit/prompts-experiments/openground.mdxdeleted file mode 100644index 01663bb7..00000000--- a/docs/latest/openlit/prompts-experiments/openground.mdx+++ /dev/null@@ -1,72 +0,0 @@-----title: 'OpenGround'-description: 'Test and compare different LLMs side-by-side based on performance, cost, and other key metrics'-sidebarTitle: 'OpenGround'------OpenGround enables developers to conduct side-by-side tests and comparisons of different Large Language Models (LLMs). By evaluating key factors such as performance and cost, OpenGround assists AI engineers and data scientists in selecting the most suitable LLM for their specific needs. Whether you're optimizing for efficiency, affordability, or other important metrics, OpenGround offers the tools you need to thoroughly assess various models efficiently.--## Features--- **Side-by-Side Comparison**: Simultaneously evaluate multiple LLMs to understand how they perform in real-time across various scenarios.-- **Performance Metrics**: Examine essential performance indicators like response time and token usage to gain deeper insights into each LLM's capabilities.-- **Response Comparison**: Compare the responses generated by different LLMs, assessing the quality, relevance, and appropriateness for specific tasks.-- **Cost Analysis**: Evaluate the cost implications of using different LLMs, helping you balance budget constraints with performance needs.-- **Intuitive Interface**: Use a user-friendly interface that simplifies the process of setting up tests, visualizing results, and making comparisons.-- **Comprehensive Reporting**: Generate detailed reports that compile and visualize comparison data, supporting informed decision-making.--## Get started--<Steps>- <Step title="List existing experiments">- Get a quick overview of all experiments created.- 1. Navigate to the **OpenGround** in OpenLIT.- 2. Explore the previosly created experiments.- <Frame>- <img src="/images/docs-openground-1.png" />- </Frame>- </Step>- <Step title="Create a new experiment">- Set up new experiments to compare different LLMs side-by-side.- 1. Click on **Create new** button to start a new experiment.- 2. In the editor, choose your first LLM provider. Configure the LLM by setting its parameters and enter your API key to enable requests.- 3. Repeat the process for your second LLM provider. Choose the provider and configure its LLM parameters.- 4. Once both LLMs are set up, enter your prompt and click **Compare Response**.- 5. Review and analyze the responses from both LLMs to see how they compare.- <Frame>- <img src="/images/docs-openground-2.png" />- <img src="/images/docs-openground-3.png" />- </Frame>- <Frame>- <img src="/images/docs-openground-4.png" />- <img src="/images/docs-openground-5.png" />- </Frame>- </Step>- <Step title="View experiment details" stepNumber=" ">- Once the experiment is creaetd, You can see information about the experiment along with details.- <Frame>- <img src="/images/docs-openground-5.png" />- </Frame>- </Step>-</Steps>-------<CardGroup cols={3}>- <Card title="Manage LLM secrets" href="/latest/openlit/developer-resources/vault" icon='vault'>- Centrally store LLM API keys that applications can retrieve remotely without restarts- </Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking- </Card>-</CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/openlit/prompts-experiments/openground/overview.mdx b/docs/latest/openlit/prompts-experiments/openground/overview.mdxnew file mode 100644index 00000000..f256de00--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/openground/overview.mdx@@ -0,0 +1,52 @@+---+title: "Overview"+description: "OpenGround is OpenLIT's AI model comparison tool - an LLM playground for running one prompt across many providers at once and comparing cost, speed, and quality"+---++OpenGround is OpenLIT's **AI model comparison tool** - an LLM playground built for a specific job: run **one prompt across many providers and models at once**, then compare the results side-by-side. Pick a prompt, pick the providers you want to test, click **Evaluate providers**, and see cost, speed, and quality laid out next to each other. Every run is saved automatically to a searchable history, so you can build up a record of what you've tested and reopen or reload any past run later.++## What you get++<CardGroup cols={2}>+ <Card title="Run a Comparison" href="/latest/openlit/prompts-experiments/openground/run-a-comparison" icon='flask'>+ Configure a prompt, pick providers and models, tune settings, and evaluate+ </Card>+ <Card title="Results & History" href="/latest/openlit/prompts-experiments/openground/results-and-history" icon='chart-line'>+ Read the metrics, response-time waterfall, and cost breakdown, then find any past run again later+ </Card>+</CardGroup>++## Model and provider management++Model and provider management itself - adding providers and editing pricing - lives on **Monitor → Costs → Manage models**. OpenGround and Manage models share the same underlying providers and models, so anything you configure in one is immediately available in the other.++---++<CardGroup cols={3}>+ <Card title="Manage models" href="/latest/openlit/costs/manage-models/overview" icon='sliders'>+ Add custom models, edit pricing, and manage providers used across OpenGround, Chat, and Pricing+ </Card>+ <Card title="Manage LLM secrets" href="/latest/openlit/developer-resources/vault/overview" icon='vault'>+ Centrally store LLM API keys that applications can retrieve remotely without restarts+ </Card>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>+ Version, deploy, and collaborate on prompts with centralized management and tracking+ </Card>+</CardGroup>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is OpenGround used for?">+ OpenGround runs the same prompt across multiple LLM providers and models at once, so you can compare cost, speed, and response quality side-by-side before choosing a model for production.+ </Accordion>+ <Accordion title="How many providers are supported out of the box?">+ 14: OpenAI, Anthropic, Google, Mistral, Groq, Perplexity, Azure, Cohere, Together AI, Fireworks, DeepSeek, xAI, Hugging Face, and Replicate. See [Run a Comparison](/latest/openlit/prompts-experiments/openground/run-a-comparison) for how to configure one.+ </Accordion>+ <Accordion title="Can I test a model that isn't built in?">+ Yes. When configuring a provider in OpenGround, switch to "Use Custom Model" and type any model ID directly, with its own input/output pricing. There's no field for a custom endpoint or base URL - only the model ID and its pricing are configurable.+ </Accordion>+ <Accordion title="Are comparison runs saved automatically?">+ Yes. Every run is saved to a searchable history with cost, speed, and token-efficiency summaries - there's no separate save step. See [Results & History](/latest/openlit/prompts-experiments/openground/results-and-history) for how to find and reopen past runs.+ </Accordion>+</AccordionGroup>diff --git a/docs/latest/openlit/prompts-experiments/openground/results-and-history.mdx b/docs/latest/openlit/prompts-experiments/openground/results-and-history.mdxnew file mode 100644index 00000000..6310145d--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/openground/results-and-history.mdx@@ -0,0 +1,59 @@+---+title: "Results & History"+description: "Read the metrics overview, response-time waterfall, and cost breakdown for a run, then find, reopen, or reload it later from history"+---++## Reading a run's results++Once you click **Evaluate providers**, results appear on the same page in a few sections.++### Metrics overview++Four stat cards summarize the run: **Fastest Response**, **Lowest Cost**, **Most Efficient**, and **Success Rate**.++### Response Time Comparison++A horizontal waterfall of response times, fastest at the top with a **Fastest** badge, plus a fastest/average/slowest summary underneath.++### Cost Breakdown++Per-provider cost cards showing prompt, completion, and total tokens plus total cost. The lowest-cost provider gets a **Cheapest** badge, and a total/average summary sits at the bottom.++### Provider Responses++One card per provider, showing:++- The model used+- Response time, tokens, and cost+- A **Success** or **Error** badge+- The actual response text (or the error message, if it failed)+- A collapsible **Show/Hide Raw Response Data** section with the full response payload++<Tip>+Once you've settled on a model, see [SDK pricing URL](/latest/openlit/costs/manage-models/sdk-pricing) for ready-to-paste Python, TypeScript, and Go SDK snippets - that dialog lives on Costs → Manage models, not in OpenGround.+</Tip>++## History++Every run is saved automatically - there's no separate save action. Find it later on the OpenGround history page (`/openground`), where the table lists:++| Column | What it shows |+|---|---|+| Prompt | The prompt text, with a small icon if it came from Prompt Hub |+| Providers | Count of successful vs. failed calls |+| Best Cost | The cheapest provider in that run |+| Best Speed | The fastest provider in that run |+| Best Efficiency | The most token-efficient provider in that run |+| Created | When the run happened |+| Actions | An eye icon and a reload/rerun icon |++- Click the **eye icon** to open the run's detail page (`/openground/[id]`), titled **"Run Details"**. It shows the same metrics overview, response-time waterfall, and cost breakdown as the original run, plus which prompt and providers were used, with a link back to the Prompt Hub prompt if one was used.+- Click the **reload/rerun icon** to load that run's exact setup into a new comparison. You'll see the note *"Evaluation loaded. Configure providers and run again."* and land on the new-comparison page, ready to tweak and re-run.++If the history is empty, you'll see a **"Get Started with Openground"** message with a **Create New Playground** call to action.++---++<Card title="Run a Comparison" href="/latest/openlit/prompts-experiments/openground/run-a-comparison" icon='flask'>+ Configure a prompt, pick providers and models, and evaluate+</Card>diff --git a/docs/latest/openlit/prompts-experiments/openground/run-a-comparison.mdx b/docs/latest/openlit/prompts-experiments/openground/run-a-comparison.mdxnew file mode 100644index 00000000..c08a9f00--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/openground/run-a-comparison.mdx@@ -0,0 +1,47 @@+---+title: "Run a Comparison"+description: "Configure a prompt, select providers, tune per-provider settings, and evaluate on the /openground/new page"+---++From the [OpenGround history page](/latest/openlit/prompts-experiments/openground/results-and-history) (`/openground`), click **Create New Playground** to open the comparison runner at `/openground/new`.++## Step 1: Prompt configuration++The first card has two tabs:++- **Custom** - a free-text prompt box (placeholder: *"Enter your prompt... Use `{{variable}}` for dynamic values"*).+- **Prompt Hub** - a dropdown (placeholder: *"Select a prompt from Prompt Hub"*) to pick a saved prompt from [Prompt Hub](/latest/openlit/prompts-experiments/prompt-hub/overview). Once selected, a version-tagged preview of the prompt's content is shown.++Either way, if the prompt text contains `{{variable}}` placeholders, a **Variables** section automatically appears below with one input per variable, along with the note *"These values will be substituted into your prompt before evaluation."*++## Step 2: Select providers to compare++A grid of provider cards follows, each showing an avatar, name, model count, and a status badge - **Configured** or **Configure**.++- Clicking a **Configured** card adds it to the comparison immediately, using its default model.+- Clicking an unconfigured card opens a setup dialog where you pick an **API Key (from Vault)** - with a shortcut to create one on the spot if your Vault is empty - and an optional default model, then click **Save Configuration**.++As soon as you've added at least one provider, a banner shows **"{n} Providers Selected"**.++OpenGround ships with **14 built-in providers**: OpenAI, Anthropic, Google, Mistral, Groq, Perplexity, Azure, Cohere, Together AI, Fireworks, DeepSeek, xAI, Hugging Face, and Replicate. These are the same providers you'd see on [Manage models](/latest/openlit/costs/manage-models/overview) under Costs - anything you add or configure there is immediately available here, and vice versa.++## Step 3: Provider settings++Once you've selected at least one provider, a settings panel appears per provider:++- **Model** - a dropdown of that provider's supported models, each showing its context-window size and price per million tokens. You can also switch to **Use Custom Model** to type an arbitrary model ID directly, with its own input/output pricing.+- A **Manage models** shortcut opens model management - the same catalog as [Costs → Manage models](/latest/openlit/costs/manage-models/overview).+- Three sliders per provider:+ - **Temperature** - `0` to `2`+ - **Max Tokens** - `100` to `4000`+ - **Top P** - `0` to `1`++<Note>+There's no field for a custom endpoint or base URL. You can point a provider at a custom model ID with your own pricing, but not at a self-hosted or custom API endpoint.+</Note>++## Run it++Click **Evaluate providers**. While it runs, the button shows *"Evaluating providers... This may take a few seconds."* Once results land, a **Reset** button appears so you can clear everything and start over.++See [Results & History](/latest/openlit/prompts-experiments/openground/results-and-history) for how to read the output and find the run again later.diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub.mdxdeleted file mode 100644index 7bca3cdc..00000000--- a/docs/latest/openlit/prompts-experiments/prompt-hub.mdx+++ /dev/null@@ -1,221 +0,0 @@-----title: 'Prompt Hub'-description: 'Manage prompts centrally, fetch versions, and use variables for dynamic prompts'------The Prompt-Hub allows you to manage prompts, fetch specific versions of the prompt, and compile prompts with variables.--## Key features--- **Prompt Management**: Create, edit, and track different versions of your prompts.-- **Versioning**: Supports major, minor, and patch updates for clear version management.-- **Dynamic Variables**: Use `{{variableName}}` placeholders that are dynamically replaced at runtime.-- **AI Prompt Improvement**: Use Otter to review a prompt, suggest targeted edits, and accept or decline changes directly in the editor.-- **Statistics**: View download stats and version history directly in the UI.---## Get started--<Steps>- <Step title="List prompts">- Get a quick overview of all prompts created.- 1. Navigate to the Prompt Hub in OpenLIT.- 2. Explore the available prompts listed.- <Frame>- <img src="/images/docs-prompt-hub-list.png" />- </Frame>- </Step>- <Step title="Create or edit a prompt">- Build new prompts or edit existing ones with ease.- 1. Click on **Create new** button to create a new prompt- 2. In the prompt editor, add the prompt name, Meta Properties and use placeholders like `{{variableName}}` to represent dynamic data that will be substituted when the prompt is compiled.- 3. Specify major, minor, or patch updates for versioning as you create or modify prompts.- 4. Click the Otter action next to the prompt editor to review and improve the prompt before saving.- <Frame>- <img src="/images/docs-prompt-hub-create.png" />- </Frame>- </Step>- <Step title="View prompt details" stepNumber=" ">- Once the prompt is creaetd, You can see information about the prompt along with details on all past versions.- <Frame>- <img src="/images/docs-prompt-hub-details.png" />- </Frame>- </Step>- <Step title="Retrieve the prompt">- <Steps>- <Step title="Create an API Key">- To authenticate your requests, you need an API key. Here's how you can create one:-- - Go to the OpenLIT.- - Navigate to the **API Keys** page.- - Click on **Create API Key**.- - Enter a name for your API key.- - Save the API key displayed. Ensure you store it securely as it will be used for authentication in the SDK.- </Step>- <Step title="Get prompt using the SDK">- <Tabs>- <Tab title="Python">- Here's how you can fetch and compile a prompt in Python:-- ```python- import openlit-- response = openlit.get_prompt(- url="http://127.0.0.1:3000",- api_key="_OPENLIT_API_KEY",- name="prompt_name",- should_compile=True,- variables={- "name": "John",- },- )-- print(response)- ```-- ```shell Output- {- err: null,- res: {- promptId: '88c4cbcd-87f9-4957-a37b-b41066e17471',- name: 'prompt_name',- version: '3.3.3',- tags: [ 'user', 'greeting' ],- metaProperties: { model: 'gpt4' },- prompt: 'Hello {{name}}, how are you today?',- compiledPrompt: 'Hello John, how are you today?'- }- }- ```- ### SDK parameters-- Below are the parameters for use with the SDK, formatted to indicate whether each is required or optional:-- | Parameter | Description |- |-------------------|------------------------------------------------------------------------------------------------------------------------------------|- | `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable. |- | `api_key` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |- | `name` | Sets the name to fetch a unique prompt. Use this or `prompt_id`. |- | `prompt_id` | Sets the ID to fetch a unique prompt. Use this or `name`. Optional |- | `version` | Set to `True` to get the prompt with variable substitution.. Optional |- | `shouldCompile` | Boolean value that compiles the prompt using the provided variables. Optional |- | `variables` | Sets the variables for prompt compilation. Optional |- | `meta_properties` | Sets the meta-properties for storing in the prompt's access history metadata. Optional |-- </Tab>- <Tab title="Typescript">- Here's how you can fetch and compile a prompt in Typescript/Javascript:-- ```typescript- import Openlit from 'openlit'; // Import OpenLIT-- const response = await Openlit.getPrompts({- name: "prompt_name", // Fetch the prompt by name- shouldCompile: true, // Compile the prompt with provided variables- variables: {- name: "John", // Pass variables for prompt compilation- }- });-- console.log(response); // Print or process the fetched and compiled prompt- ```-- ```shell Output- {- err: null,- res: {- promptId: '88c4cbcd-87f9-4957-a37b-b41066e17471',- name: 'prompt_name',- version: '3.3.3',- tags: [ 'user', 'greeting' ],- metaProperties: { model: 'gpt4' },- prompt: 'Hello {{name}}, how are you today?',- compiledPrompt: 'Hello John, how are you today?'- }- }- ```- ### SDK parameters-- Below are the parameters for use with the SDK, formatted to indicate whether each is required or optional:-- | Parameter | Description |- |------------------|------------------------------------------------------------------------------------------------------------------------------------|- | `url` | Sets the OpenLIT URL. Defaults to the `OPENLIT_URL` environment variable or `http://127.0.0.1:3000` if not set. |- | `apiKey` | Sets the OpenLIT API Key. Can also be provided via the `OPENLIT_API_KEY` environment variable. |- | `name` | Sets the name to fetch a unique prompt. Use this or `promptId`. |- | `promptId` | Sets the ID to fetch a unique prompt. Use this or `name`. Optional |- | `version` | Sets the version to retrieve a specific prompt. Optional |- | `shouldCompile` | Boolean value that compiles the prompt using the provided variables. Optional |- | `variables` | Sets the variables for prompt compilation. Optional |- | `metaProperties` | Sets the meta-properties for storing in the prompt's access history metadata. |-- </Tab>- <Tab title="API">- To retrieve the prompt via an API request, Refer to the [**GET Prompt**](/latest/openlit/developer-resources/api-reference/endpoint/prompt-hub/get) API Reference Documentation.- </Tab>- </Tabs>- </Step>- </Steps>- </Step>-</Steps>--## AI prompt improvement--Prompt Hub includes an inline Otter assistant for prompt review. It helps you improve prompt wording while keeping control over what is saved.--Use prompt improvement when you want to:--- Remove redundant wording and make the prompt more concise.-- Make instructions easier to scan and follow.-- Preserve variables such as `{{userInput}}` or `{{context}}` exactly.-- Clarify output format, constraints, and success criteria.-- Reduce ambiguity without changing the prompt's original intent.--### How it works--1. Open a prompt in create or edit mode.-2. Click the Otter action next to the prompt editor.-3. Review the default improvement dimensions, remove any that are not relevant, or add your own.-4. Run the prompt analysis.-5. Review inline suggestions in the editor.-6. Accept or decline each suggestion.-7. Save the prompt or create a new version when you are ready.--Suggestions are local to the editor until you accept them and save the prompt. On the new prompt screen, prior improvement runs are not attached to the prompt history until the prompt itself is saved.--<Note>- Otter usage for prompt improvement is tracked in Otter usage reports, but review-only prompt improvement does not create normal chat conversation history.-</Note>--### Improve prompts from Otter chat--You can also ask Otter to review an existing Prompt Hub prompt by name or ID:--```-"Can you help me improve prompt with prompt name music_recommend?"-"Review prompt customer_summary for clarity and concision"-"Suggest better output-format instructions for prompt_id <uuid>"-```--For review-only requests, Otter loads the prompt and suggests improvements without updating the saved version. It only saves changes when you explicitly ask it to save, apply, update, publish, or create a new version.-------<CardGroup cols={3}>- <Card title="Manage LLM secrets" href="/latest/openlit/developer-resources/vault" icon='vault'>- Centrally store LLM API keys that applications can retrieve remotely without restarts- </Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring- </Card>- <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground" icon='flask'>- Compare cost, duration, and response tokens across different LLMs to find the most efficient model- </Card>-</CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub/ai-improvement.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub/ai-improvement.mdxnew file mode 100644index 00000000..a3145800--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/prompt-hub/ai-improvement.mdx@@ -0,0 +1,52 @@+---+title: "AI Prompt Improvement"+description: "Let Otter review a prompt and suggest targeted edits directly in the create and edit editors"+---++While creating or editing a prompt, a collapsible bar titled **"Improve prompt with Otter"** sits above the prompt content editor. It's not available on the read-only detail page - only in create and edit mode. Click **Improve** to run it; it shows "Analyzing prompt..." while working.++## Review dimensions++Otter reviews your prompt against a set of dimensions, shown as removable chips. Five are included by default:++1. Be concise and remove redundant wording.+2. Make the instruction structure easier to follow.+3. Preserve variables like `{{variableName}}` exactly.+4. Clarify output format, constraints, and success criteria.+5. Reduce ambiguity without changing the prompt intent.++Remove any that aren't relevant, or add your own via the text box before running the review.++## Reviewing and applying suggestions++Each suggestion appears as a small diff - removed text in red, added text in green - with a short rationale underneath. You can **Accept** (green check) or **Decline** (red X) each suggestion individually.++Accepting a suggestion only updates the text in your editor - nothing is saved until you click Save Prompt (or Save, on the edit page) separately. If you've changed the prompt text since a suggestion was generated, accepting it can fail with "This suggestion no longer matches the current prompt." If Otter doesn't find anything worth changing, you'll see "Otter did not find any precise prompt changes."++Otter always preserves `{{variable}}` placeholders exactly as written - it's instructed never to alter what's inside the double braces, no matter what other wording changes it suggests.++<Note>+Otter needs a configured chat model to run this review. If none is set up, you'll see "Chat configuration not found. Configure Otter first."+</Note>++## Reviewing a prompt from Otter chat++You don't have to be on the create or edit page to get a review - you can also ask Otter to look at a saved prompt from a general chat conversation, by name or ID:++```+"Can you help me improve prompt with prompt name music_recommend?"+"Review prompt customer_summary for clarity and concision"+```++For a review-only request like this, Otter loads the prompt and suggests improvements without saving anything. It only saves, updates, publishes, or creates a new version when you explicitly ask it to. See [Chat with Otter](/latest/openlit/chat/conversations) for the general chat surface.++---++<CardGroup cols={2}>+ <Card title="Manage Prompts" href="/latest/openlit/prompts-experiments/prompt-hub/manage-prompts" icon='file-lines'>+ Where the Otter assistant bar sits on the create and edit pages+ </Card>+ <Card title="Versioning" href="/latest/openlit/prompts-experiments/prompt-hub/versioning" icon='code-branch'>+ Save your accepted edits as a draft or a new published version+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub/manage-prompts.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub/manage-prompts.mdxnew file mode 100644index 00000000..36847535--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/prompt-hub/manage-prompts.mdx@@ -0,0 +1,80 @@+---+title: "Manage Prompts"+description: "Create, edit, tag, and delete prompts from the Prompt Hub list and detail pages"+---++The **Prompt Hub** list page (`/prompt-hub`) is where every prompt in your project lives. The header has a **Learn more** button that links back to these docs, and a **Create new** button that appears once your database connection is healthy.++If you haven't created a prompt yet, you'll see a "Get Started with Prompt Hub" screen highlighting four things Prompt Hub gives you: Version Control, Variable Support, Team Collaboration, and Usage Tracking.++<Frame>+ <img src="/images/docs-prompt-hub-list.png" alt="List of prompts in Prompt Hub" />+</Frame>++## The prompt list++Each row in the table shows:++| Column | What it shows |+|---|---|+| Name | The prompt's name |+| Created By | Who created the prompt |+| Latest Version | The newest version number, or `draft` (lowercase) if nothing's been published yet |+| Downloads | Total fetch count, or `-` if the prompt hasn't been published |+| Last Released On | The date the latest version was published, or `-` if unpublished |+| Actions | A sliders icon to create and link a [rule](/latest/openlit/prompts-experiments/prompt-hub/rules) for this prompt, and a trash icon to delete it |++Clicking anywhere else on a row opens that prompt's detail page. Deleting a prompt asks you to confirm: "Are you sure you want to delete this prompt?"++## Creating a prompt++Click **Create new** to open the create page (`/prompt-hub/new`). The header has **Back** and **Save Prompt** buttons.++<Frame>+ <img src="/images/docs-prompt-hub-create.png" alt="Create or edit a prompt editor in Prompt Hub" />+</Frame>++- **Name** - required. As you type, the input is automatically lowercased, spaces become underscores, and any character outside `a-z` and `_` is stripped, matching the hint "(lowercase letters and _ only)". The name can't be changed once the prompt is created.+- **Prompt content editor** - has a hint to "Use `{{variableName}}` for dynamic variables", an inline "Improve prompt with Otter" assistant bar above it (see [AI Prompt Improvement](/latest/openlit/prompts-experiments/prompt-hub/ai-improvement)), and **Write** / **Preview** tabs below - Write is a plain text editor with line numbers, and Preview renders your content as Markdown (showing "Nothing to preview yet." if the editor is empty).+- **Version** - four cards: Draft, Major, Minor, or Patch. See [Versioning](/latest/openlit/prompts-experiments/prompt-hub/versioning) for what each one does.+- **Tags** - add free-form tag chips by pressing Enter or clicking the "+" button; remove them individually.+- **Meta Properties** - repeatable Key/Value rows for any metadata you want to attach; add rows with **Add property** and remove them individually.++Both Name and content are required - you'll see "Prompt name is required" or "Prompt content is required" if you try to save without them. Saving shows a "Creating prompt..." loading toast, then "Prompt created successfully!" or "Failed to create prompt". If the name is already taken, saving fails with "Prompt name is already taken!".++## Viewing a prompt++The detail page (`/prompt-hub/[id]`) shows the prompt's name (with a **Draft** badge if the version you're viewing is the draft), its version number, and "Published on {date}". Below that: tag badges, the read-only rendered prompt content, and a Meta Properties table if any were set.++<Frame>+ <img src="/images/docs-prompt-hub-details.png" alt="Prompt details page showing past versions" />+</Frame>++The action button changes depending on what you're looking at:++- **Publish Version** - shown when you're viewing the draft, and takes you to the edit page to finish and publish it.+- **Create New Version** - shown when you're viewing the latest published version and there's no pending draft, and takes you to the edit page to start a new version from it.++On the right side of the detail page are two tabs: **Versions** (covered in [Versioning](/latest/openlit/prompts-experiments/prompt-hub/versioning)) and **Rules** (covered in [Rules](/latest/openlit/prompts-experiments/prompt-hub/rules)).++You can also open any past version directly by adding `?version=X` to the detail page URL.++## Editing a prompt++The edit page (`/prompt-hub/[id]/edit`) has the same layout as the create page, minus the Name field. It loads your current draft if one exists, or otherwise the latest published version as a starting point for a new one. The subheading tells you which mode you're in:++- "Editing draft - publish when ready" - you're modifying the existing draft.+- "Creating a new version from the latest published" - you're starting a new version on top of the latest published one.++The same Draft / Major / Minor / Patch version picker appears here too, except the Draft option reads "No version change" / "Keep as draft - not published" instead of the create page's wording. The Save button shows "Saving..." while in progress, then "Prompt saved!" on success or "Failed to save prompt." on failure.++---++<CardGroup cols={2}>+ <Card title="Versioning" href="/latest/openlit/prompts-experiments/prompt-hub/versioning" icon='code-branch'>+ What Draft, Major, Minor, and Patch actually do+ </Card>+ <Card title="Retrieve a Prompt" href="/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt" icon='cloud-arrow-down'>+ Fetch and compile a saved prompt from your application+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub/overview.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub/overview.mdxnew file mode 100644index 00000000..81d7a59e--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/prompt-hub/overview.mdx@@ -0,0 +1,57 @@+---+title: "Overview"+description: "Prompt Hub is OpenLIT's prompt management platform for versioning, publishing, and fetching prompts with dynamic variables at runtime"+---++Prompt Hub is OpenLIT's prompt management platform: a place to create prompts, save them as explicit versions, and fetch them at runtime from your application instead of hardcoding prompt text in your codebase. It's built for teams who want prompt changes to ship independently of a code deploy, with a full history of what changed and when.++At a high level, Prompt Hub covers three things: **creating and editing** prompts in a dedicated editor with variable support, **versioning** each save as either a draft or a permanent published version, and **fetching** a prompt (by name or ID, optionally a specific version) from the SDK or API with `{{variableName}}` placeholders compiled in for you.++<CardGroup cols={2}>+ <Card title="Manage Prompts" href="/latest/openlit/prompts-experiments/prompt-hub/manage-prompts" icon='file-lines'>+ Create, edit, tag, and delete prompts from the list and detail pages+ </Card>+ <Card title="Versioning" href="/latest/openlit/prompts-experiments/prompt-hub/versioning" icon='code-branch'>+ Drafts, semantic versions, and how the Versions tab tracks history+ </Card>+ <Card title="AI Prompt Improvement" href="/latest/openlit/prompts-experiments/prompt-hub/ai-improvement" icon='wand-magic-sparkles'>+ Let Otter review a prompt and suggest edits directly in the editor+ </Card>+ <Card title="Rules" href="/latest/openlit/prompts-experiments/prompt-hub/rules" icon='sliders'>+ Link a prompt to Rule Engine rules so it's returned automatically at evaluation time+ </Card>+ <Card title="Retrieve a Prompt" href="/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt" icon='cloud-arrow-down'>+ Fetch and compile prompts at runtime via the SDK or API+ </Card>+</CardGroup>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is prompt management with Prompt Hub?">+ Prompt Hub lets you treat prompts as versioned artifacts - create them once, save changes as a draft or a permanent version, and fetch them at runtime via SDK or API instead of hardcoding prompt text in your application.+ </Accordion>+ <Accordion title="Can I roll back a prompt version?">+ There's no rollback or restore button in the UI - viewing an older version on the Versions tab is read-only. To bring old content back into the editor, you'd copy it into a new edit and save it as a new version. If you just want your application to use an older version's text, fetch that specific version directly via the SDK/API `version` parameter without touching the published history in the UI.+ </Accordion>+ <Accordion title="How do dynamic variables work?">+ Use `{{variableName}}` placeholders anywhere in a prompt's content, then pass a `variables` map when fetching it via the SDK or API with compilation enabled - OpenLIT substitutes the values into the text at request time.+ </Accordion>+ <Accordion title="Do I need to publish a version, or can I just save a draft?">+ Both are supported. Saving as a Draft keeps your work private and repeatedly editable without publishing anything. Choosing Major, Minor, or Patch always publishes a new, permanent version - publishing never overwrites an earlier one.+ </Accordion>+</AccordionGroup>++---++<CardGroup cols={3}>+ <Card title="Manage LLM secrets" href="/latest/openlit/developer-resources/vault/overview" icon='vault'>+ Centrally store LLM API keys that applications can retrieve remotely without restarts+ </Card>+ <Card title="LLM playground" href="/latest/openlit/prompts-experiments/openground/overview" icon='flask'>+ Run a saved prompt across multiple LLMs side-by-side and compare cost, speed, and quality+ </Card>+ <Card title="Rule Engine" href="/latest/openlit/prompts-experiments/rule-engine/overview" icon='gavel'>+ Match runtime inputs against conditions to return prompts and other linked resources automatically+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt.mdxnew file mode 100644index 00000000..d575a9a3--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt.mdx@@ -0,0 +1,42 @@+---+title: "Retrieve a Prompt"+description: "Fetch a saved prompt by name or ID, optionally a specific version, and compile it with dynamic variables via the SDK or API"+---++Once a prompt exists in Prompt Hub, your application fetches it at runtime instead of hardcoding the text. You can look it up by name or by ID, and optionally ask for a specific version - if you don't, you get the latest available content. If you don't ask for compilation, you get the raw prompt text back untouched, including any `{{variableName}}` placeholders still in place.++## Dynamic variables++Any `{{variableName}}` placeholder in a prompt's content is replaced with the value you pass at fetch time when compilation is enabled. It's plain text substitution - there's no escaping, no nested variables, and no default values, so make sure you pass a value for every placeholder the prompt uses.++Every fetch is counted, so the Downloads column on the prompt list and the per-version counts on the [Versions tab](/latest/openlit/prompts-experiments/prompt-hub/versioning) reflect real usage from your application.++## Create an API key++To authenticate SDK or API requests, you need an API key:++<Steps>+ <Step title="Create an API Key">+ - Go to OpenLIT and navigate to the **API Keys** page.+ - Click **Create API Key**.+ - Enter a name for your API key.+ - Save the key displayed - store it securely, since you'll pass it to the SDK for authentication.+ </Step>+</Steps>++## Get a prompt using the SDK or API++<Card title="GET Prompt API reference" href="/latest/openlit/developer-resources/api-reference/endpoint/prompt-hub/get" icon='cloud-arrow-down'>+ Full request/response schema, plus Python and TypeScript SDK examples for fetching and compiling a prompt+</Card>++---++<CardGroup cols={2}>+ <Card title="Versioning" href="/latest/openlit/prompts-experiments/prompt-hub/versioning" icon='code-branch'>+ Fetch a specific version by number instead of always getting the latest+ </Card>+ <Card title="Manage Prompts" href="/latest/openlit/prompts-experiments/prompt-hub/manage-prompts" icon='file-lines'>+ Where prompts, names, and IDs come from in the first place+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub/rules.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub/rules.mdxnew file mode 100644index 00000000..b56f4e41--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/prompt-hub/rules.mdx@@ -0,0 +1,34 @@+---+title: "Rules"+description: "Link a prompt to Rule Engine rules so it's returned automatically when a rule matches at evaluation time"+---++The prompt detail page has a **Rules** tab, labeled with a badge showing how many rules are already linked. Linking a rule to a prompt means that when the rule matches at evaluation time, the prompt is returned as one of the rule's linked resources.++## Linking a rule++You can link a rule to the current prompt two ways:++- **New Rule** - opens a dialog to create a brand-new rule that's automatically linked to this prompt.+- **Link existing rule** - a toggle that opens a dropdown of rules not yet linked to this prompt (or "All rules already linked" if there are none left to add), plus an **Associate** button to confirm the link.++Each linked rule is listed with its name, description, and status. Clicking a linked rule opens that rule's own detail page. If nothing is linked yet, the tab shows "No rules linked yet."++## Learn more about the Rule Engine++Rules themselves - conditions, matching logic, and what else can be linked besides prompts - are covered in the Rule Engine docs:++<CardGroup cols={2}>+ <Card title="Rule Engine Overview" href="/latest/openlit/prompts-experiments/rule-engine/overview" icon='gavel'>+ How rules, conditions, and matching work+ </Card>+ <Card title="Linked Entities" href="/latest/openlit/prompts-experiments/rule-engine/linked-entities" icon='link'>+ Everything a rule can return when it matches, prompts included+ </Card>+</CardGroup>++---++<Card title="Manage Prompts" href="/latest/openlit/prompts-experiments/prompt-hub/manage-prompts" icon='file-lines'>+ Back to the prompt list, create, and detail pages+</Card>diff --git a/docs/latest/openlit/prompts-experiments/prompt-hub/versioning.mdx b/docs/latest/openlit/prompts-experiments/prompt-hub/versioning.mdxnew file mode 100644index 00000000..7a80ec87--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/prompt-hub/versioning.mdx@@ -0,0 +1,45 @@+---+title: "Versioning"+description: "Drafts vs. published versions, semantic versioning, and what the Versions tab shows"+---++Every save on a prompt's create or edit page goes through the same four-option picker: **Draft**, **Major**, **Minor**, or **Patch**. You choose which one applies every time - Prompt Hub never infers a version bump automatically from how much the content changed.++## Draft vs. publishing++- **Draft** - "Save as a draft - not yet published." Saves your work without publishing it. A draft is mutable: you can save over it repeatedly while you're still iterating, and it never shows up as a numbered version until you publish.+- **Major** - "Significant changes, not backwards compatible" (shown as v1.0.0).+- **Minor** - "New features, backwards compatible" (shown as v0.1.0).+- **Patch** - "Bug fixes and minor updates" (shown as v0.0.1).++Choosing Major, Minor, or Patch always publishes a new, permanent version. Publishing never overwrites an earlier version - each one is retained. On the edit page, the same four options appear, but the Draft option reads "No version change" / "Keep as draft - not published" instead.++Version numbers follow semantic versioning (`major.minor.patch`), starting at `0.0.0` for a brand-new prompt.++## The Versions tab++The prompt detail page has a **Versions** tab that lists every published version, plus the draft if one exists. Each entry shows:++- A **latest** badge on the newest published version.+- A **Draft** badge on the draft, if there is one.+- The download count for that version.+- A relative "updated X ago" timestamp.++Clicking any version opens its content in read-only view (the same as adding `?version=X` to the detail page URL).++## There's no rollback button++Viewing an older version through the Versions tab is read-only - there's no "restore" or "rollback" action in the UI that makes an old version become the new draft or the new published head. If you want to bring old content back into circulation, copy it into a new edit and save it as a new version yourself.++If what you actually want is for your application to use an older version's text without touching the UI's published history at all, fetch that specific version directly - see [Retrieve a Prompt](/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt) for the SDK/API `version` parameter. That's the practical way to "roll back" what your application uses, independent of what's marked latest in the UI.++---++<CardGroup cols={2}>+ <Card title="Manage Prompts" href="/latest/openlit/prompts-experiments/prompt-hub/manage-prompts" icon='file-lines'>+ Where Draft, Major, Minor, and Patch appear on the create and edit pages+ </Card>+ <Card title="Retrieve a Prompt" href="/latest/openlit/prompts-experiments/prompt-hub/retrieve-a-prompt" icon='cloud-arrow-down'>+ Fetch a specific version by name, ID, or version number+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine.mdx b/docs/latest/openlit/prompts-experiments/rule-engine.mdxdeleted file mode 100644index d7a4f5cf..00000000--- a/docs/latest/openlit/prompts-experiments/rule-engine.mdx+++ /dev/null@@ -1,289 +0,0 @@-----title: 'Rule Engine'-description: 'Define conditional rules with AND/OR logic to match runtime inputs and retrieve linked AI resources'------The Rule Engine lets you define flexible matching rules that evaluate incoming field values against conditions. When a rule matches, it returns references to linked entities — such as contexts, prompts, datasets, or meta configs — along with their full data if requested. This enables dynamic, condition-driven retrieval of AI resources at runtime.--## Key features--- **Condition groups**: Organise conditions into groups. Each group uses its own **AND/OR** logic, and groups are combined using a top-level **AND/OR** group operator.-- **Rich operators**: Supports `equals`, `not_equals`, `contains`, `not_contains`, `starts_with`, `ends_with`, `regex`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte`, and `between` across `string`, `number`, and `boolean` data types.-- **Entity linking**: Associate a rule with one or more entities — **Context**, **Prompt**, **Dataset**, or **Meta Config** — so matching rules return the relevant resources.-- **Status control**: Enable or disable rules without deletion using **Active** / **Inactive** status.-- **External API**: Evaluate rules from any application using Bearer token authentication, without requiring a dashboard session.--## How it works--```-Input fields → Rule Engine → Matching rules → Linked entity data-{ model: "gpt-4", (evaluates all (context content,- user_tier: "premium" } ACTIVE rules) compiled prompt, ...)-```--The evaluate API receives a flat key-value map of input fields and an `entity_type` filter. It runs all active rules against those inputs and returns the IDs of matching rules plus their linked entities (optionally with full data).--## Get started--<Steps>- <Step title="List rules">- <Frame>- <img src="/images/rule-engine-list.png" />- </Frame>-- 1. Navigate to **Rule Engine** in the OpenLIT sidebar.- 2. Browse existing rules with their name, status, and group operator.- </Step>- <Step title="Create a rule">- 1. Click **Create Rule** in the top-right corner.- 2. Enter a **Name** (required) and optional **Description**.- 3. Choose the top-level **Group Operator** — **AND** means all condition groups must match; **OR** means any group must match.- 4. Set **Status** to *Active* or *Inactive*.- 5. Click **Create**.- </Step>- <Step title="Add condition groups">- After creating a rule, open its detail page to add conditions.-- <Frame>- <img src="/images/rule-engine-conditions.png" />- </Frame>- 1. Click **Add Condition Group**.- 2. Choose the group's **Condition Operator** (AND / OR).- 3. Add one or more conditions:- - **Field**: The input key to match against (e.g. `model`, `user_tier`, `token_count`).- - **Operator**: One of the supported comparison operators.- - **Value**: The value to compare against.- - **Data Type**: `string`, `number`, or `boolean`.- 4. Add more groups as needed. Save all changes with **Save Conditions**.-- **Example**: Match requests where `model` equals `gpt-4` **AND** `token_count` is greater than `1000`:- ```- Group 1 (AND):- field=model, operator=equals, value=gpt-4, data_type=string- field=token_count, operator=gt, value=1000, data_type=number- ```- </Step>- <Step title="Link entities">- Rules become actionable when they reference resources to return.- 1. On the rule detail page, scroll to the **Linked Entities** panel.- 2. Select an **Entity Type** (Context, Prompt, Dataset, or Meta Config).- 3. Enter the **Entity ID** or select from the dropdown.- 4. Click **Add Entity**.-- When the rule matches, all linked entities are returned in the evaluate response.-- You can also link rules to contexts or prompts directly from their detail pages.- </Step>- <Step title="Evaluate rules via API">- <Steps>- <Step title="Create an API Key">- - Navigate to **Settings → API Keys** in OpenLIT.- - Click **Create API Key**, enter a name, and save the key securely.- </Step>- <Step title="Call the evaluate endpoint">- Send a `POST` to `/api/rule-engine/evaluate` using curl or one of the OpenLIT SDKs.-- **Required fields:**- - `entity_type` — Filter results to a specific entity type: `context`, `prompt`, `dataset`, or `meta_config`.- - `fields` — Key-value map of input values to evaluate against rule conditions.-- **Optional fields:**- - `include_entity_data` — Set to `true` to fetch full entity records in the response.- - `entity_inputs` — Extra parameters specific to the entity type (e.g. `variables` and `shouldCompile` for `prompt`).-- <Tabs>- <Tab title="Python">- ```python- import openlit-- # Retrieve contexts- result = openlit.evaluate_rule(- entity_type="context",- fields={"model": "gpt-4", "user_tier": "premium"},- include_entity_data=True,- )-- # Retrieve compiled prompts- result = openlit.evaluate_rule(- entity_type="prompt",- fields={"model": "gpt-4", "user_tier": "premium"},- include_entity_data=True,- entity_inputs={- "variables": {"user_name": "Alice", "product": "OpenLIT"},- "shouldCompile": True,- },- )- ```- </Tab>- <Tab title="TypeScript">- ```typescript- import Openlit from 'openlit';-- // Retrieve contexts- const result = await Openlit.evaluateRule({- entityType: 'context',- fields: { model: 'gpt-4', user_tier: 'premium' },- includeEntityData: true,- });-- // Retrieve compiled prompts- const promptResult = await Openlit.evaluateRule({- entityType: 'prompt',- fields: { model: 'gpt-4', user_tier: 'premium' },- includeEntityData: true,- entityInputs: {- variables: { user_name: 'Alice', product: 'OpenLIT' },- shouldCompile: true,- },- });- ```- </Tab>- <Tab title="Go">- ```go- // Retrieve contexts- result, err := openlit.EvaluateRule(ctx, openlit.EvaluateRuleOptions{- EntityType: openlit.RuleEntityContext,- Fields: map[string]interface{}{"model": "gpt-4", "user_tier": "premium"},- IncludeEntityData: true,- })-- // Retrieve compiled prompts- result, err = openlit.EvaluateRule(ctx, openlit.EvaluateRuleOptions{- EntityType: openlit.RuleEntityPrompt,- Fields: map[string]interface{}{"model": "gpt-4", "user_tier": "premium"},- IncludeEntityData: true,- EntityInputs: map[string]interface{}{- "variables": map[string]string{"user_name": "Alice", "product": "OpenLIT"},- "shouldCompile": true,- },- })- ```- </Tab>- <Tab title="curl">- ```bash- # Retrieve contexts- curl -X POST https://your-openlit-instance/api/rule-engine/evaluate \- -H "Authorization: Bearer YOUR_API_KEY" \- -H "Content-Type: application/json" \- -d '{- "entity_type": "context",- "fields": {- "model": "gpt-4",- "user_tier": "premium"- },- "include_entity_data": true- }'-- # Retrieve compiled prompts- curl -X POST https://your-openlit-instance/api/rule-engine/evaluate \- -H "Authorization: Bearer YOUR_API_KEY" \- -H "Content-Type: application/json" \- -d '{- "entity_type": "prompt",- "fields": {- "model": "gpt-4",- "user_tier": "premium"- },- "include_entity_data": true,- "entity_inputs": {- "variables": { "user_name": "Alice", "product": "OpenLIT" },- "shouldCompile": true- }- }'- ```- </Tab>- </Tabs>-- ```json Example Response (Context)- {- "matchingRuleIds": ["rule-uuid-1"],- "entities": [- { "rule_id": "rule-uuid-1", "entity_type": "context", "entity_id": "ctx-uuid-1" }- ],- "entity_data": {- "context:ctx-uuid-1": {- "id": "ctx-uuid-1",- "name": "Premium System Prompt",- "content": "You are a helpful AI assistant...",- "status": "ACTIVE"- }- }- }- ```-- ```json Example Response (Prompt)- {- "matchingRuleIds": ["rule-uuid-1"],- "entities": [- { "rule_id": "rule-uuid-1", "entity_type": "prompt", "entity_id": "prompt-uuid-1" }- ],- "entity_data": {- "prompt:prompt-uuid-1": {- "promptId": "prompt-uuid-1",- "name": "Onboarding Prompt",- "prompt": "Hello {{user_name}}, welcome to {{product}}!",- "compiledPrompt": "Hello Alice, welcome to OpenLIT!",- "version": "1.0.0",- "tags": ["onboarding"],- "metaProperties": {}- }- }- }- ```-- <Info>For detailed SDK parameters and error handling, see the [SDK Rule Engine feature doc](/latest/sdk/features/rule-engine).</Info>- </Step>- </Steps>- </Step>-</Steps>--## Condition operators reference--### String operators--| Operator | Description | Example |-|----------|-------------|---------|-| `equals` | Exact match | `model equals gpt-4` |-| `not_equals` | Does not match | `model not_equals gpt-3.5` |-| `contains` | Substring match | `model contains gpt` |-| `not_contains` | Substring not present | `model not_contains turbo` |-| `starts_with` | Prefix match | `model starts_with gpt` |-| `ends_with` | Suffix match | `model ends_with 4` |-| `regex` | Regular expression | `model regex ^gpt-[0-9]+$` |-| `in` | Value in comma-separated list | `model in gpt-4,claude-3` |-| `not_in` | Value not in list | `model not_in gpt-3.5,davinci` |--### Number operators--| Operator | Description |-|----------|-------------|-| `equals` | Exact numeric match |-| `not_equals` | Does not match numerically |-| `gt` | Greater than |-| `gte` | Greater than or equal |-| `lt` | Less than |-| `lte` | Less than or equal |-| `between` | Inclusive range — value as `min,max` |--### Boolean operators--| Operator | Description |-|----------|-------------|-| `equals` | Matches `true` or `false` |-------<CardGroup cols={2}>- <Card title="Context" href="/latest/openlit/prompts-experiments/context" icon='file-text'>- Store reusable knowledge and system instructions that can be retrieved when rules match- </Card>- <Card title="Prompt Hub" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking- </Card>- <Card title="SDK Rule Engine" href="/latest/sdk/features/rule-engine" icon='puzzle-piece'>- Use evaluate_rule() from Python, TypeScript, or Go with full parameter reference and examples- </Card>- <Card title="API Reference: Evaluate" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon='code'>- Full reference for the Rule Engine evaluate endpoint with request and response schemas- </Card>-</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine/conditions.mdx b/docs/latest/openlit/prompts-experiments/rule-engine/conditions.mdxnew file mode 100644index 00000000..1f777739--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/rule-engine/conditions.mdx@@ -0,0 +1,99 @@+---+title: "Conditions"+description: "Build condition groups on a rule's detail page - pick fields and operators from dropdowns, and combine groups with AND/OR"+---++<Frame>+ <img src="/images/rule-engine-conditions.png" alt="Adding condition groups to a rule" />+</Frame>++A rule's matching logic lives in its **Condition Groups** section, on the rule's detail page. Conditions are organized into groups: each group has its own AND/OR operator for the conditions inside it, and the rule's top-level **Group Operator** (set when you created the rule, editable from **Edit Details**) decides how the groups themselves are combined.++## Adding groups and conditions++- On a rule with no conditions yet, you'll see "No condition groups yet." and an **Add First Group** button.+- On a rule that already has groups, click **Add Group** to add another one.+- Inside a group, click **Add Condition** to add a condition. Each group shows its own AND/OR choice, labeled "within group", for combining the conditions inside it.+- When you're done, click **Save Conditions** to persist your changes.++## Fields, operators, and values++Each condition has three parts:++- **Field** - a dropdown. You choose from a fixed list of fields (there's no way to type an arbitrary custom field name).+- **Operator** - a dropdown whose options depend on the field's data type (string or number - see below).+- **Value** - a text input. For fields that commonly take one of a known set of real-world values, you'll see suggestions labeled "Top 100 values from traces. Press Enter to add a custom value."++### Available fields++All fields come from your trace/telemetry data. The number fields are marked below; everything else is a string field.++| Field | Type |+|---|---|+| Service Name | string |+| Span Name | string |+| Span Kind | string |+| Duration (ms) | number |+| Status Code | string |+| Deployment Env | string |+| Gen AI System | string |+| Model | string |+| Input Tokens | number |+| Output Tokens | number |+| Total Cost ($) | number |+| Temperature | number |++For example, **Model** is the model name reported by your LLM calls, and **Total Cost ($)** is the cost recorded for a request.++### String field operators++| Operator | Description |+|---|---|+| `equals` | Exact match |+| `not_equals` | Does not match |+| `contains` | Substring match |+| `not_contains` | Substring not present |+| `starts_with` | Prefix match |+| `ends_with` | Suffix match |+| `regex` | Regular expression match |+| `in` | Value is one of a comma-separated list |+| `not_in` | Value is not one of a comma-separated list |++### Number field operators++| Operator | Description |+|---|---|+| `equals` | Exact numeric match |+| `not_equals` | Does not match numerically |+| `gt` | Greater than |+| `gte` | Greater than or equal to |+| `lt` | Less than |+| `lte` | Less than or equal to |+| `between` | Inclusive range, entered as `min,max` |++<Note>+There's currently no boolean-type field in the condition builder, so boolean operators aren't reachable from the UI today. The underlying evaluation logic does support an `equals` check for boolean values at the API level, but that's a forward-looking detail rather than something you can build visually yet.+</Note>++## Worked example++Say you want a rule to match requests to a specific, more expensive model where the cost is also high - a candidate for a cost alert:++1. Click **Add First Group** (or **Add Group** if the rule already has one).+2. Leave the group's "within group" operator on AND.+3. Click **Add Condition**, choose Field = **Model**, Operator = **equals**, Value = the model name (e.g. `gpt-4`).+4. Click **Add Condition** again, choose Field = **Total Cost ($)**, Operator = **gt**, Value = `1000`.+5. Click **Save Conditions**.++This group now matches any request where Model equals your chosen model **AND** Total Cost is greater than 1000. If you add a second group, the rule's top-level Group Operator decides whether both groups must match (AND) or either one is enough (OR).++---++<CardGroup cols={2}>+ <Card title="Preview" href="/latest/openlit/prompts-experiments/rule-engine/preview" icon='flask-vial'>+ Test your saved conditions against recent traces before relying on them+ </Card>+ <Card title="Linked Entities" href="/latest/openlit/prompts-experiments/rule-engine/linked-entities" icon='link'>+ Connect the rule to a Context, Prompt, or Evaluation Type+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine/create-and-manage-rules.mdx b/docs/latest/openlit/prompts-experiments/rule-engine/create-and-manage-rules.mdxnew file mode 100644index 00000000..ade26189--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/rule-engine/create-and-manage-rules.mdx@@ -0,0 +1,51 @@+---+title: "Create & Manage Rules"+description: "Create a rule from the Rule Engine list page, edit its details, and delete rules you no longer need"+---++<Frame>+ <img src="/images/rule-engine-list.png" alt="List of rules with name, description, group operator, status, and created by columns" />+</Frame>++The **Rule Engine** list page (`/rule-engine`) shows every rule you've created, with columns for Name, Description, Group Operator (badge), Status (Active/Inactive badge), Created By, Created At, and Actions. Clicking anywhere on a row opens that rule's detail page.++## Create a rule++Once your database connection is healthy, a **Create Rule** button appears on the list page. Clicking it opens a dialog - there's no separate page for creating a rule.++The dialog title is **"Create a new rule"** when opened from the list page. If you instead start from a Context or Prompt's own page, the same dialog opens titled **"Create rule and link to context"** or **"Create rule and link to prompt"**, and pre-fills the link to that entity for you.++The dialog has these fields:++- **Name** - required, e.g. "High cost alert".+- **Description** - optional, describing what the rule does.+- **Group Operator** - required. Choose "AND - all groups match" or "OR - any group matches" (default AND). This sets how the rule's condition groups are combined once you add them - see [Conditions](/latest/openlit/prompts-experiments/rule-engine/conditions).+- **Status** - required. Choose Active or Inactive (default Active). Only Active rules are ever evaluated.++Click **Create rule** to save. The rule is created without any conditions or linked entities yet - you add those from its detail page.++## The rule detail page++Opening a rule (`/rule-engine/[id]`) shows a **Back** button to return to the list, plus:++- A **rule info** card and the condition builder on the left.+- A **Rule Preview** card and an **Associated Entities** card, stacked on the right.++The rule info card shows the Name, Status badge, Group Operator badge, and "Created by {name} · {date}", plus the Description if one was set. Click the **Edit Details** (pencil) button to switch into edit mode for Name, Description, Group Operator, and Status, with **Cancel** and **Save** buttons.++See [Conditions](/latest/openlit/prompts-experiments/rule-engine/conditions) for how to build the matching logic, [Preview](/latest/openlit/prompts-experiments/rule-engine/preview) for testing it against recent traces, and [Linked Entities](/latest/openlit/prompts-experiments/rule-engine/linked-entities) for connecting the rule to a Context, Prompt, or Evaluation Type.++## Delete a rule++From the list page, click the delete (trash) icon in a rule's Actions column. You'll be asked to confirm:++> Are you sure you want to delete this rule?+> Deleting this rule will also remove all its conditions and entity associations.++Deleting a rule removes it entirely, along with its condition groups and any entity links - this can't be undone.++---++<Card title="Conditions" href="/latest/openlit/prompts-experiments/rule-engine/conditions" icon='filter'>+ Build condition groups, pick fields and operators, and combine groups with AND/OR+</Card>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine/evaluate-api.mdx b/docs/latest/openlit/prompts-experiments/rule-engine/evaluate-api.mdxnew file mode 100644index 00000000..3f2ac7e6--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/rule-engine/evaluate-api.mdx@@ -0,0 +1,48 @@+---+title: "Evaluate API"+description: "Call the Rule Engine evaluate endpoint from any application with a Bearer API key to retrieve resources linked to matching rules"+---++Your application uses the evaluate endpoint to actually use a rule at runtime - this is separate from the [Preview](/latest/openlit/prompts-experiments/rule-engine/preview) card, which only tests a rule while you're building it.++## Authentication++The evaluate endpoint accepts a Bearer API-key token, so any external application can call it - no dashboard login needed.++## Request fields++**Required:**++- `entity_type` - one of `context`, `prompt`, or `evaluation`.+- `fields` - a flat key-value map of the current input values to check against your rules' conditions (e.g. `{"model": "gpt-4", "user_tier": "premium"}`). Up to 50 fields.++**Optional:**++- `include_entity_data` - set to `true` to get full entity records back in the response, not just IDs.+- `entity_inputs` - extra, per-entity-type options. For example, when retrieving a `prompt`, you can pass `variables` and `shouldCompile` to get the prompt back with its `{{variableName}}` placeholders filled in.++Only rules with Status = Active are ever evaluated - Inactive rules are skipped entirely.++## Response shape++- `matchingRuleIds` - the IDs of the rules that matched.+- `entities` - references to each matched rule's linked entities.+- `entity_data` - present only if `include_entity_data` was set, keyed by `"{entity_type}:{entity_id}"` with the full record. Full entity data is currently only populated for `context` and `prompt` entity types, not for `evaluation`.++## Create an API key++- Navigate to **Settings → API Keys** in OpenLIT.+- Click **Create API Key**, enter a name, and save the key securely.++## Call the evaluate endpoint++Send a `POST` to `/api/rule-engine/evaluate`. The full code examples and request/response schema live in two places, depending on what you need:++<CardGroup cols={2}>+ <Card title="SDK Rule Engine" href="/latest/sdk/features/rule-engine" icon='puzzle-piece'>+ Python, TypeScript, and Go examples for retrieving contexts, prompts, and evaluation configs, plus full parameter tables and error handling+ </Card>+ <Card title="API Reference: Evaluate" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon='code'>+ Full request/response schema and a curl example, with a live "try it" playground+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine/linked-entities.mdx b/docs/latest/openlit/prompts-experiments/rule-engine/linked-entities.mdxnew file mode 100644index 00000000..fde42c48--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/rule-engine/linked-entities.mdx@@ -0,0 +1,55 @@+---+title: "Linked Entities"+description: "Connect a rule to a Context, Prompt, or Evaluation Type so matching returns real resources"+---++A rule only becomes useful once it's connected to something worth returning. The **Associated Entities** card on a rule's detail page lets you link the rule to resources that should come back when it matches.++## Entity types++The entity types you can link today are:++- **Context** - reusable knowledge or system instructions, from the [Context](/latest/openlit/prompts-experiments/context/overview) feature.+- **Prompt** - a versioned prompt, from [Prompt Hub](/latest/openlit/prompts-experiments/prompt-hub/overview).+- **Evaluation** - an Evaluation Type, from the Evaluations feature.++## Linking an entity++1. In the **Associated Entities** card, choose an **Entity Type**.+2. Pick from a dropdown of existing entities of that type - Context, Prompt, and Evaluation each have their own picker, populated from what already exists.+3. Click **Associate**.++If the list is empty, you'll see "No entities associated yet."++## Removing a link++Click the remove (X) icon next to a linked entity. You'll be asked to confirm:++> Remove entity association?+> This will remove the link between the rule and this entity.++This only removes the link between the rule and the entity - it doesn't delete the Context, Prompt, or Evaluation Type itself.++## What happens when a rule matches++When a rule matches at evaluation time, all of its linked entities are returned. If the caller asks for it, their full data is included too - for example, a Context's content, or a Prompt's compiled text. See the [Evaluate API](/latest/openlit/prompts-experiments/rule-engine/evaluate-api) page for the exact response shape.++## Linking from the other side++You can also start the link from a Context or Prompt's own page instead of from the rule:++- From a Context, see [Linking rules to a Context](/latest/openlit/prompts-experiments/context/link-rules).+- From a Prompt, see [Linking rules to a Prompt](/latest/openlit/prompts-experiments/prompt-hub/rules).++Both flows open the same "Create rule and link to..." dialog described in [Create & Manage Rules](/latest/openlit/prompts-experiments/rule-engine/create-and-manage-rules), pre-filled with the link back to that Context or Prompt.++---++<CardGroup cols={2}>+ <Card title="Preview" href="/latest/openlit/prompts-experiments/rule-engine/preview" icon='flask-vial'>+ Confirm your conditions match before relying on the linked entities+ </Card>+ <Card title="Evaluate API" href="/latest/openlit/prompts-experiments/rule-engine/evaluate-api" icon='code'>+ Retrieve linked entities - and their full data - from your application+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine/overview.mdx b/docs/latest/openlit/prompts-experiments/rule-engine/overview.mdxnew file mode 100644index 00000000..46f24125--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/rule-engine/overview.mdx@@ -0,0 +1,58 @@+---+title: "Overview"+description: "Rule Engine is a rules engine for conditional AI resource retrieval - define matching conditions once, and let your application or OpenLIT ask what applies right now"+---++Rule Engine lets you define matching conditions once, instead of hardcoding lookup logic like "if model is X, use prompt Y" throughout your application. A rule pairs a set of conditions on runtime fields - like the model being called, the deployment environment, or the cost of a request - with the resources that should come back when those conditions are met. Your application calls a single API to ask "what applies right now?", and OpenLIT itself uses the same mechanism when retrieving a [Context](/latest/openlit/prompts-experiments/context/overview) or [Prompt](/latest/openlit/prompts-experiments/prompt-hub/overview) that's linked to a matching rule.++<CardGroup cols={2}>+ <Card title="Create & Manage Rules" href="/latest/openlit/prompts-experiments/rule-engine/create-and-manage-rules" icon='list-check'>+ Create a rule from the dialog, edit its details, and delete rules you no longer need+ </Card>+ <Card title="Conditions" href="/latest/openlit/prompts-experiments/rule-engine/conditions" icon='filter'>+ Build condition groups, pick fields and operators, and combine groups with AND/OR+ </Card>+ <Card title="Preview" href="/latest/openlit/prompts-experiments/rule-engine/preview" icon='flask-vial'>+ Sanity-check a rule's saved conditions against your last 100 traces before relying on it+ </Card>+ <Card title="Linked Entities" href="/latest/openlit/prompts-experiments/rule-engine/linked-entities" icon='link'>+ Connect a rule to a Context, Prompt, or Evaluation Type so a match returns real resources+ </Card>+ <Card title="Evaluate API" href="/latest/openlit/prompts-experiments/rule-engine/evaluate-api" icon='code'>+ Call the evaluate endpoint from any application with a Bearer API key+ </Card>+</CardGroup>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is the Rule Engine used for?">+ Rule Engine matches runtime inputs - like model name, deployment environment, or cost - against conditions you define, then returns the Context, Prompt, or Evaluation Type linked to any rule that matches. This lets your application (or OpenLIT itself) look up the right resource for the current situation instead of hardcoding that logic in code.+ </Accordion>+ <Accordion title="What kinds of resources can a rule return?">+ A rule can be linked to a Context, a Prompt, or an Evaluation Type. When the rule matches, all of its linked entities are returned, and their full data can be included in the response if you ask for it.+ </Accordion>+ <Accordion title="Can I call the Rule Engine from outside OpenLIT?">+ Yes. The evaluate API accepts a Bearer API-key token, so any application can evaluate rules and retrieve linked entities without a dashboard session. See the [Evaluate API](/latest/openlit/prompts-experiments/rule-engine/evaluate-api) page.+ </Accordion>+ <Accordion title="How are multiple conditions combined?">+ Conditions are organized into groups, each with its own AND/OR logic. The groups themselves are then combined using the rule's top-level Group Operator, so you can express nested matching logic without writing any code. See [Conditions](/latest/openlit/prompts-experiments/rule-engine/conditions) for details.+ </Accordion>+</AccordionGroup>++---++<CardGroup cols={2}>+ <Card title="Context" href="/latest/openlit/prompts-experiments/context/overview" icon='file-lines'>+ Store reusable knowledge and system instructions that can be retrieved when rules match+ </Card>+ <Card title="Prompt Hub" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>+ Version, deploy, and collaborate on prompts with centralized management and tracking+ </Card>+ <Card title="SDK Rule Engine" href="/latest/sdk/features/rule-engine" icon='puzzle-piece'>+ Use evaluate_rule() from Python, TypeScript, or Go with full parameter reference and examples+ </Card>+ <Card title="API Reference: Evaluate" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon='code'>+ Full reference for the Rule Engine evaluate endpoint with request and response schemas+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/prompts-experiments/rule-engine/preview.mdx b/docs/latest/openlit/prompts-experiments/rule-engine/preview.mdxnew file mode 100644index 00000000..0ed8f581--- /dev/null+++ b/docs/latest/openlit/prompts-experiments/rule-engine/preview.mdx@@ -0,0 +1,32 @@+---+title: "Preview"+description: "Use the Rule Preview card on a rule's detail page to test its saved conditions against recent traces before relying on it"+---++Before you wire a rule into your application, you can sanity-check it against real, recent activity. The **Rule Preview** card sits on a rule's detail page, next to the **Associated Entities** card.++## Running a preview++Click **Run Preview** to test the rule's current *saved* conditions - so save your conditions first, or the preview will run against whatever was last saved rather than your unsaved edits. The button shows "Running…" while it works.++A preview checks your last 100 traces and shows up to the top 5 that matched, each with its service name, span name, trace ID, and a **Matched** / **No Match** badge.++## States you'll see++- Before running it for the first time: "Click 'Run Preview' to test your saved rule conditions against recent traces."+- If none of the last 100 traces matched: "No matching traces found in the last 100 records."++## What preview is - and isn't - for++Preview is a sanity check for building and tuning a rule. It isn't connected to your live application traffic, and running it doesn't affect anything downstream. When your application actually needs to use a rule, it calls the separate [evaluate API](/latest/openlit/prompts-experiments/rule-engine/evaluate-api) - preview is only there to help you confirm a rule's conditions look right before you rely on that API.++---++<CardGroup cols={2}>+ <Card title="Conditions" href="/latest/openlit/prompts-experiments/rule-engine/conditions" icon='filter'>+ Build the condition groups you're previewing+ </Card>+ <Card title="Evaluate API" href="/latest/openlit/prompts-experiments/rule-engine/evaluate-api" icon='code'>+ Use the rule from your application, once you're happy with it+ </Card>+</CardGroup>diff --git a/docs/latest/openlit/quickstart-ai-observability.mdx b/docs/latest/openlit/quickstart-ai-observability.mdxindex e7a40c26..7047a8de 100644--- a/docs/latest/openlit/quickstart-ai-observability.mdx+++ b/docs/latest/openlit/quickstart-ai-observability.mdx@@ -1,6 +1,6 @@---title: "Get started with AI Observability"-sidebarTitle: "LLM Observability"+sidebarTitle: "Monitor AI Agents"description: "Start monitoring your AI applications with the OpenLIT SDK in a few steps"---diff --git a/docs/latest/openlit/quickstart-evals.mdx b/docs/latest/openlit/quickstart-evals.mdxindex c62a4526..2ec0ef2d 100644--- a/docs/latest/openlit/quickstart-evals.mdx+++ b/docs/latest/openlit/quickstart-evals.mdx@@ -1,22 +1,131 @@---title: "Evaluations"-description: "Use LLMs to evaluate AI application quality, safety, and performance with automated scoring and detailed analysis"+sidebarTitle: "Set Up Evals"+description: "Score AI outputs automatically for hallucination, bias, toxicity, and more - or add your own human feedback"----import LLMJudge from '/snippets/llm-as-a-judge.mdx';+Automated AI evaluation to assess and monitor the quality, safety, and performance of your LLM outputs across development and production environments. Open **Monitor → Evaluations** for Analytics, Evaluators, and Configuration. For evaluation types and custom evaluators, see [Evaluators](/latest/openlit/evaluations/evaluators).-<LLMJudge />+<video+ autoPlay+ muted+ loop+ controls+ className="w-full aspect-video rounded-xl"+ src="/images/evaluations.mp4"+ alt="Demo of OpenLIT's automated evaluation scoring"+ >+ </video>++## Find the right feature++| If you want to... | Use this OpenLIT feature |+|---|---|+| Automatically score every production trace | [Configuration](/latest/openlit/evaluations/configuration) (Auto Evaluation) |+| Score one specific trace on demand | [LLM-as-a-Judge](/latest/openlit/evaluations/llm-as-a-judge) from that trace's Evaluation tab |+| Rate a trace yourself instead of an LLM judge | [Manual Feedback](/latest/openlit/evaluations/manual-feedback) - Good / Bad / Neutral plus a comment |+| Judge responses against your own ground truth, not just the model's knowledge | [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) + [Context](/latest/openlit/prompts-experiments/context/overview) |+| Evaluate on criteria beyond the 11 built-in types | [Custom evaluators](/latest/openlit/evaluations/evaluators#custom-evaluators) |+| Only run certain evaluators for certain models, providers, or environments | [Evaluators](/latest/openlit/evaluations/evaluators) + [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) |+| Test prompt or model changes before shipping | [Programmatic evaluations](/latest/sdk/quickstart-programmatic-evals) via the SDK |+| Block a deploy on a quality regression | [Programmatic evaluations](/latest/sdk/quickstart-programmatic-evals) in a CI/CD pipeline |+| See pass-rate trends across evaluators and time | [Analytics](/latest/openlit/evaluations/analytics) |++## Setup++<Tabs>+ <Tab title="Online">+ ```mermaid+ flowchart LR;+ A[LLM / Agent call] -->|creates| B[Trace in OpenLIT];+ B -->|Auto Eval schedule or manual Run Evaluation| C[LLM Judge];+ C -->|score + classification + reasoning| D[Trace detail & Dashboard];+ ```++ <Steps>+ <Step title="Pick a judge model">+ Go to **Evaluations → Settings**, choose a provider and model to act as the judge (OpenAI, Anthropic, Google, Mistral, and 7+ others), and add its API key from [Vault](/latest/openlit/developer-resources/vault/overview).+ </Step>+ <Step title="Enable evaluators">+ Switch to the **Evaluation Types** tab and turn on the evaluators you want. Hallucination, Bias, and Toxicity are enabled by default; Relevance, Coherence, Safety, and 5 others are opt-in.+ </Step>+ <Step title="Turn on Auto Evaluation">+ Back in Settings, enable **Auto Evaluation** with a cron schedule so every new trace gets scored automatically - or skip this and click **Run Evaluation** from any trace's Evaluation tab to score it on demand.+ </Step>+ <Step title="Review results">+ Open any trace's **Evaluation** tab for its score, classification, and reasoning, or check the **Evaluations** dashboard for aggregate trends across models and time.+ </Step>+ </Steps>+ </Tab>+ <Tab title="Offline">+ ```mermaid+ flowchart LR;+ E[Dataset] -->|SDK evaluate call| F[LLM Judge];+ F -->|score| G[Pass / fail gate];+ ```++ <Steps>+ <Step title="Get an API key">+ In OpenLIT, go to **Settings → API Keys** and create a key. Offline evaluations call your OpenLIT server, so you'll need this key plus your OpenLIT URL.+ </Step>+ <Step title="Run an evaluation">+ <Tabs>+ <Tab title="Python">+ ```python+ import openlit++ openlit.init(+ openlit_url="http://localhost:3000",+ openlit_api_key="openlit-xxxxx",+ )++ result = openlit.eval(+ prompt="What is the capital of France?",+ response="The capital of France is Lyon.",+ contexts=["Paris is the capital and largest city of France."],+ )++ assert result.passed, f"Evaluation failed: {result.failed_evals}"+ ```+ </Tab>+ <Tab title="Typescript">+ ```typescript+ import openlit, { isPassed, getFailedEvals } from "openlit"++ openlit.init({+ openlitUrl: "http://localhost:3000",+ openlitApiKey: "openlit-xxxxx",+ })++ const result = await openlit.eval({+ prompt: "What is the capital of France?",+ response: "The capital of France is Lyon.",+ contexts: ["Paris is the capital and largest city of France."],+ })++ console.log(isPassed(result), getFailedEvals(result))+ ```+ </Tab>+ </Tabs>+ This uses the exact same evaluators, custom types, and Rule Engine context matching configured in your dashboard - see the [Programmatic evaluations quickstart](/latest/sdk/quickstart-programmatic-evals) for batch runs and CI/CD gating.+ </Step>+ </Steps>+ </Tab>+</Tabs>----<CardGroup cols={3}>- <Card title="Quickstart: LLM Guardrails" href="/latest/openlit/quickstart-guard" icon='bolt'>- Protect and secure your LLM responses in 2 simple steps+<CardGroup cols={2}>+ <Card title="Evaluators" href="/latest/openlit/evaluations/evaluators" icon='list-check'>+ Built-in and custom evaluation types+ </Card>+ <Card title="Configuration" href="/latest/openlit/evaluations/configuration" icon='gear'>+ Auto Evaluation schedule and judge model</Card>- <Card title="Integrations" href="/latest/sdk/integrations/overview" icon='circle-nodes'>- 60+ AI integrations with automatic instrumentation and performance tracking+ <Card title="LLM-as-a-Judge" href="/latest/openlit/evaluations/llm-as-a-judge" icon='gavel'>+ Use advanced LLMs to evaluate AI application quality with automated scoring</Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring+ <Card title="Programmatic evaluations" href="/latest/sdk/quickstart-programmatic-evals" icon='bolt'>+ Quick start guide for implementing custom evaluations in your code</Card></CardGroup>\ No newline at end of filediff --git a/docs/latest/openlit/quickstart-gpu.mdx b/docs/latest/openlit/quickstart-gpu.mdxindex e6f06fdc..c8e4e790 100644--- a/docs/latest/openlit/quickstart-gpu.mdx+++ b/docs/latest/openlit/quickstart-gpu.mdx@@ -17,7 +17,7 @@ import GPUQuickstart from '/snippets/quickstart-gpu.mdx';<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>Create custom visualizations with flexible widgets, queries, and real-time AI monitoring</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card></CardGroup>\ No newline at end of filediff --git a/docs/latest/openlit/quickstart-guard.mdx b/docs/latest/openlit/quickstart-guard.mdxindex cc2e76af..431748c3 100644--- a/docs/latest/openlit/quickstart-guard.mdx+++ b/docs/latest/openlit/quickstart-guard.mdx@@ -17,7 +17,7 @@ import GuardQuickstart from '/snippets/quickstart-guard.mdx';<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>Create custom visualizations with flexible widgets, queries, and real-time AI monitoring</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card></CardGroup>\ No newline at end of filediff --git a/docs/latest/openlit/quickstart-mcp-observability.mdx b/docs/latest/openlit/quickstart-mcp-observability.mdxindex b6df6166..aed4c20c 100644--- a/docs/latest/openlit/quickstart-mcp-observability.mdx+++ b/docs/latest/openlit/quickstart-mcp-observability.mdx@@ -21,14 +21,7 @@ OpenLIT automatically instruments MCP alongside LLMs, VectorDBs, and frameworks<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>Create custom visualizations with flexible widgets, queries, and real-time AI monitoring</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/openlit/quickstart-vectordb-observability.mdx b/docs/latest/openlit/quickstart-vectordb-observability.mdxindex 28eb56df..d1a06de2 100644--- a/docs/latest/openlit/quickstart-vectordb-observability.mdx+++ b/docs/latest/openlit/quickstart-vectordb-observability.mdx@@ -21,14 +21,7 @@ OpenLIT automatically instruments VectorDBs alongside LLMs, MCP, and frameworks<Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>Create custom visualizations with flexible widgets, queries, and real-time AI monitoring</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>+ <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub/overview" icon='message'>Version, deploy, and collaborate on prompts with centralized management and tracking</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/overview.mdx b/docs/latest/overview.mdxindex b4b9b4b9..8a6296bb 100644--- a/docs/latest/overview.mdx+++ b/docs/latest/overview.mdx@@ -1,109 +1,85 @@----title: OpenLIT+title: "OpenLIT Overview"sidebarTitle: Overview-description: 'Open-source AI engineering platform'+description: "OpenLIT is an open-source AI Engineering platform for LLM observability, evaluations, prompt management, and cost tracking - built on OpenTelemetry, fully self-hostable."-----<video- controls- className="w-full aspect-video"- src="https://openlit.io/static/images/demo.mp4"-></video>--OpenLIT is an open-source AI engineering platform that helps teams build, evaluate and observe AI applications across the entire lifecycle from development to production.--OpenLIT provides the following OSS tools to support this goal:--- [OpenLIT](/latest/openlit/installation) - Open-source platform for tracing, prompt management, evaluations, and scalable AI observability with dashboards, metrics, logs, and remote collectors.-- [OpenLIT SDKs](/latest/sdk/overview) - OpenTelemetry-native auto-instrumentation to trace LLMs, agents, vector databases and GPUs with zero-code.-- [OpenLIT Controller](/latest/controller/overview) - Zero-code LLM and Agent observability for Kubernetes, Docker, and Linux using eBPF and automatic SDK injection.--The project proudly follows OpenTelemetry Semantic Conventions for AI observability standards.--## Features-+Understand what your AI agents are doing, measure their performance, and improve them in production.+With OpenLIT teams can Trace LLM and agent calls, compare models, iterate on prompts, detect regressions, and optimize cost and performance using real production data.<Tabs>- <Tab title="Tracing" icon="bars-staggered">+ <Tab title="Develop" icon="code-branch"><videoautoPlaymutedloopcontrolsclassName="w-full aspect-video rounded-xl"- src="/images/trace-details.mp4"+ src="/images/prompts.mp4"+ alt="Demo of managing prompts in Prompt Hub"></video>- OpenLIT provides distributed tracing capabilities for understanding and debugging AI applications:- - [OpenTelemetry-native SDKs](/latest/sdk/overview) - Automatic instrumentation for LLMs, agents, frameworks, Vector databases, MCP and GPUs.- - [Exceptions Monitoring](/latest/openlit/observability/error) - Track and debug application errors with detailed stack traces.- - **Universal Compatibility** - View traces from any OpenTelemetry-instrumented tool or LLM instrumentation frameworks like OpenInference, OpenLLMetry.--+ Prompt management, prompt versioning, and model testing tools for teams building AI applications:+ - **Prompt versioning and deployment** - [Prompt Hub](/latest/openlit/prompts-experiments/prompt-hub/overview) treats prompts as versioned artifacts. Edit, version, and deploy to any environment without code changes. Roll back instantly when needed.+ - **Collaborate on prompts** - Edit prompts as a team with full change history. Every version is tracked and linked to the evaluations and traces it produced, giving you full traceability from prompt to output.+ - **Model comparison and A/B testing** - [OpenGround](/latest/openlit/prompts-experiments/openground/overview) runs side-by-side LLM prompt testing across models, comparing cost, latency, and output quality so you pick the right model before you ship.</Tab>- <Tab title="Evaluations" icon="ruler">+ <Tab title="Observe" icon="bars-staggered"><videoautoPlaymutedloopcontrolsclassName="w-full aspect-video rounded-xl"- src="/images/evaluations.mp4"+ src="/images/trace-details.mp4"+ alt="Demo of viewing trace details in OpenLIT"></video>- OpenLIT provides automated LLM evaluation and testing capabilities for production AI applications:- - [Online LLM-as-a-jduge](/latest/openlit/evaluations/llm-as-a-judge) - Zero-setup AI quality monitoring that automatically evaluates your LLM responses from traces.- - [Programmatic Evaluations](/latest/openlit/evaluations/programmatic-evals) - Programmatic AI evaluation tools for custom testing workflows and development pipelines.-+ LLM observability and AI agent observability for agents built with CrewAI, LangGraph, or any agent framework. OpenLIT brings OpenTelemetry-native instrumentation so you can analyze how your agents perform in production with full stack visibility:+ - **Tracing** - Instrument once via [OpenTelemetry-native SDKs](/latest/sdk/overview) for 90+ LLMs and agent frameworks to produce OpenTelemetry traces.+ - **Lowest level transparency** - [Exceptions Monitoring](/latest/openlit/observability/telemetry#exceptions) surfaces errors with full stack traces and span context so you know exactly where and why something failed. All telemetry is automatically redacted for sensitive data before it leaves your stack.+ - **Understand cost and latency** - Track token consumption, spend per model, provider, and request. Monitor GPU utilization alongside LLM call latency in [custom dashboards](/latest/openlit/dashboards/overview) built on your raw telemetry.</Tab>- <Tab title="Prompts" icon="message">- <video- autoPlay- muted- loop- controls- className="w-full aspect-video rounded-xl"- src="/images/prompts.mp4"- >- </video>- OpenLIT's [Prompt Hub](/latest/openlit/prompts-experiments/prompt-hub) provides capabilities to centrally manage and maintain prompts:- - Version and edit prompts collaboratively- - Deploy prompts to any environment without code changes- - Track prompt evolution and changes over time- </Tab>- <Tab title="Experiments" icon="flask">+ <Tab title="Coding Agents" icon="laptop-code"><videoautoPlaymutedloopcontrolsclassName="w-full aspect-video rounded-xl"- src="/images/experiments.mp4"+ src="/images/trace-details.mp4"+ alt="Demo of coding agent observability for Claude Code, Cursor, and Codex"></video>- OpenLIT provides experimentation tools for comparing and optimizing AI models:- - [OpenGround](/latest/openlit/prompts-experiments/openground) - Compare cost, duration, and response tokens across LLMs to choose the most efficient model for your use case.++ Coding agent observability for Claude Code, Cursor, and Codex. Understand token consumption, AI spend, and developer productivity across your entire engineering team:+ - [One-command setup](/latest/openlit/coding-agents/setup-and-configure) - Install the `openlit` CLI and wire any vendor in under a minute. No SDK, no code changes in your repos.+ - **Session monitoring** - Track cost per session, token usage, lines added and removed, model acceptance rate, commits, and PRs per user and per team.+ - **AI spend visibility** - See exactly how much each developer, team, and project spends on AI coding tools. Break down cost by model and vendor across Claude Code, Cursor, and Codex in one place.+ - **Privacy controls** - Automatic secret redaction, usage breakdowns by repo, and cohort-floor anonymization so individual data stays protected.</Tab>- <Tab title="Dashboards" icon="grid">++ <Tab title="Improve" icon="arrow-trend-up"><videoautoPlaymutedloopcontrolsclassName="w-full aspect-video rounded-xl"- src="/images/dashboards.mp4"+ src="/images/evaluations.mp4"+ alt="Demo of OpenLIT's automated evaluation scoring"></video>- [Dashboards](/latest/openlit/dashboards/overview) in OpenLIT provide powerful visualization capabilities for AI monitoring:- - Create and resize custom widgets with flexible configurations and layouts- - Visualize telemetry from any OpenTelemetry-instrumented tool or LLM instrumentation frameworks like OpenInference, OpenLLMetry- - Write custom SQL queries to analyze your AI telemetry data- - Export and import dashboard configurations as JSON files++ LLM evaluation in production and development using the same evaluators. Close the loop between what you ship and what you measure:+ - **Online LLM evaluation** - [LLM-as-a-judge](/latest/openlit/evaluations/llm-as-a-judge) runs automatically on your live traces in production. Create custom evaluators with your own scoring criteria and quality gates. No setup required.+ - **Offline evaluation and CI/CD testing** - Use the same evaluators via the [programmatic evaluations API](/latest/openlit/evaluations/programmatic-evals) for regression testing in CI pipelines and local development. Catch quality regressions before they reach production.+ - **Traceability** - Every evaluation score is linked back to the exact prompt version, model, and trace that produced it so you know what changed and why quality shifted.</Tab>- <Tab title="Secrets" icon="vault">++ <Tab title="Manage" icon="sliders"><videoautoPlaymuted@@ -111,49 +87,59 @@ The project proudly follows OpenTelemetry Semantic Conventions for AI observabilcontrolsclassName="w-full aspect-video rounded-xl"src="/images/secrets.mp4"+ alt="Demo of managing API keys in Vault"></video>- OpenLIT provides secure credential management for AI applications:- - [Vault](/latest/openlit/developer-resources/vault) - Centrally store LLM API keys that applications can retrieve remotely without restarts for seamless key rotation.- </Tab>-</Tabs>--## Upgrade Information-<Warning>-**Important Upgrade Notice**: When upgrading from OpenLIT versions prior to 1.15.0 (which introduced Fleet Hub), special attention is required for Docker Compose deployments.--If you're running OpenLIT using Docker Compose, you **must** use the `--remove-orphans` flag when upgrading:+ Manage the agents, keys, and pricing behind your AI stack from one place:+ - **Track and manage agents** - See every agent version running in production, monitor health, and manage deployments from the `/agents` page.+ - **Vault** - [Centrally store and rotate](/latest/openlit/developer-resources/vault/overview) LLM API keys that agents retrieve at runtime. No restarts, no code changes, no secrets in your codebase.+ - **AI cost tracking** - [Costs](/latest/openlit/costs/overview) under Monitor combines spend analytics, [Manage models](/latest/openlit/costs/manage-models/overview), and Auto Pricing. OpenLIT ships with pricing for popular models and applies it across traces, dashboards, and cost reports.+ </Tab>-```bash-# Upgrade with orphan removal-docker-compose up -d --remove-orphans-```+ <Tab title="Otter" icon="message-bot">+ <img src="/images/otter.svg" width="60" className="mb-4" alt="Otter, the OpenLIT AI chat copilot" />-This is necessary because the OpenTelemetry Collector has been integrated directly into the OpenLIT container, and the standalone `otel-collector` container is no longer needed. Without `--remove-orphans`, the old collector container will conflict with the new integrated collector on ports 4317 and 4318.-</Warning>+ **Otter** is a built-in AI copilot that works across all of OpenLIT. Query your telemetry, analyze traces, improve prompts, and generate dashboards through natural language:+ - [Natural language queries](/latest/openlit/chat/overview) - Ask questions about traces, costs, and tokens. Otter converts them to SQL and runs them inline.+ - **Trace and span analysis** - Get structured AI analysis across performance, reliability, cost, token efficiency, and execution path.+ - **Prompt improvement** - Ask Otter to review and suggest improvements to any Prompt Hub prompt.+ - **Dashboard generation** - Describe the dashboard you want and Otter generates a complete importable layout with working queries.+ - **Resource management** - Create rules, contexts, vault secrets, and custom models through conversation.+ </Tab>+</Tabs>-## Getting Started-Choose your path to start building better AI applications with OpenLIT:+## Get Started<CardGroup cols={2}>- <Card title="Quickstart: LLM Observability" href="/latest/openlit/quickstart-ai-observability" icon='bolt'>- Production-ready AI monitoring setup in 2 simple steps with zero code changes+ <Card title="Instrument an AI Agent" href="/latest/openlit/quickstart-ai-observability" icon="bolt">+ Production-ready AI Observability in 2 steps with zero code changes</Card>- <Card title="Deploy OpenLIT" href="/latest/openlit/installation" icon='circle-down'>- Deployment options for scalable LLM monitoring infrastructure+ <Card title="Observe coding agents" href="/latest/openlit/coding-agents/setup-and-configure" icon="code">+ Track spend and usage for Claude Code, Cursor, & more across teams</Card>- <Card title="Create a dashboard" href="/latest/openlit/dashboards/overview" icon='grid'>- Create custom visualizations with flexible widgets, queries, and real-time AI monitoring+ <Card title="Deploy OpenLIT" href="/latest/openlit/installation" icon="circle-down">+ Self-host the full platform with Docker Compose or Helm</Card>- <Card title="Manage prompts" href="/latest/openlit/prompts-experiments/prompt-hub" icon='message'>- Version, deploy, and collaborate on prompts with centralized management and tracking+ <Card title="Evaluate LLM responses" href="/latest/openlit/evaluations/llm-as-a-judge" icon="clipboard-check">+ Score live traces automatically with prebuilt LLM-as-a-judge evaluators</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is OpenLIT?">+ OpenLIT is an open-source AI Engineering platform. It traces LLM and agent calls via OpenTelemetry, tracks cost and token usage, runs automated evaluations, and manages prompts - all self-hostable.+ </Accordion>+ <Accordion title="Is OpenLIT open source?">+ Yes. OpenLIT is fully open source and self-hostable via Docker Compose or Helm, so your telemetry and prompts never have to leave your infrastructure.+ </Accordion>+ <Accordion title="Does OpenLIT require code changes to instrument my app?">+ No. OpenLIT's SDK auto-instruments 90+ LLMs, agent frameworks, and vector databases with zero code changes, or you can call `openlit.init()` once for manual instrumentation if you want more control - both produce the same OpenTelemetry traces.+ </Accordion>+ <Accordion title="How is OpenLIT different from other LLM observability tools?">+ OpenLIT combines AI observability, evaluation, cost tracking, and prompt management in one open-source platform, instead of requiring separate tools for tracing, evals, and prompt versioning.+ </Accordion>+</AccordionGroup>+diff --git a/docs/latest/sdk/configuration.mdx b/docs/latest/sdk/configuration.mdxindex 5c1a6a03..ba0040c9 100644--- a/docs/latest/sdk/configuration.mdx+++ b/docs/latest/sdk/configuration.mdx@@ -201,10 +201,3 @@ Enterprise-grade secret management for Node.js AI applications. Configure secureSend elemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/destinations/dash0.mdx b/docs/latest/sdk/destinations/dash0.mdxindex ecdbca6d..267566a4 100644--- a/docs/latest/sdk/destinations/dash0.mdx+++ b/docs/latest/sdk/destinations/dash0.mdx@@ -1,6 +1,6 @@---title: 'Dash0'-description: 'LLM Observability with Dash0 and OpenLIT'+description: 'Send OpenLIT AI observability traces and metrics to Dash0, an OpenTelemetry-native platform, for real-time LLM monitoring'---import Intro from '/snippets/destinations/dash0/intro.mdx';diff --git a/docs/latest/sdk/destinations/datadog.mdx b/docs/latest/sdk/destinations/datadog.mdxindex c0842ae0..60c6990a 100644--- a/docs/latest/sdk/destinations/datadog.mdx+++ b/docs/latest/sdk/destinations/datadog.mdx@@ -1,6 +1,6 @@---title: 'DataDog'-description: 'LLM Observability with DataDog and OpenLIT'+description: 'Export OpenLIT AI observability data to Datadog for unified LLM and infrastructure monitoring in one dashboard'---import Intro from '/snippets/destinations/datadog/intro.mdx';diff --git a/docs/latest/sdk/destinations/dynatrace.mdx b/docs/latest/sdk/destinations/dynatrace.mdxindex afb9cb5e..6d419eb5 100644--- a/docs/latest/sdk/destinations/dynatrace.mdx+++ b/docs/latest/sdk/destinations/dynatrace.mdx@@ -1,6 +1,6 @@---title: 'Dynatrace'-description: 'LLM Observability with Dynatrace and OpenLIT'+description: 'Send OpenLIT AI observability traces and metrics to Dynatrace for enterprise full-stack LLM monitoring'---import Intro from '/snippets/destinations/dynatrace/intro.mdx';diff --git a/docs/latest/sdk/destinations/elastic.mdx b/docs/latest/sdk/destinations/elastic.mdxindex 12a6172e..ac1663d8 100644--- a/docs/latest/sdk/destinations/elastic.mdx+++ b/docs/latest/sdk/destinations/elastic.mdx@@ -1,6 +1,6 @@---title: 'Elastic'-description: 'LLM Observability with Elastic and OpenLIT'+description: 'Send OpenLIT AI observability data to Elastic Observability for search-powered LLM trace and log analysis'---import Intro from '/snippets/destinations/elastic/intro.mdx';diff --git a/docs/latest/sdk/destinations/grafanacloud.mdx b/docs/latest/sdk/destinations/grafanacloud.mdxindex b360ed92..c0ba3555 100644--- a/docs/latest/sdk/destinations/grafanacloud.mdx+++ b/docs/latest/sdk/destinations/grafanacloud.mdx@@ -1,6 +1,6 @@---title: 'Grafana Cloud'-description: 'LLM Observability with Grafana Cloud and OpenLIT'+description: 'Send OpenLIT AI observability traces and metrics to Grafana Cloud to view LLM data alongside your existing Grafana dashboards'---diff --git a/docs/latest/sdk/destinations/highlight.mdx b/docs/latest/sdk/destinations/highlight.mdxindex f3023a4c..48aa9d20 100644--- a/docs/latest/sdk/destinations/highlight.mdx+++ b/docs/latest/sdk/destinations/highlight.mdx@@ -1,6 +1,6 @@---title: 'Highlight.io'-description: 'LLM Observability with Highlight.io and OpenLIT'+description: 'Send OpenLIT AI observability data to Highlight.io, an open-source platform, for LLM monitoring alongside session replay'---import Intro from '/snippets/destinations/highlight/intro.mdx';diff --git a/docs/latest/sdk/destinations/hyperdx.mdx b/docs/latest/sdk/destinations/hyperdx.mdxindex f0f7366e..91f912de 100644--- a/docs/latest/sdk/destinations/hyperdx.mdx+++ b/docs/latest/sdk/destinations/hyperdx.mdx@@ -1,6 +1,6 @@---title: 'HyperDX'-description: 'LLM Observability with HyperDX and OpenLIT'+description: 'Send OpenLIT AI observability data to HyperDX, an open-source ClickHouse-backed platform, for unified LLM traces, logs, and metrics'---import Intro from '/snippets/destinations/hyperdx/intro.mdx';diff --git a/docs/latest/sdk/destinations/langfuse.mdx b/docs/latest/sdk/destinations/langfuse.mdxindex 07ee2b85..7eb20947 100644--- a/docs/latest/sdk/destinations/langfuse.mdx+++ b/docs/latest/sdk/destinations/langfuse.mdx@@ -1,6 +1,6 @@---title: 'Langfuse'-description: 'LLM Observability with Langfuse and OpenLIT'+description: 'Send OpenLIT AI observability traces to Langfuse via OpenTelemetry for LLM-specific analytics and evaluation'---import Intro from '/snippets/destinations/langfuse/intro.mdx';diff --git a/docs/latest/sdk/destinations/middleware.mdx b/docs/latest/sdk/destinations/middleware.mdxindex a5853a47..4f38c981 100644--- a/docs/latest/sdk/destinations/middleware.mdx+++ b/docs/latest/sdk/destinations/middleware.mdx@@ -1,6 +1,6 @@---title: 'Middleware'-description: 'LLM Observability with Middleware and OpenLIT'+description: 'Send OpenLIT AI observability data to Middleware.io for full-stack LLM and infrastructure monitoring'---import Intro from '/snippets/destinations/middleware/intro.mdx';diff --git a/docs/latest/sdk/destinations/murnitur.mdx b/docs/latest/sdk/destinations/murnitur.mdxindex 3c94656e..79bbc0df 100644--- a/docs/latest/sdk/destinations/murnitur.mdx+++ b/docs/latest/sdk/destinations/murnitur.mdx@@ -1,6 +1,6 @@---title: "Murnitur"-description: "LLM Observability with Murnitur and OpenLIT"+description: "Send OpenLIT AI observability traces to Murnitur for LLM-focused monitoring and guardrails"---import Intro from '/snippets/destinations/murnitur/intro.mdx';diff --git a/docs/latest/sdk/destinations/new-relic.mdx b/docs/latest/sdk/destinations/new-relic.mdxindex 90e74083..6fe029d8 100644--- a/docs/latest/sdk/destinations/new-relic.mdx+++ b/docs/latest/sdk/destinations/new-relic.mdx@@ -1,6 +1,6 @@---title: 'New Relic'-description: 'LLM Observability with New Relic and OpenLIT'+description: 'Export OpenLIT AI observability data to New Relic for full-stack LLM monitoring alongside existing APM data'---import Intro from '/snippets/destinations/newrelic/intro.mdx';diff --git a/docs/latest/sdk/destinations/oneuptime.mdx b/docs/latest/sdk/destinations/oneuptime.mdxindex 3ca9b3a6..f2578f70 100644--- a/docs/latest/sdk/destinations/oneuptime.mdx+++ b/docs/latest/sdk/destinations/oneuptime.mdx@@ -1,6 +1,6 @@---title: 'OneUptime'-description: 'LLM Observability with OneUptime and OpenLIT'+description: 'Send OpenLIT AI observability data to OneUptime, an open-source platform, for LLM monitoring with status pages and incidents'---import Intro from '/snippets/destinations/oneuptime/intro.mdx';diff --git a/docs/latest/sdk/destinations/oodle.mdx b/docs/latest/sdk/destinations/oodle.mdxindex 63bffe34..1d08e3bb 100644--- a/docs/latest/sdk/destinations/oodle.mdx+++ b/docs/latest/sdk/destinations/oodle.mdx@@ -1,6 +1,6 @@---title: 'Oodle'-description: 'LLM Observability with Oodle using OpenLIT'+description: 'Send OpenLIT AI observability data to Oodle for high-scale, cost-efficient LLM metrics and log storage'---import Intro from '/snippets/destinations/oodle/intro.mdx';diff --git a/docs/latest/sdk/destinations/openlit.mdx b/docs/latest/sdk/destinations/openlit.mdxindex 4dda2a2d..a182c4ce 100644--- a/docs/latest/sdk/destinations/openlit.mdx+++ b/docs/latest/sdk/destinations/openlit.mdx@@ -1,6 +1,6 @@---title: 'OpenLIT'-description: 'LLM Observability with OpenLIT Platform and OpenLIT'+description: 'Send SDK telemetry to the native OpenLIT platform for AI observability, cost tracking, and evaluation out of the box'---import Intro from '/snippets/destinations/openlit/intro.mdx';diff --git a/docs/latest/sdk/destinations/openobserve.mdx b/docs/latest/sdk/destinations/openobserve.mdxindex af7b41e2..6f1fdc3a 100644--- a/docs/latest/sdk/destinations/openobserve.mdx+++ b/docs/latest/sdk/destinations/openobserve.mdx@@ -1,6 +1,6 @@---title: 'OpenObserve'-description: 'LLM Observability with OpenObserve and OpenLIT'+description: 'Send OpenLIT AI observability data to OpenObserve, an open-source Rust-based platform, for cost-efficient LLM log and trace storage'---import Intro from '/snippets/destinations/openobserve/intro.mdx';diff --git a/docs/latest/sdk/destinations/otelcol.mdx b/docs/latest/sdk/destinations/otelcol.mdxindex d90f0803..e255118d 100644--- a/docs/latest/sdk/destinations/otelcol.mdx+++ b/docs/latest/sdk/destinations/otelcol.mdx@@ -1,6 +1,6 @@---title: 'OpenTelemetry Collector'-description: 'LLM Observability with OpenTelemetry Collector and OpenLIT'+description: 'Route OpenLIT AI observability data through the OpenTelemetry Collector to any vendor-neutral telemetry backend'---import Intro from '/snippets/destinations/otelcol/intro.mdx';diff --git a/docs/latest/sdk/destinations/overview.mdx b/docs/latest/sdk/destinations/overview.mdxindex a19ac81b..3288abc8 100644--- a/docs/latest/sdk/destinations/overview.mdx+++ b/docs/latest/sdk/destinations/overview.mdx@@ -3257,10 +3257,3 @@ The OpenLIT SDK can send AI observability data (traces, metrics, logs) from LLMsDeployment options for scalable LLM monitoring infrastructure</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/destinations/prometheus-jaeger.mdx b/docs/latest/sdk/destinations/prometheus-jaeger.mdxindex f729c64c..9038cfe8 100644--- a/docs/latest/sdk/destinations/prometheus-jaeger.mdx+++ b/docs/latest/sdk/destinations/prometheus-jaeger.mdx@@ -1,6 +1,6 @@---title: 'Prometheus + Jaeger'-description: 'LLM Observability with Prometheus and Jaeger using OpenLIT'+description: 'Send OpenLIT AI observability data to Prometheus for LLM metrics and Jaeger for distributed tracing'---import Intro from '/snippets/destinations/prometheus-jaeger/intro.mdx';diff --git a/docs/latest/sdk/destinations/prometheus-tempo.mdx b/docs/latest/sdk/destinations/prometheus-tempo.mdxindex da24a3ea..901b0c33 100644--- a/docs/latest/sdk/destinations/prometheus-tempo.mdx+++ b/docs/latest/sdk/destinations/prometheus-tempo.mdx@@ -1,6 +1,6 @@---title: 'Prometheus + Tempo'-description: 'LLM Observability with Prometheus and Grafana Tempo using OpenLIT'+description: 'Send OpenLIT AI observability data to Prometheus for LLM metrics and Grafana Tempo for distributed tracing'---import Intro from '/snippets/destinations/prometheus-tempo/intro.mdx';diff --git a/docs/latest/sdk/destinations/siglens.mdx b/docs/latest/sdk/destinations/siglens.mdxindex a79867a8..ec475e73 100644--- a/docs/latest/sdk/destinations/siglens.mdx+++ b/docs/latest/sdk/destinations/siglens.mdx@@ -1,6 +1,6 @@---title: 'SigLens'-description: 'LLM Observability with SigLens and OpenLIT'+description: 'Send OpenLIT AI observability data to SigLens, an open-source platform, for fast, cost-efficient LLM log search'---import Intro from '/snippets/destinations/siglens/intro.mdx';diff --git a/docs/latest/sdk/destinations/signoz.mdx b/docs/latest/sdk/destinations/signoz.mdxindex 05b04df4..2d20b96b 100644--- a/docs/latest/sdk/destinations/signoz.mdx+++ b/docs/latest/sdk/destinations/signoz.mdx@@ -1,6 +1,6 @@---title: 'SigNoz'-description: 'LLM Observability with SigNoz and OpenLIT'+description: 'Send OpenLIT AI observability data to SigNoz, an open-source OpenTelemetry-native platform, for all-in-one LLM monitoring'---import Intro from '/snippets/destinations/signoz/intro.mdx';diff --git a/docs/latest/sdk/destinations/victoriametrics-stack.mdx b/docs/latest/sdk/destinations/victoriametrics-stack.mdxindex 04742202..6850728c 100644--- a/docs/latest/sdk/destinations/victoriametrics-stack.mdx+++ b/docs/latest/sdk/destinations/victoriametrics-stack.mdx@@ -1,6 +1,6 @@---title: 'Victoria Stack'-description: 'LLM Observability with Victoria Stack using OpenLIT'+description: 'Send OpenLIT AI observability data to the VictoriaMetrics stack for high-performance, cost-efficient LLM metrics and logs'---import Intro from '/snippets/destinations/victoriametrics-stack/intro.mdx';diff --git a/docs/latest/sdk/features/custom-attributes.mdx b/docs/latest/sdk/features/custom-attributes.mdxnew file mode 100644index 00000000..1fd4caaa--- /dev/null+++ b/docs/latest/sdk/features/custom-attributes.mdx@@ -0,0 +1,90 @@+---+title: "Custom Attributes"+description: "Attach custom attributes to spans and metrics beyond OTEL_RESOURCE_ATTRIBUTES - globally, scoped to a block of code, or per agent"+---++[`OTEL_RESOURCE_ATTRIBUTES`](/latest/sdk/features/tracing#add-custom-resource-attributes) adds attributes to every span and metric at the resource level (set once, applies to the whole process). The APIs on this page let you go further - attach custom attributes globally from code, scope them to a specific block or function call, or tag spans with agent identity - without restarting your process or touching environment variables.++## Global custom attributes++Pass `custom_span_attributes` and/or `custom_metrics_attributes` to `openlit.init()` to have every auto-instrumented span or metric include them:++```python+import openlit++openlit.init(+ custom_span_attributes={"team": "platform", "tier": "premium"},+ custom_metrics_attributes={"deployment": "us-east-1"},+)+```++These behave like resource attributes but are set in code rather than via environment variable, and can be computed at startup (e.g. from config you've already loaded).++## Scope attributes to a block of code++`using_attributes()` attaches attributes only to spans created while it's active - as a context manager or as a decorator:++```python+import openlit++# As a context manager+with openlit.using_attributes(user_id="u_123", session_id="s_456"):+ completion = client.chat.completions.create(...)++# As a decorator[email protected]_attributes(feature="checkout")+def handle_checkout(order):+ completion = client.chat.completions.create(...)+```++Attributes set this way apply only to spans created inside the `with` block or during the decorated function call - they don't leak into unrelated spans elsewhere in your application.++## Attach attributes to a single call++`inject_additional_attributes()` runs a callable with attributes attached for that call only, useful when you don't want to wrap a whole block:++```python+import openlit++result = openlit.inject_additional_attributes(+ lambda: client.chat.completions.create(...),+ {"request_source": "batch_job"},+)+```++## Tag spans with agent identity++`agent_context()` and `agent_version_context()` tag spans and metrics with `gen_ai.agent.name` and `gen_ai.agent.version` respectively. These are what power the [Agents page's](/latest/openlit/observability/agents/overview) per-agent grouping and version snapshots - use them if you want explicit control over agent identity instead of relying on inferred grouping from `service.name`.++```python+import openlit++with openlit.agent_context(name="support-agent"):+ with openlit.agent_version_context(version="v3"):+ completion = client.chat.completions.create(...)+```++## Manual spans with custom attributes++For manually created spans (as opposed to attaching attributes to auto-instrumented ones), use `openlit.start_trace()` and set custom metadata directly on the returned span - see [Manual Tracing](/latest/sdk/features/tracing#manual-tracing) for the full `@openlit.trace` / `start_trace()` reference.++```python+with openlit.start_trace(name="checkout-flow") as trace:+ completion = client.chat.completions.create(...)+ trace.set_result(completion.choices[0].message.content)+ trace.set_metadata({"order_id": "o_789", "cart_value": 129.99})+```++---++<CardGroup cols={3}>+ <Card title="Tracing" href="/latest/sdk/features/tracing" icon='route'>+ Custom tracers, resource attributes, and manual tracing+ </Card>+ <Card title="Agents" href="/latest/openlit/observability/agents/overview" icon='robot'>+ How agent identity and versions surface in the product from these attributes+ </Card>+ <Card title="Guardrails" href="/latest/sdk/features/guardrails" icon='shield-halved'>+ Redact sensitive data before it's captured in spans at all+ </Card>+</CardGroup>diff --git a/docs/latest/sdk/features/evaluations.mdx b/docs/latest/sdk/features/evaluations.mdxindex 615487c7..0ddc03cd 100644--- a/docs/latest/sdk/features/evaluations.mdx+++ b/docs/latest/sdk/features/evaluations.mdx@@ -1,11 +1,11 @@---title: 'Evaluations'-description: 'Evaluate your LLM responses for hallucination, bias, toxicity, and more — using the same evals for dev and prod'+description: 'Evaluate your LLM responses for hallucination, bias, toxicity, and more - using the same evals for dev and prod'---## Overview-The OpenLIT SDK provides server-side evaluations via `openlit.eval()` (Python) and `openlit.eval()` (JS/TS). Evaluations use the same engine, rules, contexts, and custom eval types configured in the OpenLIT dashboard — working identically for development (offline) and production (online) stages.+The OpenLIT SDK provides server-side evaluations via `openlit.eval()` (Python) and `openlit.eval()` (JS/TS). Evaluations use the same engine, rules, contexts, and custom eval types configured in the OpenLIT dashboard - working identically for development (offline) and production (online) stages.<CardGroup cols={2}><Card title="Quick Start" href="#quick-start">@@ -83,7 +83,7 @@ const result = await openlit.eval({// Use in assertionsconsole.log(result.success); // true-console.log(isPassed(result)); // false — hallucination detected+console.log(isPassed(result)); // false - hallucination detectedconsole.log(getFailedEvals(result)); // [{ type: 'hallucination', ... }]```@@ -112,8 +112,8 @@ export OPENLIT_API_KEY="openlit-xxxxx"<Tab title="Python">| Parameter | Type | Description | Default ||-----------|------|-------------|---------|-| `prompt` | `str` | The user prompt sent to the LLM. **Required.** | — |-| `response` | `str` | The LLM's response to evaluate. **Required.** | — |+| `prompt` | `str` | The user prompt sent to the LLM. **Required.** | - |+| `response` | `str` | The LLM's response to evaluate. **Required.** | - || `contexts` | `list[str]` | Ground truth context for the evaluation. | `None` || `eval_types` | `list[str]` | Specific eval types to run (e.g. `["hallucination", "toxicity"]`). Runs all enabled types if omitted. | `None` || `attributes` | `dict` | Trace attributes for rule engine matching (overrides auto-resolved attributes). | `None` |@@ -128,8 +128,8 @@ export OPENLIT_API_KEY="openlit-xxxxx"<Tab title="TypeScript / JavaScript">| Parameter | Type | Description | Default ||-----------|------|-------------|---------|-| `prompt` | `string` | The user prompt sent to the LLM. **Required.** | — |-| `response` | `string` | The LLM's response to evaluate. **Required.** | — |+| `prompt` | `string` | The user prompt sent to the LLM. **Required.** | - |+| `response` | `string` | The LLM's response to evaluate. **Required.** | - || `contexts` | `string[]` | Ground truth context for the evaluation. | `undefined` || `evalTypes` | `string[]` | Specific eval types to run (e.g. `["hallucination", "toxicity"]`). Runs all enabled types if omitted. | `undefined` || `attributes` | `Record<string, string \| number \| boolean>` | Trace attributes for rule engine matching (overrides auto-resolved attributes). | `undefined` |@@ -281,7 +281,7 @@ console.log(`All passed: ${isAllPassed(batchResult)}`);| Parameter | Type | Description | Default ||-----------|------|-------------|---------|-| `dataset` | `list[dict]` | List of items with `prompt` and `response` keys. **Required.** | — |+| `dataset` | `list[dict]` | List of items with `prompt` and `response` keys. **Required.** | - || `eval_types` | `list[str]` | Default eval types (can be overridden per item). | `None` || `attributes` | `dict` | Default attributes (can be overridden per item). | `None` || `threshold_score` | `float` | Default threshold score. | `0.5` |@@ -436,10 +436,3 @@ For `openlit_api_key` and `openlit_url`, the resolution order is:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/features/gpu.mdx b/docs/latest/sdk/features/gpu.mdxindex 9dd529b0..7b569eae 100644--- a/docs/latest/sdk/features/gpu.mdx+++ b/docs/latest/sdk/features/gpu.mdx@@ -146,10 +146,3 @@ GPU monitoring can be implemented in two ways depending on your setup and requirSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/features/guardrails.mdx b/docs/latest/sdk/features/guardrails.mdxindex a6970376..b6212be4 100644--- a/docs/latest/sdk/features/guardrails.mdx+++ b/docs/latest/sdk/features/guardrails.mdx@@ -5,7 +5,7 @@ description: 'Secure your app from Prompt Injection, Sensitive Topics, and Topic## Overview-With the OpenLIT SDK, you can set up guardrails to keep your apps safe by handling tricky or risky prompts sent to AI models. We offer four main guardrails:+With the OpenLIT SDK, you can set up guardrails to keep your apps safe by handling tricky or risky prompts sent to AI models. We offer five main guardrails:<CardGroup cols={2}><Card title="All" href="#all-detector">@@ -20,6 +20,9 @@ With the OpenLIT SDK, you can set up guardrails to keep your apps safe by handli<Card title="Topic Restriction" href="#topic-restriction">Detects and ensures conversations stay within approved topics.</Card>+ <Card title="PII / Data Redaction" href="#pii-data-redaction">+ Detects and redacts secrets, API keys, and personal data before it's sent or traced.+ </Card></CardGroup>## Guardrails@@ -682,6 +685,50 @@ The JSON object returned includes:</Accordion>+### PII / Data Redaction++Detects and redacts secrets, credentials, and personal data - API keys (OpenAI, Anthropic, AWS, GCP, GitHub, Stripe, Slack, and more), emails, phone numbers, SSNs, credit card numbers, IP addresses, bearer/basic auth headers, private keys, connection strings, and generic `password=`/`secret=` patterns - using built-in regex patterns, with support for your own custom patterns.++Unlike the other guardrails, which return a yes/no verdict for you to act on, the PII guard is wired directly into `openlit.init()` and runs automatically around every instrumented LLM call - no separate `detect()` call needed.++#### Usage++```python+import openlit++openlit.init(+ guards=[openlit.PII(action="redact")],+ guard_fail_open=True,+)+```++With `action="redact"`, matched text is replaced with `[REDACTED:<label>]` (e.g. `[REDACTED:email]`) both in the outgoing request and in the captured telemetry - so sensitive values never leave your stack or land in a trace.++#### Parameters++<AccordionGroup>+<Accordion title="`openlit.PII()` Class Parameters">++| Name | Description | Default Value | Example Value |+|------|-------------|----------------|----------------|+| `action` | What to do when PII is detected: `"redact"` replaces matches, `"deny"` blocks the call, `"warn"` logs without blocking. | `"redact"` | `"deny"` |+| `custom_patterns` | Additional regex patterns to detect, as a `{label: regex}` mapping. Combined with the built-in patterns. | `None` | `{"internal_id": r"INT-\d{6}"}` |++</Accordion>+</AccordionGroup>++#### How it works++<Accordion title="Explanation">+1. **Preflight**: Before an instrumented LLM call is made, the guard scans the outgoing prompt/messages. If `action="redact"`, matches are replaced before the request is sent - so the redacted text is what the model (and your telemetry) actually sees.+2. **Postflight**: After a non-streaming response returns, the guard scans response content (e.g. `choices[].message.content`) the same way, on a best-effort basis.+3. **Combine with other guardrails**: `PII` can be combined with `PromptInjection`, `SensitiveTopic`, `TopicRestriction`, and `Moderation` by passing multiple guards in the `guards=[...]` list - each runs as part of the same pipeline.++<Warning>+Streaming responses are not scanned postflight - redaction for streamed output content is a known limitation.+</Warning>+</Accordion>+---<CardGroup cols={3}>@@ -695,10 +742,3 @@ The JSON object returned includes:Send elemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/features/metrics.mdx b/docs/latest/sdk/features/metrics.mdxindex a2fa3bfd..76fa10f6 100644--- a/docs/latest/sdk/features/metrics.mdx+++ b/docs/latest/sdk/features/metrics.mdx@@ -3,12 +3,6 @@ title: 'Metrics'description: "Visualize and monitor AI application metrics in OpenLIT platform with custom dashboards and advanced analytics"----import MetricsQuickstart from '/snippets/quickstart-metrics.mdx';--<MetricsQuickstart />------## SDK Configuration OptionsOpenLIT offers automatic instrumentation with OpenTelemetry for various LLM providers, frameworks, and VectorDBs, enabling you to gain valuable insights into the behavior and performance of your AI applications through metrics.@@ -77,10 +71,3 @@ export OTEL_RESOURCE_ATTRIBUTES="service.instance.id=YOUR_SERVICE_ID,k8s.pod.namSend elemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/features/pricing.mdx b/docs/latest/sdk/features/pricing.mdxindex 3f54efde..277db770 100644--- a/docs/latest/sdk/features/pricing.mdx+++ b/docs/latest/sdk/features/pricing.mdx@@ -50,10 +50,3 @@ Users can provide their own `pricing_json` variable, which can either be a fileSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/features/rule-engine.mdx b/docs/latest/sdk/features/rule-engine.mdxindex 0c062bd7..37e03f8a 100644--- a/docs/latest/sdk/features/rule-engine.mdx+++ b/docs/latest/sdk/features/rule-engine.mdx@@ -5,7 +5,7 @@ description: 'Evaluate rules and retrieve matching contexts, prompts, and evalua## Overview-The OpenLIT SDKs provide a function to evaluate rules against the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine) from your application code. At runtime, send trace attributes (model, provider, service name, etc.) and get back matching rules and their linked entities — contexts, prompts, or evaluation configurations.+The OpenLIT SDKs provide a function to evaluate rules against the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) from your application code. At runtime, send trace attributes (model, provider, service name, etc.) and get back matching rules and their linked entities - contexts, prompts, or evaluation configurations.This enables dynamic, condition-driven retrieval of AI resources without hardcoding logic in your application.@@ -25,13 +25,13 @@ This enables dynamic, condition-driven retrieval of AI resources without hardcod<Steps><Step title="Set up OpenLIT">- Ensure you have an OpenLIT instance running. See [Quick Start](/latest/openlit/quickstart) for setup instructions.+ Ensure you have an OpenLIT instance running. See [Installation](/latest/openlit/installation) for setup instructions.</Step><Step title="Create an API Key">Navigate to **Settings > API Keys** in OpenLIT. Click **Create API Key** and save the key securely.</Step><Step title="Create Rules">- Set up rules with conditions and linked entities in the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine) UI.+ Set up rules with conditions and linked entities in the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) UI.</Step></Steps>@@ -290,7 +290,7 @@ Determine which evaluation types (hallucination, bias, etc.) are linked to rules## Parameters-### Python — `openlit.evaluate_rule()`+### Python - `openlit.evaluate_rule()`| Parameter | Type | Required | Description ||-----------|------|----------|-------------|@@ -301,7 +301,7 @@ Determine which evaluation types (hallucination, bias, etc.) are linked to rules| `include_entity_data` | `bool` | No | Include full entity data in response. Default: `False` || `entity_inputs` | `dict` | No | Inputs for entity resolution (e.g. prompt variables) |-### TypeScript — `Openlit.evaluateRule()`+### TypeScript - `Openlit.evaluateRule()`| Parameter | Type | Required | Description ||-----------|------|----------|-------------|@@ -312,7 +312,7 @@ Determine which evaluation types (hallucination, bias, etc.) are linked to rules| `includeEntityData` | `boolean` | No | Include full entity data. Default: `false` || `entityInputs` | Object | No | Inputs for entity resolution |-### Go — `openlit.EvaluateRule()`+### Go - `openlit.EvaluateRule()`| Field | Type | Required | Description ||-------|------|----------|-------------|@@ -363,7 +363,7 @@ All SDKs return the same response structure:```pythonresult = openlit.evaluate_rule(entity_type="context", fields={"key": "val"})if result is None:- print("Rule evaluation failed — check logs for details")+ print("Rule evaluation failed - check logs for details")```</Tab><Tab title="TypeScript">@@ -391,7 +391,7 @@ All SDKs return the same response structure:---<CardGroup cols={2}>- <Card title="Rule Engine Guide" href="/latest/openlit/prompts-experiments/rule-engine" icon="gears">+ <Card title="Rule Engine Guide" href="/latest/openlit/prompts-experiments/rule-engine/overview" icon="gears">Learn how to create rules, add conditions, and link entities in the OpenLIT UI</Card><Card title="API Reference" href="/latest/openlit/developer-resources/api-reference/endpoint/rule-engine/evaluate" icon="code">diff --git a/docs/latest/sdk/features/tracing.mdx b/docs/latest/sdk/features/tracing.mdxindex 94a325f9..3f4482fc 100644--- a/docs/latest/sdk/features/tracing.mdx+++ b/docs/latest/sdk/features/tracing.mdx@@ -170,10 +170,3 @@ def guess_one_liner(one_liner: str):Send elemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/go-overview.mdx b/docs/latest/sdk/go-overview.mdxindex bf5b66a9..61418fc7 100644--- a/docs/latest/sdk/go-overview.mdx+++ b/docs/latest/sdk/go-overview.mdx@@ -2,7 +2,7 @@title: 'Go SDK Overview'sidebarTitle: 'Go SDK Overview'icon: 'golang'-description: 'OpenTelemetry-native observability for Go AI applications. Monitor OpenAI and Anthropic with automatic token tracking, cost calculation, and distributed tracing.'+description: 'OpenTelemetry-native observability for Go AI applications. Monitor OpenAI, Anthropic, and vLLM with automatic token tracking, cost calculation, and distributed tracing.'---The OpenLIT Go SDK provides **OpenTelemetry-native observability** for Go applications using LLM APIs. It wraps your provider clients with lightweight instrumentation that automatically collects traces, metrics, and cost data without requiring any changes to your application logic.@@ -11,10 +11,13 @@ The OpenLIT Go SDK provides **OpenTelemetry-native observability** for Go applic<CardGroup cols={2}><Card title="OpenAI" icon="robot">- Chat completions, streaming, embeddings, and image generation — with full token usage and cost tracking.+ Chat completions, streaming, embeddings, and image generation - with full token usage and cost tracking.</Card><Card title="Anthropic" icon="robot">- Claude messages and streaming — with cache token tracking and full OpenTelemetry semantic conventions.+ Claude messages and streaming - with cache token tracking and full OpenTelemetry semantic conventions.+ </Card>+ <Card title="vLLM" icon="robot">+ Chat completions and streaming against vLLM's OpenAI-compatible API - with full token usage tracking.</Card></CardGroup>@@ -22,10 +25,10 @@ The OpenLIT Go SDK provides **OpenTelemetry-native observability** for Go applicEvery instrumented call automatically records:-- **Distributed traces** — spans with request/response details, model name, token counts, and cost-- **OTel metrics** — `gen_ai.client.token.usage`, `gen_ai.client.operation.duration`, `gen_ai.server.time_to_first_token`, `gen_ai.server.time_per_output_token`, `gen_ai.server.request.duration`-- **Streaming metrics** — time-to-first-chunk and per-chunk latency observations-- **Cost tracking** — automatic cost calculation using the built-in pricing data+- **Distributed traces** - spans with request/response details, model name, token counts, and cost+- **OTel metrics** - `gen_ai.client.token.usage`, `gen_ai.client.operation.duration`, `gen_ai.server.time_to_first_token`, `gen_ai.server.time_per_output_token`, `gen_ai.server.request.duration`+- **Streaming metrics** - time-to-first-chunk and per-chunk latency observations+- **Cost tracking** - automatic cost calculation using the built-in pricing data## Installation@@ -96,21 +99,21 @@ Pass a `Config` struct to `openlit.Init()`:| Field | Environment Variable | Description | Default ||-------|---------------------|-------------|---------|| `OtlpEndpoint` | `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP backend URL | `http://127.0.0.1:4318` |-| `OtlpHeaders` | — | Additional HTTP headers for OTLP requests | `{}` |-| `ApplicationName` | — | Name of your application | `default` |-| `Environment` | — | Deployment environment label | `default` |-| `ServiceVersion` | — | Service version string | `""` |-| `DisableTracing` | — | Disable trace collection | `false` |-| `DisableMetrics` | — | Disable metrics collection | `false` |-| `DisableBatch` | — | Disable batch export (useful for testing) | `false` |-| `DisableCaptureMessageContent` | — | Omit prompt/completion text from spans | `false` |-| `DetailedTracing` | — | Enable component-level tracing detail | `false` |-| `DisablePricingFetch` | — | Skip fetching remote pricing data | `false` |-| `PricingEndpoint` | — | URL for custom pricing JSON | built-in |-| `PricingInfo` | — | In-process pricing overrides | `{}` |-| `TraceExporterTimeout` | — | Timeout for trace exports | `10s` |-| `MetricExporterTimeout` | — | Timeout for metric exports | `10s` |-| `MetricExportInterval` | — | Interval for metric exports | `30s` |+| `OtlpHeaders` | - | Additional HTTP headers for OTLP requests | `{}` |+| `ApplicationName` | - | Name of your application | `default` |+| `Environment` | - | Deployment environment label | `default` |+| `ServiceVersion` | - | Service version string | `""` |+| `DisableTracing` | - | Disable trace collection | `false` |+| `DisableMetrics` | - | Disable metrics collection | `false` |+| `DisableBatch` | - | Disable batch export (useful for testing) | `false` |+| `DisableCaptureMessageContent` | - | Omit prompt/completion text from spans | `false` |+| `DetailedTracing` | - | Enable component-level tracing detail | `false` |+| `DisablePricingFetch` | - | Skip fetching remote pricing data | `false` |+| `PricingEndpoint` | - | URL for custom pricing JSON | built-in |+| `PricingInfo` | - | In-process pricing overrides | `{}` |+| `TraceExporterTimeout` | - | Timeout for trace exports | `10s` |+| `MetricExporterTimeout` | - | Timeout for metric exports | `10s` |+| `MetricExportInterval` | - | Interval for metric exports | `30s` |### Via environment variable@@ -128,12 +131,15 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318## Getting started<CardGroup cols={2}>- <Card title="OpenAI Integration" href="/latest/sdk/integrations/go-openai" icon="robot">+ <Card title="OpenAI Integration" href="/latest/sdk/integrations/openai" icon="robot">Monitor chat completions, streaming, embeddings, and image generation</Card>- <Card title="Anthropic Integration" href="/latest/sdk/integrations/go-anthropic" icon="robot">+ <Card title="Anthropic Integration" href="/latest/sdk/integrations/anthropic" icon="robot">Monitor Claude messages and streaming with cache token tracking</Card>+ <Card title="vLLM Integration" href="/latest/sdk/integrations/vllm" icon="robot">+ Monitor chat completions and streaming against your vLLM server+ </Card><Card title="Destinations" href="/latest/sdk/destinations/overview" icon="link">Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card>diff --git a/docs/latest/sdk/instrumentation-methods.mdx b/docs/latest/sdk/instrumentation-methods.mdxindex 4ce45f37..42d2a982 100644--- a/docs/latest/sdk/instrumentation-methods.mdx+++ b/docs/latest/sdk/instrumentation-methods.mdx@@ -60,10 +60,3 @@ OpenLIT SDK offers two ways to add opentelemetry-native observability to your AISend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/ag2.mdx b/docs/latest/sdk/integrations/ag2.mdxindex 9e2bac84..9171d68e 100644--- a/docs/latest/sdk/integrations/ag2.mdx+++ b/docs/latest/sdk/integrations/ag2.mdx@@ -1,5 +1,6 @@---title: 'Monitor AutoGen using OpenTelemetry'+description: 'OpenLIT auto-instruments AutoGen (AG2) agents via OpenTelemetry, giving you AI observability and tracing into multi-agent performance and operations.'sidebarTitle: 'AutoGen / AG2'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/agent-framework.mdx b/docs/latest/sdk/integrations/agent-framework.mdxnew file mode 100644index 00000000..f4280fee--- /dev/null+++ b/docs/latest/sdk/integrations/agent-framework.mdx@@ -0,0 +1,39 @@+---+title: 'Monitor Microsoft Agent Framework using OpenTelemetry'+description: 'OpenLIT auto-instruments Microsoft Agent Framework via OpenTelemetry, tracing agent execution, tool calls, and workflows for AI observability.'+sidebarTitle: 'Agent Framework'+---++import IntegrationMethods from '/snippets/integration-methods.mdx';++OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor AI agents built with Microsoft Agent Framework. This includes tracing agent construction, agent execution, tool invocation, and workflow runs, giving you AI observability into multi-agent and multi-step workflows.++Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up. OpenLIT disables Agent Framework's built-in telemetry layer and replaces it with OTel GenAI semantic-convention-compliant spans, so you don't get duplicate traces.++The integration is compatible with `agent-framework >= 1.0.0rc1`.++## Get started++<Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ ```shell+ pip install openlit+ ```+ </Step>+ <IntegrationMethods />+</Steps>++---++<CardGroup cols={3}>+ <Card title="Quickstart: LLM Observability" href="/latest/sdk/quickstart-ai-observability" icon='bolt'>+ Production-ready AI monitoring setup in 2 simple steps with zero code changes+ </Card>+ <Card title="Configuration" href="/latest/sdk/configuration" icon='bolt'>+ Configure the OpenLIT SDK according to you requirements.+ </Card>+ <Card title="Destinations" href="/latest/sdk/destinations/overview" icon='link'>+ Send telemetry to Datadog, Grafana, New Relic, and other observability stacks+ </Card>+</CardGroup>diff --git a/docs/latest/sdk/integrations/agent-governance-toolkit.mdx b/docs/latest/sdk/integrations/agent-governance-toolkit.mdxindex 0a8da3f9..e6d678e1 100644--- a/docs/latest/sdk/integrations/agent-governance-toolkit.mdx+++ b/docs/latest/sdk/integrations/agent-governance-toolkit.mdx@@ -1,5 +1,6 @@---title: 'Monitor AI Agent Governance using OpenTelemetry'+description: "OpenLIT integrates with Microsoft's Agent Governance Toolkit for AI observability into policy evaluations, trust scores, and capability violations."sidebarTitle: 'Agent Governance Toolkit'---@@ -76,10 +77,10 @@ Agent Governance Toolkit emits OpenTelemetry signals for every governance decisiCreate a governance-specific dashboard in OpenLIT to monitor:-1. **Policy evaluation rate** — How many governance checks per minute-2. **Violation trend** — Spikes in policy violations over time-3. **Evaluation latency** — Are governance checks meeting the <5ms SLA-4. **Trust score distribution** — Current trust scores across your agent fleet+1. **Policy evaluation rate** - How many governance checks per minute+2. **Violation trend** - Spikes in policy violations over time+3. **Evaluation latency** - Are governance checks meeting the <5ms SLA+4. **Trust score distribution** - Current trust scores across your agent fleetUse the [Dashboard Builder](/latest/openlit/dashboards/create-dashboard) with these metric names:@@ -110,12 +111,12 @@ metrics.start()A pre-built **Governance Health** dashboard is available for import into OpenLIT. It provides:-- **Total Policy Evaluations** — count with trend-- **Total Violations** — count with severity breakdown-- **Violations Over Time** — time-series chart-- **Policy Eval Latency** — p50/p99 latency-- **Violations by Agent** — which agents trigger the most violations-- **Recent Violations** — audit table with agent, tool, policy, verdict+- **Total Policy Evaluations** - count with trend+- **Total Violations** - count with severity breakdown+- **Violations Over Time** - time-series chart+- **Policy Eval Latency** - p50/p99 latency+- **Violations by Agent** - which agents trigger the most violations+- **Recent Violations** - audit table with agent, tool, policy, verdictImport via **OpenLIT → Dashboards → Import** using the JSON below:@@ -124,7 +125,7 @@ Import via **OpenLIT → Dashboards → Import** using the JSON below:{"id": "a1c2d3e4-f5a6-4b7c-8d9e-0f1a2b3c4d5e","title": "Governance Health",- "description": "Monitor AI agent governance — policy evaluations, violations, trust scores, and enforcement latency across your agent fleet.",+ "description": "Monitor AI agent governance - policy evaluations, violations, trust scores, and enforcement latency across your agent fleet.","parentId": null,"isMainDashboard": false,"isPinned": false,diff --git a/docs/latest/sdk/integrations/ai21.mdx b/docs/latest/sdk/integrations/ai21.mdxindex 2ed33c79..748e36c4 100644--- a/docs/latest/sdk/integrations/ai21.mdx+++ b/docs/latest/sdk/integrations/ai21.mdx@@ -1,5 +1,6 @@---title: 'Monitor AI21 using OpenTelemetry'+description: 'OpenLIT auto-instruments AI21 models with OpenTelemetry for AI observability, capturing performance, token usage, and cost tracking across your LLM calls.'sidebarTitle: 'AI21'---@@ -46,10 +47,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/amd-gpu.mdx b/docs/latest/sdk/integrations/amd-gpu.mdxindex 15c78b86..37f76f42 100644--- a/docs/latest/sdk/integrations/amd-gpu.mdx+++ b/docs/latest/sdk/integrations/amd-gpu.mdx@@ -1,5 +1,6 @@---title: 'Monitor AMD GPUs using OpenTelemetry'+description: 'OpenLIT uses OpenTelemetry to deliver AI observability for AMD GPUs, tracking utilization, temperature, memory usage, and power consumption metrics.'sidebarTitle: 'AMD GPUs'---@@ -104,7 +105,7 @@ OpenLIT uses OpenTelemetry to help you monitor AMD GPUs. This includes tracking</Accordion></AccordionGroup>-<Card title="Collected Metrics" href="/latest/openlit/observability/metrics#gpu-metrics" icon='table'>+<Card title="Collected Metrics" href="/latest/openlit/observability/telemetry#metrics" icon='table'>Details on the types of metrics collected and their descriptions.</Card>@@ -121,10 +122,3 @@ Details on the types of metrics collected and their descriptions.Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/anthropic.mdx b/docs/latest/sdk/integrations/anthropic.mdxindex 3d8c2903..a4299487 100644--- a/docs/latest/sdk/integrations/anthropic.mdx+++ b/docs/latest/sdk/integrations/anthropic.mdx@@ -1,38 +1,246 @@---title: 'Monitor Claude using OpenTelemetry'+description: 'OpenLIT auto-instruments Anthropic Claude with OpenTelemetry, enabling AI observability with cost tracking, token usage, and performance tracing for LLM apps.'sidebarTitle: 'Anthropic'---import IntegrationMethods from '/snippets/integration-methods.mdx';-OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor LLM applications built using Anthropic models. This includes tracking performance, token usage, costs, and how users interact with the application.--Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.--The integration is compatible with-- Anthropic Python SDK client `>= 0.21.0`-- Anthropic Typescript SDK client `>= 0.21.0`--## Get started--<Steps>- <Step title="Install OpenLIT">- Open your command line or terminal and run:- <Tabs>- <Tab title="Python">- ```shell- pip install openlit- ```- </Tab>- <Tab title="Typescript">- ```shell- npm install openlit- ```- </Tab>- </Tabs>- </Step>- <IntegrationMethods />-</Steps>+OpenLIT uses OpenTelemetry instrumentation to help you monitor LLM applications built using Anthropic models. This includes tracking performance, token usage, costs, and how users interact with the application.++The integration is compatible with:+- Anthropic Python SDK client `>= 0.21.0`+- Anthropic Typescript SDK client `>= 0.21.0`+- Anthropic Go SDK, via OpenLIT's `InstrumentedClient` wrapper++## Get started++<Tabs>+ <Tab title="Python / TypeScript">+ Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.++ <Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ <Tabs>+ <Tab title="Python">+ ```shell+ pip install openlit+ ```+ </Tab>+ <Tab title="Typescript">+ ```shell+ npm install openlit+ ```+ </Tab>+ </Tabs>+ </Step>+ <IntegrationMethods />+ </Steps>+ </Tab>++ <Tab title="Go">+ The Go SDK wraps your Anthropic client with an `InstrumentedClient` that automatically emits traces and metrics for every API call - with zero changes to your application logic.++ The integration supports:+ - Messages (standard and streaming)+ - Prompt caching token tracking (`cache_creation_input_tokens`, `cache_read_input_tokens`)+ - Tool use++ <Steps>+ <Step title="Install the Go SDK">+ Open your terminal and run:++ <Tabs>+ <Tab title="Latest Version">+ ```shell+ go get github.com/openlit/openlit/sdk/go+ ```+ </Tab>+ <Tab title="Specific Version">+ ```shell+ go get github.com/openlit/openlit/sdk/[email protected]+ ```+ Replace `v1.2.3` with the version you want to install.+ </Tab>+ </Tabs>+ </Step>++ <Step title="Initialize OpenLIT">+ Add this once at the start of your application (e.g. in `main()`):++ <Tabs>+ <Tab title="Via Config Struct">+ ```go+ import (+ "context"+ openlit "github.com/openlit/openlit/sdk/go"+ )++ if err := openlit.Init(openlit.Config{+ OtlpEndpoint: "YOUR_OTEL_ENDPOINT",+ ApplicationName: "my-ai-app",+ Environment: "production",+ }); err != nil {+ log.Fatal(err)+ }+ defer openlit.Shutdown(context.Background())+ ```+ </Tab>+ <Tab title="Via Environment Variable">+ ```go+ import (+ "context"+ openlit "github.com/openlit/openlit/sdk/go"+ )++ if err := openlit.Init(openlit.Config{+ ApplicationName: "my-ai-app",+ }); err != nil {+ log.Fatal(err)+ }+ defer openlit.Shutdown(context.Background())+ ```++ Then set your OTLP endpoint via environment variable:+ ```shell+ export OTEL_EXPORTER_OTLP_ENDPOINT=YOUR_OTEL_ENDPOINT+ ```+ </Tab>+ </Tabs>++ **Replace** `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` for a local OpenLIT deployment.+ </Step>++ <Step title="Create an instrumented client">+ Replace your existing Anthropic client creation with the OpenLIT instrumented client:++ ```go+ import "github.com/openlit/openlit/sdk/go/instrumentation/anthropic"++ client := anthropic.NewClient("your-anthropic-api-key")+ ```++ Optional configuration:+ ```go+ // Custom API version or base URL+ client := anthropic.NewClient("your-api-key",+ anthropic.WithAPIVersion("2023-06-01"),+ anthropic.WithBaseURL("https://api.anthropic.com/v1"),+ )+ ```+ </Step>++ <Step title="Use the client">+ Use the instrumented client exactly as you would a normal Anthropic client:++ **Message:**+ ```go+ resp, err := client.CreateMessage(ctx, anthropic.MessageRequest{+ Model: "claude-opus-4-5",+ MaxTokens: 256,+ System: "You are a helpful assistant.",+ Messages: []anthropic.Message{+ {Role: "user", Content: "What is OpenTelemetry?"},+ },+ })+ if err != nil {+ return err+ }++ for _, block := range resp.Content {+ if block.Type == "text" {+ fmt.Println(block.Text)+ }+ }+ ```++ **Streaming:**+ ```go+ stream, err := client.CreateMessageStream(ctx, anthropic.MessageRequest{+ Model: "claude-opus-4-5",+ MaxTokens: 1024,+ Messages: []anthropic.Message{+ {Role: "user", Content: "Tell me a story."},+ },+ })+ if err != nil {+ return err+ }+ defer stream.Close()++ for {+ event, err := stream.Recv()+ if err == io.EOF {+ break+ }+ if err != nil {+ return err+ }+ if event.Type == "content_block_delta" && event.Delta != nil {+ fmt.Print(event.Delta.Text)+ }+ }+ ```++ **Tool use:**+ ```go+ resp, err := client.CreateMessage(ctx, anthropic.MessageRequest{+ Model: "claude-opus-4-5",+ MaxTokens: 512,+ Messages: []anthropic.Message{+ {Role: "user", Content: "What is the weather in San Francisco?"},+ },+ Tools: []anthropic.Tool{+ {+ Name: "get_weather",+ Description: "Get the current weather in a location",+ InputSchema: map[string]interface{}{+ "type": "object",+ "properties": map[string]interface{}{+ "location": map[string]interface{}{+ "type": "string",+ "description": "The city and state",+ },+ },+ "required": []string{"location"},+ },+ },+ },+ })+ ```+ </Step>+ </Steps>++ ### What gets collected++ Every call to the instrumented client automatically records:++ | Data | Attribute |+ |------|-----------|+ | Operation name | `gen_ai.operation.name` |+ | Model requested | `gen_ai.request.model` |+ | Model used | `gen_ai.response.model` |+ | Response ID | `gen_ai.response.id` |+ | Input tokens | `gen_ai.usage.input_tokens` |+ | Output tokens | `gen_ai.usage.output_tokens` |+ | Cache creation tokens | `gen_ai.usage.prompt_tokens_details.cache_write` |+ | Cache read tokens | `gen_ai.usage.prompt_tokens_details.cache_read` |+ | Estimated cost | `gen_ai.usage.cost` |+ | Finish reason | `gen_ai.response.finish_reasons` |+ | Tool calls | `gen_ai.tool.name`, `gen_ai.tool.call.id` |+ | Time to first token | `gen_ai.server.time_to_first_token` (streaming) |+ | Time per output token | `gen_ai.server.time_per_output_token` (streaming) |++ Metrics emitted:+ - `gen_ai.client.token.usage` - token usage histogram (input/output)+ - `gen_ai.client.operation.duration` - total operation duration+ - `gen_ai.server.time_to_first_token` - TTFT for streaming+ - `gen_ai.client.operation.time_to_first_chunk` - client-side TTFT+ - `gen_ai.client.operation.time_per_output_chunk` - per-chunk latency+ - `gen_ai.server.request.duration` - estimated server processing time+ </Tab>+</Tabs>---@@ -47,10 +255,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/assemblyai.mdx b/docs/latest/sdk/integrations/assemblyai.mdxindex 485639fa..af55f1a5 100644--- a/docs/latest/sdk/integrations/assemblyai.mdx+++ b/docs/latest/sdk/integrations/assemblyai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Assembly AI using OpenTelemetry'+description: "OpenLIT brings AI observability to AssemblyAI's speech and audio models via OpenTelemetry, tracking performance, cost tracking, and voice input settings."sidebarTitle: 'Assembly AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/astradb.mdx b/docs/latest/sdk/integrations/astradb.mdxindex 8be08ab8..7ec18118 100644--- a/docs/latest/sdk/integrations/astradb.mdx+++ b/docs/latest/sdk/integrations/astradb.mdx@@ -1,5 +1,6 @@---title: 'Monitor AstraDB using OpenTelemetry'+description: 'OpenLIT auto-instruments AstraDB as a vector store using OpenTelemetry, providing AI observability with tracing into performance and operation stats.'sidebarTitle: 'AstraDB'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/azure-ai-inference.mdx b/docs/latest/sdk/integrations/azure-ai-inference.mdxindex 5834e5ce..3d5e6024 100644--- a/docs/latest/sdk/integrations/azure-ai-inference.mdx+++ b/docs/latest/sdk/integrations/azure-ai-inference.mdx@@ -1,5 +1,6 @@---title: 'Monitor Azure AI Inference using OpenTelemetry'+description: 'OpenLIT auto-instruments Azure AI Inference models through OpenTelemetry, delivering AI observability with cost tracking, token usage, and performance data.'sidebarTitle: 'Azure AI Inference'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/azure-openai.mdx b/docs/latest/sdk/integrations/azure-openai.mdxindex 37eef5b2..cbe467da 100644--- a/docs/latest/sdk/integrations/azure-openai.mdx+++ b/docs/latest/sdk/integrations/azure-openai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Azure OpenAI using OpenTelemetry'+description: 'OpenLIT auto-instruments Azure OpenAI models via OpenTelemetry, providing AI observability with cost tracking, token usage, and performance monitoring.'sidebarTitle: 'Azure OpenAI'---@@ -48,10 +49,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/bedrock.mdx b/docs/latest/sdk/integrations/bedrock.mdxindex 2d8647b7..17b748a1 100644--- a/docs/latest/sdk/integrations/bedrock.mdx+++ b/docs/latest/sdk/integrations/bedrock.mdx@@ -1,5 +1,6 @@---title: 'Monitor Amazon Bedrock using OpenTelemetry'+description: 'OpenLIT auto-instruments Amazon Bedrock models using OpenTelemetry, offering AI observability with cost tracking, token usage, and performance tracing.'sidebarTitle: 'Amazon Bedrock'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/browser-use.mdx b/docs/latest/sdk/integrations/browser-use.mdxindex 34dee0c2..06aee20c 100644--- a/docs/latest/sdk/integrations/browser-use.mdx+++ b/docs/latest/sdk/integrations/browser-use.mdx@@ -1,5 +1,6 @@---title: 'Monitor Browser Use using OpenTelemetry'+description: 'OpenLIT auto-instruments Browser Use agents through OpenTelemetry, adding AI observability with tracing into browser agent performance and operations.'sidebarTitle: 'Browser Use'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/chromadb.mdx b/docs/latest/sdk/integrations/chromadb.mdxindex 6a124edc..894b0d2a 100644--- a/docs/latest/sdk/integrations/chromadb.mdx+++ b/docs/latest/sdk/integrations/chromadb.mdx@@ -1,5 +1,6 @@---title: 'Monitor ChromaDB using OpenTelemetry'+description: 'OpenLIT auto-instruments ChromaDB as a vector store with OpenTelemetry, giving you AI observability and tracing into query performance and operation stats.'sidebarTitle: 'ChromaDB'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/claude-agent-sdk.mdx b/docs/latest/sdk/integrations/claude-agent-sdk.mdxindex 372504a3..112063b4 100644--- a/docs/latest/sdk/integrations/claude-agent-sdk.mdx+++ b/docs/latest/sdk/integrations/claude-agent-sdk.mdx@@ -1,5 +1,6 @@---title: 'Monitor Claude Agent SDK using OpenTelemetry'+description: "OpenLIT auto-instruments Anthropic's Claude Agent SDK via OpenTelemetry, adding AI observability with tracing across agent runs, tool calls, and LLM calls."sidebarTitle: 'Claude Agent SDK'---@@ -47,12 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Running in Kubernetes? Try the OpenLIT Operator"- icon={<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">- <title>Kubernetes</title>- <path fill="#FFA500" d="M10.204 14.35l.007.01-.999 2.413a5.171 5.171 0 0 1-2.075-2.597l2.578-.437.004.005a.44.44 0 0 1 .484.606zm-.833-2.129a.44.44 0 0 0 .173-.756l.002-.011L7.585 9.7a5.143 5.143 0 0 0-.73 3.255l2.514-.725.002-.009zm1.145-1.98a.44.44 0 0 0 .699-.337l.01-.005.15-2.62a5.144 5.144 0 0 0-3.01 1.442l2.147 1.523.004-.002zm.76 2.75l.723.349.722-.347.18-.78-.5-.623h-.804l-.5.623.179.779zm1.5-3.095a.44.44 0 0 0 .7.336l.008.003 2.134-1.513a5.188 5.188 0 0 0-2.992-1.442l.148 2.615.002.001zm10.876 5.97l-5.773 7.181a1.6 1.6 0 0 1-1.248.594l-9.261.003a1.6 1.6 0 0 1-1.247-.596l-5.776-7.18a1.583 1.583 0 0 1-.307-1.34L2.1 5.573c.108-.47.425-.864.863-1.073L11.305.513a1.606 1.606 0 0 1 1.385 0l8.345 3.985c.438.209.755.604.863 1.073l2.062 8.955c.108.47-.005.963-.308 1.34zm-3.289-2.057c-.042-.01-.103-.026-.145-.034-.174-.033-.315-.025-.479-.038-.35-.037-.638-.067-.895-.148-.105-.04-.18-.165-.216-.216l-.201-.059a6.45 6.45 0 0 0-.105-2.332 6.465 6.465 0 0 0-.936-2.163c.052-.047.15-.133.177-.159.008-.09.001-.183.094-.282.197-.185.444-.338.743-.522.142-.084.273-.137.415-.242.032-.024.076-.062.11-.089.24-.191.295-.52.123-.736-.172-.216-.506-.236-.745-.045-.034.027-.08.062-.111.088-.134.116-.217.23-.33.35-.246.25-.45.458-.673.609-.097.056-.239.037-.303.033l-.19.135a6.545 6.545 0 0 0-4.146-2.003l-.012-.223c-.065-.062-.143-.115-.163-.25-.022-.268.015-.557.057-.905.023-.163.061-.298.068-.475.001-.04-.001-.099-.001-.142 0-.306-.224-.555-.5-.555-.275 0-.499.249-.499.555l.001.014c0 .041-.002.092 0 .128.006.177.044.312.067.475.042.348.078.637.056.906a.545.545 0 0 1-.162.258l-.012.211a6.424 6.424 0 0 0-4.166 2.003 8.373 8.373 0 0 1-.18-.128c-.09.012-.18.04-.297-.029-.223-.15-.427-.358-.673-.608-.113-.12-.195-.234-.329-.349-.03-.026-.077-.062-.111-.088a.594.594 0 0 0-.348-.132.481.481 0 0 0-.398.176c-.172.216-.117.546.123.737l.007.005.104.083c.142.105.272.159.414.242.299.185.546.338.743.522.076.082.09.226.1.288l.16.143a6.462 6.462 0 0 0-1.02 4.506l-.208.06c-.055.072-.133.184-.215.217-.257.081-.546.11-.895.147-.164.014-.305.006-.48.039-.037.007-.09.02-.133.03l-.004.002-.007.002c-.295.071-.484.342-.423.608.061.267.349.429.645.365l.007-.001.01-.003.129-.029c.17-.046.294-.113.448-.172.33-.118.604-.217.87-.256.112-.009.23.069.288.101l.217-.037a6.5 6.5 0 0 0 2.88 3.596l-.09.218c.033.084.069.199.044.282-.097.252-.263.517-.452.813-.091.136-.185.242-.268.399-.02.037-.045.095-.064.134-.128.275-.034.591.213.71.248.12.556-.007.69-.282v-.002c.02-.039.046-.09.062-.127.07-.162.094-.301.144-.458.132-.332.205-.68.387-.897.05-.06.13-.082.215-.105l.113-.205a6.453 6.453 0 0 0 4.609.012l.106.192c.086.028.18.042.256.155.136.232.229.507.342.84.05.156.074.295.145.457.016.037.043.09.062.129.133.276.442.402.69.282.247-.118.341-.435.213-.71-.02-.039-.045-.096-.065-.134-.083-.156-.177-.261-.268-.398-.19-.296-.346-.541-.443-.793-.04-.13.007-.21.038-.294-.018-.022-.059-.144-.083-.202a6.499 6.499 0 0 0 2.88-3.622c.064.01.176.03.213.038.075-.05.144-.114.28-.104.266.039.54.138.87.256.154.06.277.128.448.173.036.01.088.019.13.028l.009.003.007.001c.297.064.584-.098.645-.365.06-.266-.128-.537-.423-.608zM16.4 9.701l-1.95 1.746v.005a.44.44 0 0 0 .173.757l.003.01 2.526.728a5.199 5.199 0 0 0-.108-1.674A5.208 5.208 0 0 0 16.4 9.7zm-4.013 5.325a.437.437 0 0 0-.404-.232.44.44 0 0 0-.372.233h-.002l-1.268 2.292a5.164 5.164 0 0 0 3.326.003l-1.27-2.296h-.01zm1.888-1.293a.44.44 0 0 0-.27.036.44.44 0 0 0-.214.572l-.003.004 1.01 2.438a5.15 5.15 0 0 0 2.081-2.615l-2.6-.44-.004.005z"/></svg>}- href="/latest/operator/overview"- >- Automatically inject instrumentation into existing workloads without modifying pod specs, container images, or application code.- </Card>diff --git a/docs/latest/sdk/integrations/cohere.mdx b/docs/latest/sdk/integrations/cohere.mdxindex 4fd86727..a446fdea 100644--- a/docs/latest/sdk/integrations/cohere.mdx+++ b/docs/latest/sdk/integrations/cohere.mdx@@ -1,5 +1,6 @@---title: 'Monitor Cohere using OpenTelemetry'+description: 'OpenLIT auto-instruments Cohere models using OpenTelemetry, delivering AI observability with cost tracking and token usage monitoring for LLM apps.'sidebarTitle: 'Cohere'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/controlflow.mdx b/docs/latest/sdk/integrations/controlflow.mdxindex cf1103e9..53682ae9 100644--- a/docs/latest/sdk/integrations/controlflow.mdx+++ b/docs/latest/sdk/integrations/controlflow.mdx@@ -1,5 +1,6 @@---title: 'Monitor ControlFlow using OpenTelemetry'+description: 'OpenLIT auto-instruments AI agents built with ControlFlow via OpenTelemetry, adding AI observability and tracing into performance and operation stats.'sidebarTitle: 'ControlFlow'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/crawl4ai.mdx b/docs/latest/sdk/integrations/crawl4ai.mdxindex 786bba8c..ef435f1c 100644--- a/docs/latest/sdk/integrations/crawl4ai.mdx+++ b/docs/latest/sdk/integrations/crawl4ai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Crawl4AI using OpenTelemetry'+description: 'OpenLIT auto-instruments Crawl4AI crawling agents with OpenTelemetry, bringing AI observability and tracing into crawler performance and operation stats.'sidebarTitle: 'Crawl4AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/crewai.mdx b/docs/latest/sdk/integrations/crewai.mdxindex f6d9b0a0..ce037589 100644--- a/docs/latest/sdk/integrations/crewai.mdx+++ b/docs/latest/sdk/integrations/crewai.mdx@@ -1,5 +1,6 @@---title: 'Monitor CrewAI using OpenTelemetry'+description: 'OpenLIT auto-instruments AI agents built with CrewAI via OpenTelemetry, providing AI observability and tracing into multi-agent performance and operations.'sidebarTitle: 'CrewAI'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/deepseek.mdx b/docs/latest/sdk/integrations/deepseek.mdxindex 2e689ae2..71c4c0ba 100644--- a/docs/latest/sdk/integrations/deepseek.mdx+++ b/docs/latest/sdk/integrations/deepseek.mdx@@ -1,5 +1,6 @@---title: 'Monitor DeepSeek using OpenTelemetry'+description: 'OpenLIT auto-instruments DeepSeek models (via the OpenAI SDK) with OpenTelemetry, enabling AI observability with cost tracking and token usage monitoring.'sidebarTitle: 'DeepSeek'---@@ -38,10 +39,3 @@ As DeepSeek uses the OpenAI SDK, This integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/digitalocean-gradient.mdx b/docs/latest/sdk/integrations/digitalocean-gradient.mdxindex 4596c395..364eceb0 100644--- a/docs/latest/sdk/integrations/digitalocean-gradient.mdx+++ b/docs/latest/sdk/integrations/digitalocean-gradient.mdx@@ -1,5 +1,6 @@---title: 'Monitor DigitalOcean Gradient using OpenTelemetry'+description: "OpenLIT auto-instruments DigitalOcean's Gradient AI Platform via the gradient SDK, giving AI observability with cost tracking across chat, image, and RAG calls."sidebarTitle: 'DigitalOcean Gradient'---@@ -42,10 +43,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/digitalocean.mdx b/docs/latest/sdk/integrations/digitalocean.mdxindex 3622a04a..1be7ba37 100644--- a/docs/latest/sdk/integrations/digitalocean.mdx+++ b/docs/latest/sdk/integrations/digitalocean.mdx@@ -1,5 +1,6 @@---title: 'Monitor DigitalOcean GenAI (pydo) using OpenTelemetry'+description: "OpenLIT auto-instruments DigitalOcean's GenAI Platform via the pydo SDK, delivering AI observability with cost tracking across chat, embeddings, and agents."sidebarTitle: 'DigitalOcean (pydo)'---@@ -42,10 +43,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/dspy.mdx b/docs/latest/sdk/integrations/dspy.mdxindex f51de9d4..640f92d8 100644--- a/docs/latest/sdk/integrations/dspy.mdx+++ b/docs/latest/sdk/integrations/dspy.mdx@@ -1,5 +1,6 @@---title: 'Monitor DSPy using OpenTelemetry'+description: 'OpenLIT auto-instruments AI applications built with DSPy using OpenTelemetry, adding AI observability and tracing into pipeline performance and operations.'sidebarTitle: 'DSPy'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/dynamiq.mdx b/docs/latest/sdk/integrations/dynamiq.mdxindex 6e1bcd0d..0ce26418 100644--- a/docs/latest/sdk/integrations/dynamiq.mdx+++ b/docs/latest/sdk/integrations/dynamiq.mdx@@ -1,5 +1,6 @@---title: 'Monitor Dynamiq using OpenTelemetry'+description: 'OpenLIT auto-instruments AI agents built with Dynamiq through OpenTelemetry, providing AI observability and tracing into agent performance and operations.'sidebarTitle: 'Dynamiq'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/elevenlabs.mdx b/docs/latest/sdk/integrations/elevenlabs.mdxindex 74ec6479..ca62c21d 100644--- a/docs/latest/sdk/integrations/elevenlabs.mdx+++ b/docs/latest/sdk/integrations/elevenlabs.mdx@@ -1,5 +1,6 @@---title: 'Monitor ElevenLabs using OpenTelemetry'+description: "OpenLIT auto-instruments ElevenLabs' voice and speech models via OpenTelemetry, offering AI observability with cost tracking across voice generation calls."sidebarTitle: 'ElevenLabs'---@@ -49,7 +50,7 @@ const audio = await client.textToSpeech.convert("voice-id", {|--------|----------|| `captureMessageContent` | Gates `gen_ai.input.messages` / `gen_ai.output.messages` on spans and in events || `disableEvents` | Suppresses all inference events |-| `disableMetrics` | Global — metrics instruments are not set up |+| `disableMetrics` | Global - metrics instruments are not set up || `disabledInstrumentors: ['elevenlabs']` | Disables this instrumentation |<Note>@@ -92,10 +93,3 @@ On the Python SDK, inference events are only emitted when `capture_message_conteSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/featherless.mdx b/docs/latest/sdk/integrations/featherless.mdxindex e08218f7..833d832c 100644--- a/docs/latest/sdk/integrations/featherless.mdx+++ b/docs/latest/sdk/integrations/featherless.mdx@@ -1,5 +1,6 @@---title: 'Monitor Featherless using OpenTelemetry'+description: 'OpenLIT auto-instruments Featherless models (via the OpenAI SDK) with OpenTelemetry, enabling AI observability with cost tracking and token usage monitoring.'sidebarTitle: 'Featherless'---@@ -37,10 +38,3 @@ As Featherless uses the OpenAI SDK, This integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/firecrawl.mdx b/docs/latest/sdk/integrations/firecrawl.mdxindex d43cbc86..0031e554 100644--- a/docs/latest/sdk/integrations/firecrawl.mdx+++ b/docs/latest/sdk/integrations/firecrawl.mdx@@ -1,5 +1,6 @@---title: 'Monitor FireCrawl using OpenTelemetry'+description: 'Track FireCrawl web crawling agents with OpenTelemetry auto-instrumentation for AI observability into crawl performance and operation stats.'sidebarTitle: 'FireCrawl'---@@ -45,10 +46,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/github-models.mdx b/docs/latest/sdk/integrations/github-models.mdxindex 2280b9d1..564633f9 100644--- a/docs/latest/sdk/integrations/github-models.mdx+++ b/docs/latest/sdk/integrations/github-models.mdx@@ -1,5 +1,6 @@---title: 'Monitor GitHub Models using OpenTelemetry'+description: 'Monitor GitHub Models LLM apps built on azure-ai-inference with OpenTelemetry auto-instrumentation for AI observability, tokens, and cost.'sidebarTitle: 'GitHub Models'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/go-anthropic.mdx b/docs/latest/sdk/integrations/go-anthropic.mdxdeleted file mode 100644index 77464e37..00000000--- a/docs/latest/sdk/integrations/go-anthropic.mdx+++ /dev/null@@ -1,222 +0,0 @@-----title: 'Monitor Claude using OpenTelemetry (Go)'-sidebarTitle: 'Anthropic (Go)'------OpenLIT uses OpenTelemetry instrumentation to help you monitor Go applications built with Anthropic models. This includes tracking performance, token usage (including prompt cache tokens), costs, and how users interact with the application.--The Go SDK wraps your Anthropic client with an `InstrumentedClient` that automatically emits traces and metrics for every API call — with zero changes to your application logic.--The integration supports:-- Messages (standard and streaming)-- Prompt caching token tracking (`cache_creation_input_tokens`, `cache_read_input_tokens`)-- Tool use--## Get started--<Steps>- <Step title="Install the Go SDK">- Open your terminal and run:-- <Tabs>- <Tab title="Latest Version">- ```shell- go get github.com/openlit/openlit/sdk/go- ```- </Tab>- <Tab title="Specific Version">- ```shell- go get github.com/openlit/openlit/sdk/[email protected]- ```- Replace `v1.2.3` with the version you want to install.- </Tab>- </Tabs>- </Step>-- <Step title="Initialize OpenLIT">- Add this once at the start of your application (e.g. in `main()`):-- <Tabs>- <Tab title="Via Config Struct">- ```go- import (- "context"- openlit "github.com/openlit/openlit/sdk/go"- )-- if err := openlit.Init(openlit.Config{- OtlpEndpoint: "YOUR_OTEL_ENDPOINT",- ApplicationName: "my-ai-app",- Environment: "production",- }); err != nil {- log.Fatal(err)- }- defer openlit.Shutdown(context.Background())- ```- </Tab>- <Tab title="Via Environment Variable">- ```go- import (- "context"- openlit "github.com/openlit/openlit/sdk/go"- )-- if err := openlit.Init(openlit.Config{- ApplicationName: "my-ai-app",- }); err != nil {- log.Fatal(err)- }- defer openlit.Shutdown(context.Background())- ```-- Then set your OTLP endpoint via environment variable:- ```shell- export OTEL_EXPORTER_OTLP_ENDPOINT=YOUR_OTEL_ENDPOINT- ```- </Tab>- </Tabs>-- **Replace** `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` for a local OpenLIT deployment.- </Step>-- <Step title="Create an instrumented client">- Replace your existing Anthropic client creation with the OpenLIT instrumented client:-- ```go- import "github.com/openlit/openlit/sdk/go/instrumentation/anthropic"-- client := anthropic.NewClient("your-anthropic-api-key")- ```-- Optional configuration:- ```go- // Custom API version or base URL- client := anthropic.NewClient("your-api-key",- anthropic.WithAPIVersion("2023-06-01"),- anthropic.WithBaseURL("https://api.anthropic.com/v1"),- )- ```- </Step>-- <Step title="Use the client">- Use the instrumented client exactly as you would a normal Anthropic client:-- **Message:**- ```go- resp, err := client.CreateMessage(ctx, anthropic.MessageRequest{- Model: "claude-opus-4-5",- MaxTokens: 256,- System: "You are a helpful assistant.",- Messages: []anthropic.Message{- {Role: "user", Content: "What is OpenTelemetry?"},- },- })- if err != nil {- return err- }-- for _, block := range resp.Content {- if block.Type == "text" {- fmt.Println(block.Text)- }- }- ```-- **Streaming:**- ```go- stream, err := client.CreateMessageStream(ctx, anthropic.MessageRequest{- Model: "claude-opus-4-5",- MaxTokens: 1024,- Messages: []anthropic.Message{- {Role: "user", Content: "Tell me a story."},- },- })- if err != nil {- return err- }- defer stream.Close()-- for {- event, err := stream.Recv()- if err == io.EOF {- break- }- if err != nil {- return err- }- if event.Type == "content_block_delta" && event.Delta != nil {- fmt.Print(event.Delta.Text)- }- }- ```-- **Tool use:**- ```go- resp, err := client.CreateMessage(ctx, anthropic.MessageRequest{- Model: "claude-opus-4-5",- MaxTokens: 512,- Messages: []anthropic.Message{- {Role: "user", Content: "What is the weather in San Francisco?"},- },- Tools: []anthropic.Tool{- {- Name: "get_weather",- Description: "Get the current weather in a location",- InputSchema: map[string]interface{}{- "type": "object",- "properties": map[string]interface{}{- "location": map[string]interface{}{- "type": "string",- "description": "The city and state",- },- },- "required": []string{"location"},- },- },- },- })- ```- </Step>-</Steps>--## What gets collected--Every call to the instrumented client automatically records:--| Data | Attribute |-|------|-----------|-| Operation name | `gen_ai.operation.name` |-| Model requested | `gen_ai.request.model` |-| Model used | `gen_ai.response.model` |-| Response ID | `gen_ai.response.id` |-| Input tokens | `gen_ai.usage.input_tokens` |-| Output tokens | `gen_ai.usage.output_tokens` |-| Cache creation tokens | `gen_ai.usage.prompt_tokens_details.cache_write` |-| Cache read tokens | `gen_ai.usage.prompt_tokens_details.cache_read` |-| Estimated cost | `gen_ai.usage.cost` |-| Finish reason | `gen_ai.response.finish_reasons` |-| Tool calls | `gen_ai.tool.name`, `gen_ai.tool.call.id` |-| Time to first token | `gen_ai.server.time_to_first_token` (streaming) |-| Time per output token | `gen_ai.server.time_per_output_token` (streaming) |--Metrics emitted:-- `gen_ai.client.token.usage` — token usage histogram (input/output)-- `gen_ai.client.operation.duration` — total operation duration-- `gen_ai.server.time_to_first_token` — TTFT for streaming-- `gen_ai.client.operation.time_to_first_chunk` — client-side TTFT-- `gen_ai.client.operation.time_per_output_chunk` — per-chunk latency-- `gen_ai.server.request.duration` — estimated server processing time-------<CardGroup cols={3}>- <Card title="Go SDK Overview" href="/latest/sdk/go-overview" icon="golang">- Full configuration reference and setup guide- </Card>- <Card title="Configuration" href="/latest/sdk/configuration" icon="bolt">- Configure the OpenLIT SDK according to your requirements- </Card>- <Card title="Destinations" href="/latest/sdk/destinations/overview" icon="link">- Send telemetry to Datadog, Grafana, New Relic, and other observability stacks- </Card>-</CardGroup>diff --git a/docs/latest/sdk/integrations/go-openai.mdx b/docs/latest/sdk/integrations/go-openai.mdxdeleted file mode 100644index f4a638c9..00000000--- a/docs/latest/sdk/integrations/go-openai.mdx+++ /dev/null@@ -1,204 +0,0 @@-----title: 'Monitor OpenAI using OpenTelemetry (Go)'-sidebarTitle: 'OpenAI (Go)'------OpenLIT uses OpenTelemetry instrumentation to help you monitor Go applications built with OpenAI models. This includes tracking performance, token usage, costs, and how users interact with the application.--The Go SDK wraps your OpenAI client with an `InstrumentedClient` that automatically emits traces and metrics for every API call — with zero changes to your application logic.--The integration supports:-- Chat completions (standard and streaming)-- Embeddings-- Image generation (`dall-e-2`, `dall-e-3`)--## Get started--<Steps>- <Step title="Install the Go SDK">- Open your terminal and run:-- <Tabs>- <Tab title="Latest Version">- ```shell- go get github.com/openlit/openlit/sdk/go- ```- </Tab>- <Tab title="Specific Version">- ```shell- go get github.com/openlit/openlit/sdk/[email protected]- ```- Replace `v1.2.3` with the version you want to install.- </Tab>- </Tabs>- </Step>-- <Step title="Initialize OpenLIT">- Add this once at the start of your application (e.g. in `main()`):-- <Tabs>- <Tab title="Via Config Struct">- ```go- import (- "context"- openlit "github.com/openlit/openlit/sdk/go"- )-- if err := openlit.Init(openlit.Config{- OtlpEndpoint: "YOUR_OTEL_ENDPOINT",- ApplicationName: "my-ai-app",- Environment: "production",- }); err != nil {- log.Fatal(err)- }- defer openlit.Shutdown(context.Background())- ```- </Tab>- <Tab title="Via Environment Variable">- ```go- import (- "context"- openlit "github.com/openlit/openlit/sdk/go"- )-- if err := openlit.Init(openlit.Config{- ApplicationName: "my-ai-app",- }); err != nil {- log.Fatal(err)- }- defer openlit.Shutdown(context.Background())- ```-- Then set your OTLP endpoint via environment variable:- ```shell- export OTEL_EXPORTER_OTLP_ENDPOINT=YOUR_OTEL_ENDPOINT- ```- </Tab>- </Tabs>-- **Replace** `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` for a local OpenLIT deployment.- </Step>-- <Step title="Create an instrumented client">- Replace your existing OpenAI client creation with the OpenLIT instrumented client:-- ```go- import "github.com/openlit/openlit/sdk/go/instrumentation/openai"-- client := openai.NewClient("your-openai-api-key")- ```-- Optional configuration:- ```go- // Custom base URL (e.g. for Azure OpenAI or local models)- client := openai.NewClient("your-api-key",- openai.WithBaseURL("https://your-custom-endpoint/v1"),- )- ```- </Step>-- <Step title="Use the client">- Use the instrumented client exactly as you would a normal OpenAI client:-- **Chat completion:**- ```go- resp, err := client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{- Model: "gpt-4o",- Messages: []openai.Message{- {Role: "system", Content: "You are a helpful assistant."},- {Role: "user", Content: "What is OpenTelemetry?"},- },- MaxTokens: 256,- Temperature: 0.7,- })- if err != nil {- return err- }- fmt.Println(resp.Choices[0].Message.Content)- ```-- **Streaming:**- ```go- stream, err := client.CreateChatCompletionStream(ctx, openai.ChatCompletionRequest{- Model: "gpt-4o",- Messages: []openai.Message{- {Role: "user", Content: "Tell me a story."},- },- })- if err != nil {- return err- }- defer stream.Close()-- for {- chunk, err := stream.Recv()- if err == io.EOF {- break- }- if err != nil {- return err- }- if len(chunk.Choices) > 0 {- fmt.Print(chunk.Choices[0].Delta.Content)- }- }- ```-- **Embeddings:**- ```go- resp, err := client.CreateEmbedding(ctx, openai.EmbeddingRequest{- Model: "text-embedding-3-small",- Input: "The quick brown fox",- })- ```-- **Image generation:**- ```go- resp, err := client.CreateImage(ctx, openai.ImageRequest{- Model: "dall-e-3",- Prompt: "A futuristic city skyline at sunset",- N: 1,- Size: "1024x1024",- })- ```- </Step>-</Steps>--## What gets collected--Every call to the instrumented client automatically records:--| Data | Attribute |-|------|-----------|-| Operation name | `gen_ai.operation.name` |-| Model requested | `gen_ai.request.model` |-| Model used | `gen_ai.response.model` |-| Input tokens | `gen_ai.usage.input_tokens` |-| Output tokens | `gen_ai.usage.output_tokens` |-| Estimated cost | `gen_ai.usage.cost` |-| Finish reason | `gen_ai.response.finish_reasons` |-| Tool calls | `gen_ai.tool.name`, `gen_ai.tool.call.id` |-| Time to first token | `gen_ai.server.time_to_first_token` (streaming) |-| Time per output token | `gen_ai.server.time_per_output_token` (streaming) |--Metrics emitted:-- `gen_ai.client.token.usage` — token usage histogram (input/output)-- `gen_ai.client.operation.duration` — total operation duration-- `gen_ai.server.time_to_first_token` — TTFT for streaming-- `gen_ai.client.operation.time_to_first_chunk` — client-side TTFT-- `gen_ai.client.operation.time_per_output_chunk` — per-chunk latency-- `gen_ai.server.request.duration` — estimated server processing time-------<CardGroup cols={3}>- <Card title="Go SDK Overview" href="/latest/sdk/go-overview" icon="golang">- Full configuration reference and setup guide- </Card>- <Card title="Configuration" href="/latest/sdk/configuration" icon="bolt">- Configure the OpenLIT SDK according to your requirements- </Card>- <Card title="Destinations" href="/latest/sdk/destinations/overview" icon="link">- Send telemetry to Datadog, Grafana, New Relic, and other observability stacks- </Card>-</CardGroup>diff --git a/docs/latest/sdk/integrations/google-adk.mdx b/docs/latest/sdk/integrations/google-adk.mdxindex 4e1208af..bcd29c54 100644--- a/docs/latest/sdk/integrations/google-adk.mdx+++ b/docs/latest/sdk/integrations/google-adk.mdx@@ -1,5 +1,6 @@---title: 'Monitor Google ADK using OpenTelemetry'+description: 'Monitor AI agents built with Google Agent Development Kit using OpenTelemetry auto-instrumentation for AI observability into execution and tool calls.'sidebarTitle: 'Google ADK'---@@ -47,12 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Running in Kubernetes? Try the OpenLIT Operator"- icon={<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">- <title>Kubernetes</title>- <path fill="#FFA500" d="M10.204 14.35l.007.01-.999 2.413a5.171 5.171 0 0 1-2.075-2.597l2.578-.437.004.005a.44.44 0 0 1 .484.606zm-.833-2.129a.44.44 0 0 0 .173-.756l.002-.011L7.585 9.7a5.143 5.143 0 0 0-.73 3.255l2.514-.725.002-.009zm1.145-1.98a.44.44 0 0 0 .699-.337l.01-.005.15-2.62a5.144 5.144 0 0 0-3.01 1.442l2.147 1.523.004-.002zm.76 2.75l.723.349.722-.347.18-.78-.5-.623h-.804l-.5.623.179.779zm1.5-3.095a.44.44 0 0 0 .7.336l.008.003 2.134-1.513a5.188 5.188 0 0 0-2.992-1.442l.148 2.615.002.001zm10.876 5.97l-5.773 7.181a1.6 1.6 0 0 1-1.248.594l-9.261.003a1.6 1.6 0 0 1-1.247-.596l-5.776-7.18a1.583 1.583 0 0 1-.307-1.34L2.1 5.573c.108-.47.425-.864.863-1.073L11.305.513a1.606 1.606 0 0 1 1.385 0l8.345 3.985c.438.209.755.604.863 1.073l2.062 8.955c.108.47-.005.963-.308 1.34zm-3.289-2.057c-.042-.01-.103-.026-.145-.034-.174-.033-.315-.025-.479-.038-.35-.037-.638-.067-.895-.148-.105-.04-.18-.165-.216-.216l-.201-.059a6.45 6.45 0 0 0-.105-2.332 6.465 6.465 0 0 0-.936-2.163c.052-.047.15-.133.177-.159.008-.09.001-.183.094-.282.197-.185.444-.338.743-.522.142-.084.273-.137.415-.242.032-.024.076-.062.11-.089.24-.191.295-.52.123-.736-.172-.216-.506-.236-.745-.045-.034.027-.08.062-.111.088-.134.116-.217.23-.33.35-.246.25-.45.458-.673.609-.097.056-.239.037-.303.033l-.19.135a6.545 6.545 0 0 0-4.146-2.003l-.012-.223c-.065-.062-.143-.115-.163-.25-.022-.268.015-.557.057-.905.023-.163.061-.298.068-.475.001-.04-.001-.099-.001-.142 0-.306-.224-.555-.5-.555-.275 0-.499.249-.499.555l.001.014c0 .041-.002.092 0 .128.006.177.044.312.067.475.042.348.078.637.056.906a.545.545 0 0 1-.162.258l-.012.211a6.424 6.424 0 0 0-4.166 2.003 8.373 8.373 0 0 1-.18-.128c-.09.012-.18.04-.297-.029-.223-.15-.427-.358-.673-.608-.113-.12-.195-.234-.329-.349-.03-.026-.077-.062-.111-.088a.594.594 0 0 0-.348-.132.481.481 0 0 0-.398.176c-.172.216-.117.546.123.737l.007.005.104.083c.142.105.272.159.414.242.299.185.546.338.743.522.076.082.09.226.1.288l.16.143a6.462 6.462 0 0 0-1.02 4.506l-.208.06c-.055.072-.133.184-.215.217-.257.081-.546.11-.895.147-.164.014-.305.006-.48.039-.037.007-.09.02-.133.03l-.004.002-.007.002c-.295.071-.484.342-.423.608.061.267.349.429.645.365l.007-.001.01-.003.129-.029c.17-.046.294-.113.448-.172.33-.118.604-.217.87-.256.112-.009.23.069.288.101l.217-.037a6.5 6.5 0 0 0 2.88 3.596l-.09.218c.033.084.069.199.044.282-.097.252-.263.517-.452.813-.091.136-.185.242-.268.399-.02.037-.045.095-.064.134-.128.275-.034.591.213.71.248.12.556-.007.69-.282v-.002c.02-.039.046-.09.062-.127.07-.162.094-.301.144-.458.132-.332.205-.68.387-.897.05-.06.13-.082.215-.105l.113-.205a6.453 6.453 0 0 0 4.609.012l.106.192c.086.028.18.042.256.155.136.232.229.507.342.84.05.156.074.295.145.457.016.037.043.09.062.129.133.276.442.402.69.282.247-.118.341-.435.213-.71-.02-.039-.045-.096-.065-.134-.083-.156-.177-.261-.268-.398-.19-.296-.346-.541-.443-.793-.04-.13.007-.21.038-.294-.018-.022-.059-.144-.083-.202a6.499 6.499 0 0 0 2.88-3.622c.064.01.176.03.213.038.075-.05.144-.114.28-.104.266.039.54.138.87.256.154.06.277.128.448.173.036.01.088.019.13.028l.009.003.007.001c.297.064.584-.098.645-.365.06-.266-.128-.537-.423-.608zM16.4 9.701l-1.95 1.746v.005a.44.44 0 0 0 .173.757l.003.01 2.526.728a5.199 5.199 0 0 0-.108-1.674A5.208 5.208 0 0 0 16.4 9.7zm-4.013 5.325a.437.437 0 0 0-.404-.232.44.44 0 0 0-.372.233h-.002l-1.268 2.292a5.164 5.164 0 0 0 3.326.003l-1.27-2.296h-.01zm1.888-1.293a.44.44 0 0 0-.27.036.44.44 0 0 0-.214.572l-.003.004 1.01 2.438a5.15 5.15 0 0 0 2.081-2.615l-2.6-.44-.004.005z"/></svg>}- href="/latest/operator/overview"- >- Automatically inject instrumentation into existing workloads without modifying pod specs, container images, or application code.- </Card>diff --git a/docs/latest/sdk/integrations/google-ai-studio.mdx b/docs/latest/sdk/integrations/google-ai-studio.mdxindex dab3bfd0..2db63f82 100644--- a/docs/latest/sdk/integrations/google-ai-studio.mdx+++ b/docs/latest/sdk/integrations/google-ai-studio.mdx@@ -1,5 +1,6 @@---title: 'Monitor Google AI Studio using OpenTelemetry'+description: "Track LLM apps built with Google AI Studio's Gemini models via OpenTelemetry auto-instrumentation, covering AI observability, tokens, and cost tracking."sidebarTitle: 'Google AI Studio'---@@ -48,10 +49,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/gpt4all.mdx b/docs/latest/sdk/integrations/gpt4all.mdxindex 527acef8..d595f8b2 100644--- a/docs/latest/sdk/integrations/gpt4all.mdx+++ b/docs/latest/sdk/integrations/gpt4all.mdx@@ -1,5 +1,6 @@---title: 'Monitor GPT4All using OpenTelemetry'+description: 'Add AI observability to LLM apps running local GPT4All models via OpenTelemetry auto-instrumentation, tracking performance and token usage.'sidebarTitle: 'GPT4All'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/groq.mdx b/docs/latest/sdk/integrations/groq.mdxindex bae4e380..9569077a 100644--- a/docs/latest/sdk/integrations/groq.mdx+++ b/docs/latest/sdk/integrations/groq.mdx@@ -1,5 +1,6 @@---title: 'Monitor Groq using OpenTelemetry'+description: "Gain AI observability into LLM apps powered by Groq's fast inference API with OpenTelemetry auto-instrumentation, tracking tokens and cost."sidebarTitle: 'Groq'---@@ -48,10 +49,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/guardrails.mdx b/docs/latest/sdk/integrations/guardrails.mdxindex 24786876..b6a51ec7 100644--- a/docs/latest/sdk/integrations/guardrails.mdx+++ b/docs/latest/sdk/integrations/guardrails.mdx@@ -1,5 +1,6 @@---title: 'Monitor Guardrails AI using OpenTelemetry'+description: 'Apply OpenTelemetry auto-instrumentation for AI evaluation of LLM apps validated with the Guardrails AI library, tracking performance and operation stats.'sidebarTitle: 'Guardrails AI'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/haystack.mdx b/docs/latest/sdk/integrations/haystack.mdxindex 5ced4815..bed1c244 100644--- a/docs/latest/sdk/integrations/haystack.mdx+++ b/docs/latest/sdk/integrations/haystack.mdx@@ -1,5 +1,6 @@---title: 'Monitor Haystack using OpenTelemetry'+description: 'Bring OpenTelemetry tracing to Haystack retrieval-augmented pipelines, giving AI observability into component execution and operation stats.'sidebarTitle: 'Haystack'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/huggingface.mdx b/docs/latest/sdk/integrations/huggingface.mdxindex 93c43fdc..52ca1a9a 100644--- a/docs/latest/sdk/integrations/huggingface.mdx+++ b/docs/latest/sdk/integrations/huggingface.mdx@@ -1,5 +1,6 @@---title: 'Monitor HuggingFace using OpenTelemetry'+description: 'Instrument Hugging Face Transformers, the Inference API, and Transformers.js with OpenTelemetry for AI observability into tokens, cost, and inference calls.'sidebarTitle: 'HuggingFace'---@@ -51,7 +52,7 @@ const output = await generator("OpenTelemetry makes observability", {|--------|----------|| `captureMessageContent` | Gates `gen_ai.input.messages` / `gen_ai.output.messages` on spans and in events || `disableEvents` | Suppresses all inference events |-| `disableMetrics` | Global — metrics instruments are not set up |+| `disableMetrics` | Global - metrics instruments are not set up || `disabledInstrumentors: ['transformers']` | Disables the local Transformers.js instrumentation || `disabledInstrumentors: ['huggingface']` | Disables the remote Inference API instrumentation |@@ -98,10 +99,3 @@ On the Python SDK, inference events are only emitted when `capture_message_conteSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/intel-gpu.mdx b/docs/latest/sdk/integrations/intel-gpu.mdxindex cafa7a7d..5c0519c3 100644--- a/docs/latest/sdk/integrations/intel-gpu.mdx+++ b/docs/latest/sdk/integrations/intel-gpu.mdx@@ -1,5 +1,6 @@---title: 'Monitor Intel GPUs using OpenTelemetry'+description: 'Monitor Intel GPU hardware with OpenTelemetry, capturing temperature, power draw, energy use, and clock frequency via the Linux i915/Xe kernel driver.'sidebarTitle: 'Intel GPUs'---@@ -8,7 +9,7 @@ import IntegrationMethods from '/snippets/integration-methods.mdx';OpenLIT uses OpenTelemetry to help you monitor Intel GPUs. This includes tracking GPU metrics like temperature, power consumption, energy usage, and clock frequency via the Linux i915/Xe kernel driver.<Note>-Intel GPU support requires Linux with the **i915** or **Xe** kernel driver (kernel 5.10+). Metrics are read directly from the kernel's sysfs/hwmon interface — no additional software or libraries are required. Utilization and memory metrics are not available via this interface; use [Intel GPU Top](https://github.com/intel/igt-gpu-tools) or [XPUManager](https://github.com/intel/xpumanager) for those.+Intel GPU support requires Linux with the **i915** or **Xe** kernel driver (kernel 5.10+). Metrics are read directly from the kernel's sysfs/hwmon interface - no additional software or libraries are required. Utilization and memory metrics are not available via this interface; use [Intel GPU Top](https://github.com/intel/igt-gpu-tools) or [XPUManager](https://github.com/intel/xpumanager) for those.</Note>## Collected Metrics@@ -118,7 +119,7 @@ Intel GPU support requires Linux with the **i915** or **Xe** kernel driver (kern</Accordion></AccordionGroup>-<Card title="Collected Metrics" href="/latest/openlit/observability/metrics#gpu-metrics" icon='table'>+<Card title="Collected Metrics" href="/latest/openlit/observability/telemetry#metrics" icon='table'>Details on the types of metrics collected and their descriptions.</Card>diff --git a/docs/latest/sdk/integrations/julep-ai.mdx b/docs/latest/sdk/integrations/julep-ai.mdxindex fb99b515..65fe5d8e 100644--- a/docs/latest/sdk/integrations/julep-ai.mdx+++ b/docs/latest/sdk/integrations/julep-ai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Julep AI using OpenTelemetry'+description: 'Enable AI observability for agents built with Julep AI through OpenTelemetry auto-instrumentation, tracking performance and operation stats automatically.'sidebarTitle: 'Julep AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/krutrim.mdx b/docs/latest/sdk/integrations/krutrim.mdxindex 1a9c6184..251d5156 100644--- a/docs/latest/sdk/integrations/krutrim.mdx+++ b/docs/latest/sdk/integrations/krutrim.mdx@@ -1,5 +1,6 @@---title: 'Monitor Krutrim using OpenTelemetry'+description: 'Track LLM apps built on OLA Krutrim models, accessed via the OpenAI SDK, using OpenTelemetry auto-instrumentation for AI observability and cost tracking.'sidebarTitle: 'OLA Krutrim'---@@ -37,10 +38,3 @@ As Krutrim uses the OpenAI SDK, This integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/langchain.mdx b/docs/latest/sdk/integrations/langchain.mdxindex 6aad770c..a16ea623 100644--- a/docs/latest/sdk/integrations/langchain.mdx+++ b/docs/latest/sdk/integrations/langchain.mdx@@ -1,5 +1,6 @@---title: 'Monitor LangChain using OpenTelemetry'+description: 'Instrument LangChain apps built on langchain-core with OpenTelemetry auto-instrumentation, delivering AI observability into chains and execution performance.'sidebarTitle: 'LangChain'---@@ -67,10 +68,3 @@ Common installation patterns:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/langgraph.mdx b/docs/latest/sdk/integrations/langgraph.mdxindex a59aa1b5..dcd755fa 100644--- a/docs/latest/sdk/integrations/langgraph.mdx+++ b/docs/latest/sdk/integrations/langgraph.mdx@@ -1,5 +1,6 @@---title: 'Monitor LangGraph using OpenTelemetry'+description: 'Trace LangGraph agent workflows and graph execution with OpenTelemetry auto-instrumentation for AI observability into tool usage and state transitions.'sidebarTitle: 'LangGraph'---@@ -47,12 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Running in Kubernetes? Try the OpenLIT Operator"- icon={<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">- <title>Kubernetes</title>- <path fill="#FFA500" d="M10.204 14.35l.007.01-.999 2.413a5.171 5.171 0 0 1-2.075-2.597l2.578-.437.004.005a.44.44 0 0 1 .484.606zm-.833-2.129a.44.44 0 0 0 .173-.756l.002-.011L7.585 9.7a5.143 5.143 0 0 0-.73 3.255l2.514-.725.002-.009zm1.145-1.98a.44.44 0 0 0 .699-.337l.01-.005.15-2.62a5.144 5.144 0 0 0-3.01 1.442l2.147 1.523.004-.002zm.76 2.75l.723.349.722-.347.18-.78-.5-.623h-.804l-.5.623.179.779zm1.5-3.095a.44.44 0 0 0 .7.336l.008.003 2.134-1.513a5.188 5.188 0 0 0-2.992-1.442l.148 2.615.002.001zm10.876 5.97l-5.773 7.181a1.6 1.6 0 0 1-1.248.594l-9.261.003a1.6 1.6 0 0 1-1.247-.596l-5.776-7.18a1.583 1.583 0 0 1-.307-1.34L2.1 5.573c.108-.47.425-.864.863-1.073L11.305.513a1.606 1.606 0 0 1 1.385 0l8.345 3.985c.438.209.755.604.863 1.073l2.062 8.955c.108.47-.005.963-.308 1.34zm-3.289-2.057c-.042-.01-.103-.026-.145-.034-.174-.033-.315-.025-.479-.038-.35-.037-.638-.067-.895-.148-.105-.04-.18-.165-.216-.216l-.201-.059a6.45 6.45 0 0 0-.105-2.332 6.465 6.465 0 0 0-.936-2.163c.052-.047.15-.133.177-.159.008-.09.001-.183.094-.282.197-.185.444-.338.743-.522.142-.084.273-.137.415-.242.032-.024.076-.062.11-.089.24-.191.295-.52.123-.736-.172-.216-.506-.236-.745-.045-.034.027-.08.062-.111.088-.134.116-.217.23-.33.35-.246.25-.45.458-.673.609-.097.056-.239.037-.303.033l-.19.135a6.545 6.545 0 0 0-4.146-2.003l-.012-.223c-.065-.062-.143-.115-.163-.25-.022-.268.015-.557.057-.905.023-.163.061-.298.068-.475.001-.04-.001-.099-.001-.142 0-.306-.224-.555-.5-.555-.275 0-.499.249-.499.555l.001.014c0 .041-.002.092 0 .128.006.177.044.312.067.475.042.348.078.637.056.906a.545.545 0 0 1-.162.258l-.012.211a6.424 6.424 0 0 0-4.166 2.003 8.373 8.373 0 0 1-.18-.128c-.09.012-.18.04-.297-.029-.223-.15-.427-.358-.673-.608-.113-.12-.195-.234-.329-.349-.03-.026-.077-.062-.111-.088a.594.594 0 0 0-.348-.132.481.481 0 0 0-.398.176c-.172.216-.117.546.123.737l.007.005.104.083c.142.105.272.159.414.242.299.185.546.338.743.522.076.082.09.226.1.288l.16.143a6.462 6.462 0 0 0-1.02 4.506l-.208.06c-.055.072-.133.184-.215.217-.257.081-.546.11-.895.147-.164.014-.305.006-.48.039-.037.007-.09.02-.133.03l-.004.002-.007.002c-.295.071-.484.342-.423.608.061.267.349.429.645.365l.007-.001.01-.003.129-.029c.17-.046.294-.113.448-.172.33-.118.604-.217.87-.256.112-.009.23.069.288.101l.217-.037a6.5 6.5 0 0 0 2.88 3.596l-.09.218c.033.084.069.199.044.282-.097.252-.263.517-.452.813-.091.136-.185.242-.268.399-.02.037-.045.095-.064.134-.128.275-.034.591.213.71.248.12.556-.007.69-.282v-.002c.02-.039.046-.09.062-.127.07-.162.094-.301.144-.458.132-.332.205-.68.387-.897.05-.06.13-.082.215-.105l.113-.205a6.453 6.453 0 0 0 4.609.012l.106.192c.086.028.18.042.256.155.136.232.229.507.342.84.05.156.074.295.145.457.016.037.043.09.062.129.133.276.442.402.69.282.247-.118.341-.435.213-.71-.02-.039-.045-.096-.065-.134-.083-.156-.177-.261-.268-.398-.19-.296-.346-.541-.443-.793-.04-.13.007-.21.038-.294-.018-.022-.059-.144-.083-.202a6.499 6.499 0 0 0 2.88-3.622c.064.01.176.03.213.038.075-.05.144-.114.28-.104.266.039.54.138.87.256.154.06.277.128.448.173.036.01.088.019.13.028l.009.003.007.001c.297.064.584-.098.645-.365.06-.266-.128-.537-.423-.608zM16.4 9.701l-1.95 1.746v.005a.44.44 0 0 0 .173.757l.003.01 2.526.728a5.199 5.199 0 0 0-.108-1.674A5.208 5.208 0 0 0 16.4 9.7zm-4.013 5.325a.437.437 0 0 0-.404-.232.44.44 0 0 0-.372.233h-.002l-1.268 2.292a5.164 5.164 0 0 0 3.326.003l-1.27-2.296h-.01zm1.888-1.293a.44.44 0 0 0-.27.036.44.44 0 0 0-.214.572l-.003.004 1.01 2.438a5.15 5.15 0 0 0 2.081-2.615l-2.6-.44-.004.005z"/></svg>}- href="/latest/operator/overview"- >- Automatically inject instrumentation into existing workloads without modifying pod specs, container images, or application code.- </Card>diff --git a/docs/latest/sdk/integrations/letta.mdx b/docs/latest/sdk/integrations/letta.mdxindex 3957da3e..054702d7 100644--- a/docs/latest/sdk/integrations/letta.mdx+++ b/docs/latest/sdk/integrations/letta.mdx@@ -1,5 +1,6 @@---title: 'Monitor Letta using OpenTelemetry'+description: 'Monitor AI agents built with Letta using OpenTelemetry auto-instrumentation, capturing AI observability data on performance and operation stats.'sidebarTitle: 'Letta'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/litellm.mdx b/docs/latest/sdk/integrations/litellm.mdxindex 8733a114..24368df1 100644--- a/docs/latest/sdk/integrations/litellm.mdx+++ b/docs/latest/sdk/integrations/litellm.mdx@@ -1,5 +1,6 @@---title: 'Monitor LiteLLM using OpenTelemetry'+description: "Get AI observability into LLM apps routed through LiteLLM's unified gateway via OpenTelemetry auto-instrumentation, tracking performance and operation stats."sidebarTitle: 'LiteLLM'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/llama-index.mdx b/docs/latest/sdk/integrations/llama-index.mdxindex 46829530..322d5a85 100644--- a/docs/latest/sdk/integrations/llama-index.mdx+++ b/docs/latest/sdk/integrations/llama-index.mdx@@ -1,5 +1,6 @@---title: 'Monitor LlamaIndex using OpenTelemetry'+description: 'Add OpenTelemetry auto-instrumentation to LlamaIndex retrieval and query pipelines for AI observability into indexing and query engine stats.'sidebarTitle: 'LlamaIndex'---@@ -46,10 +47,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/mcp.mdx b/docs/latest/sdk/integrations/mcp.mdxnew file mode 100644index 00000000..1aabc68c--- /dev/null+++ b/docs/latest/sdk/integrations/mcp.mdx@@ -0,0 +1,39 @@+---+title: 'Monitor MCP using OpenTelemetry'+description: 'OpenLIT auto-instruments Model Context Protocol (MCP) servers and clients via OpenTelemetry, tracing JSON-RPC calls, tool execution, and session lifecycle.'+sidebarTitle: 'MCP'+---++import IntegrationMethods from '/snippets/integration-methods.mdx';++OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor applications built on the Model Context Protocol (MCP). This includes tracing MCP server and client JSON-RPC calls, tool execution, and session lifecycle events, giving you AI observability into how your agents discover and call MCP tools.++Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.++The integration is compatible with the `mcp` Python SDK `>= 0.1.0`.++## Get started++<Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ ```shell+ pip install openlit+ ```+ </Step>+ <IntegrationMethods />+</Steps>++---++<CardGroup cols={3}>+ <Card title="Quickstart: MCP Observability" href="/latest/openlit/quickstart-mcp-observability" icon='bolt'>+ Production-ready MCP monitoring setup in 2 simple steps with zero code changes+ </Card>+ <Card title="Configuration" href="/latest/sdk/configuration" icon='bolt'>+ Configure the OpenLIT SDK according to you requirements.+ </Card>+ <Card title="Destinations" href="/latest/sdk/destinations/overview" icon='link'>+ Send telemetry to Datadog, Grafana, New Relic, and other observability stacks+ </Card>+</CardGroup>diff --git a/docs/latest/sdk/integrations/mem0.mdx b/docs/latest/sdk/integrations/mem0.mdxindex 4e40d729..2062aa6c 100644--- a/docs/latest/sdk/integrations/mem0.mdx+++ b/docs/latest/sdk/integrations/mem0.mdx@@ -1,5 +1,6 @@---title: 'Monitor mem0 using OpenTelemetry'+description: 'Track the mem0 memory layer for LLM apps with OpenTelemetry auto-instrumentation, providing AI observability into memory read/write performance.'sidebarTitle: 'mem0'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/milvus.mdx b/docs/latest/sdk/integrations/milvus.mdxindex 6706c53b..7972b61b 100644--- a/docs/latest/sdk/integrations/milvus.mdx+++ b/docs/latest/sdk/integrations/milvus.mdx@@ -1,5 +1,6 @@---title: 'Monitor Milvus using OpenTelemetry'+description: 'Monitor Milvus vector database operations behind LLM apps using OpenTelemetry auto-instrumentation, gaining AI observability into query performance.'sidebarTitle: 'Milvus'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/mistral.mdx b/docs/latest/sdk/integrations/mistral.mdxindex 585ad471..9e186ea8 100644--- a/docs/latest/sdk/integrations/mistral.mdx+++ b/docs/latest/sdk/integrations/mistral.mdx@@ -1,5 +1,6 @@---title: 'Monitor Mistral AI using OpenTelemetry'+description: 'Track LLM apps built on Mistral AI models with OpenTelemetry auto-instrumentation for AI observability, covering tokens, cost, and performance.'sidebarTitle: 'Mistral'---@@ -48,10 +49,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/multion.mdx b/docs/latest/sdk/integrations/multion.mdxindex 03b25f5d..f92d0765 100644--- a/docs/latest/sdk/integrations/multion.mdx+++ b/docs/latest/sdk/integrations/multion.mdx@@ -1,5 +1,6 @@---title: 'Monitor MultiOn using OpenTelemetry'+description: 'Monitor autonomous browser agents built with MultiOn using OpenTelemetry auto-instrumentation, delivering AI observability into agent operation stats.'sidebarTitle: 'MultiOn'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/nvidia-gpu.mdx b/docs/latest/sdk/integrations/nvidia-gpu.mdxindex f7b710dc..0af44ff0 100644--- a/docs/latest/sdk/integrations/nvidia-gpu.mdx+++ b/docs/latest/sdk/integrations/nvidia-gpu.mdx@@ -1,5 +1,6 @@---title: 'Monitor NVIDIA GPUs using OpenTelemetry'+description: 'Monitor NVIDIA GPU hardware powering AI workloads with OpenTelemetry, tracking utilization, temperature, memory, and power for AI observability.'sidebarTitle: 'NVIDIA GPUs'---@@ -104,7 +105,7 @@ OpenLIT uses OpenTelemetry to help you monitor NVIDIA GPUs. This includes tracki</Accordion></AccordionGroup>-<Card title="Collected Metrics" href="/latest/openlit/observability/metrics#gpu-metrics" icon='table'>+<Card title="Collected Metrics" href="/latest/openlit/observability/telemetry#metrics" icon='table'>Details on the types of metrics collected and their descriptions.</Card>@@ -121,10 +122,3 @@ Details on the types of metrics collected and their descriptions.Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/nvidia-nim.mdx b/docs/latest/sdk/integrations/nvidia-nim.mdxindex 21834a44..63d3f7fe 100644--- a/docs/latest/sdk/integrations/nvidia-nim.mdx+++ b/docs/latest/sdk/integrations/nvidia-nim.mdx@@ -1,5 +1,6 @@---title: 'Monitor NVIDIA NIM using OpenTelemetry'+description: 'OpenLIT auto-instruments NVIDIA NIM LLM applications with OpenTelemetry, delivering AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'NVIDIA NIM'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/oci.mdx b/docs/latest/sdk/integrations/oci.mdxindex 16eee326..8348c563 100644--- a/docs/latest/sdk/integrations/oci.mdx+++ b/docs/latest/sdk/integrations/oci.mdx@@ -1,5 +1,6 @@---title: 'Monitor OCI Generative AI using OpenTelemetry'+description: 'OpenLIT brings AI observability to Oracle Cloud Infrastructure Generative AI via OpenTelemetry, tracing OCI SDK and LangChain chat, text, and embedding calls.'sidebarTitle: 'OCI GenAI'---@@ -43,10 +44,3 @@ It instruments the `GenerativeAiInferenceClient` `chat`, `generate_text`, and `eSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/ollama.mdx b/docs/latest/sdk/integrations/ollama.mdxindex 4eb40416..a5770956 100644--- a/docs/latest/sdk/integrations/ollama.mdx+++ b/docs/latest/sdk/integrations/ollama.mdx@@ -1,5 +1,6 @@---title: 'Monitor Ollama using OpenTelemetry'+description: 'OpenLIT brings AI observability to Ollama LLM apps via OpenTelemetry auto-instrumentation, tracing performance and usage across Python and TypeScript SDKs.'sidebarTitle: 'Ollama'---@@ -48,10 +49,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/openai-agents.mdx b/docs/latest/sdk/integrations/openai-agents.mdxindex ed55524e..ef6ba8b0 100644--- a/docs/latest/sdk/integrations/openai-agents.mdx+++ b/docs/latest/sdk/integrations/openai-agents.mdx@@ -1,5 +1,6 @@---title: 'Monitor OpenAI Agents'+description: 'Trace AI agents built on the OpenAI Agents SDK with OpenTelemetry auto-instrumentation for AI observability into agent performance and operation stats.'sidebarTitle: 'OpenAI Agents'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/openai.mdx b/docs/latest/sdk/integrations/openai.mdxindex 740dbead..3c1d93f3 100644--- a/docs/latest/sdk/integrations/openai.mdx+++ b/docs/latest/sdk/integrations/openai.mdx@@ -1,38 +1,228 @@---title: 'Monitor OpenAI using OpenTelemetry'+description: 'Monitor OpenAI LLM applications with OpenTelemetry auto-instrumentation, giving AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'OpenAI'---import IntegrationMethods from '/snippets/integration-methods.mdx';-OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor LLM applications built using OpenAI models. This includes tracking performance, token usage, costs, and how users interact with the application.+OpenLIT uses OpenTelemetry instrumentation to help you monitor LLM applications built using OpenAI models. This includes tracking performance, token usage, costs, and how users interact with the application.-Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.+The integration is compatible with:+- OpenAI Python SDK client `>=1.92.0`+- OpenAI Typescript SDK client `>=1.13.0`+- OpenAI Go SDK, via OpenLIT's `InstrumentedClient` wrapper-The integration is compatible with-- OpenAI Python SDK client `>=1.92.0`-- OpenAI Typescript SDK client `>=1.13.0`+## Get started-## Get started+<Tabs>+ <Tab title="Python / TypeScript">+ Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.-<Steps>- <Step title="Install OpenLIT">- Open your command line or terminal and run:- <Tabs>- <Tab title="Python">- ```shell- pip install openlit+ <Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ <Tabs>+ <Tab title="Python">+ ```shell+ pip install openlit+ ```+ </Tab>+ <Tab title="Typescript">+ ```shell+ npm install openlit+ ```+ </Tab>+ </Tabs>+ </Step>+ <IntegrationMethods />+ </Steps>+ </Tab>++ <Tab title="Go">+ The Go SDK wraps your OpenAI client with an `InstrumentedClient` that automatically emits traces and metrics for every API call - with zero changes to your application logic.++ The integration supports:+ - Chat completions (standard and streaming)+ - Embeddings+ - Image generation (`dall-e-2`, `dall-e-3`)++ <Steps>+ <Step title="Install the Go SDK">+ Open your terminal and run:++ <Tabs>+ <Tab title="Latest Version">+ ```shell+ go get github.com/openlit/openlit/sdk/go+ ```+ </Tab>+ <Tab title="Specific Version">+ ```shell+ go get github.com/openlit/openlit/sdk/[email protected]+ ```+ Replace `v1.2.3` with the version you want to install.+ </Tab>+ </Tabs>+ </Step>++ <Step title="Initialize OpenLIT">+ Add this once at the start of your application (e.g. in `main()`):++ <Tabs>+ <Tab title="Via Config Struct">+ ```go+ import (+ "context"+ openlit "github.com/openlit/openlit/sdk/go"+ )++ if err := openlit.Init(openlit.Config{+ OtlpEndpoint: "YOUR_OTEL_ENDPOINT",+ ApplicationName: "my-ai-app",+ Environment: "production",+ }); err != nil {+ log.Fatal(err)+ }+ defer openlit.Shutdown(context.Background())+ ```+ </Tab>+ <Tab title="Via Environment Variable">+ ```go+ import (+ "context"+ openlit "github.com/openlit/openlit/sdk/go"+ )++ if err := openlit.Init(openlit.Config{+ ApplicationName: "my-ai-app",+ }); err != nil {+ log.Fatal(err)+ }+ defer openlit.Shutdown(context.Background())+ ```++ Then set your OTLP endpoint via environment variable:+ ```shell+ export OTEL_EXPORTER_OTLP_ENDPOINT=YOUR_OTEL_ENDPOINT+ ```+ </Tab>+ </Tabs>++ **Replace** `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` for a local OpenLIT deployment.+ </Step>++ <Step title="Create an instrumented client">+ Replace your existing OpenAI client creation with the OpenLIT instrumented client:++ ```go+ import "github.com/openlit/openlit/sdk/go/instrumentation/openai"++ client := openai.NewClient("your-openai-api-key")+ ```++ Optional configuration:+ ```go+ // Custom base URL (e.g. for Azure OpenAI or local models)+ client := openai.NewClient("your-api-key",+ openai.WithBaseURL("https://your-custom-endpoint/v1"),+ )+ ```+ </Step>++ <Step title="Use the client">+ Use the instrumented client exactly as you would a normal OpenAI client:++ **Chat completion:**+ ```go+ resp, err := client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{+ Model: "gpt-4o",+ Messages: []openai.Message{+ {Role: "system", Content: "You are a helpful assistant."},+ {Role: "user", Content: "What is OpenTelemetry?"},+ },+ MaxTokens: 256,+ Temperature: 0.7,+ })+ if err != nil {+ return err+ }+ fmt.Println(resp.Choices[0].Message.Content)+ ```++ **Streaming:**+ ```go+ stream, err := client.CreateChatCompletionStream(ctx, openai.ChatCompletionRequest{+ Model: "gpt-4o",+ Messages: []openai.Message{+ {Role: "user", Content: "Tell me a story."},+ },+ })+ if err != nil {+ return err+ }+ defer stream.Close()++ for {+ chunk, err := stream.Recv()+ if err == io.EOF {+ break+ }+ if err != nil {+ return err+ }+ if len(chunk.Choices) > 0 {+ fmt.Print(chunk.Choices[0].Delta.Content)+ }+ }```- </Tab>- <Tab title="Typescript">- ```shell- npm install openlit++ **Embeddings:**+ ```go+ resp, err := client.CreateEmbedding(ctx, openai.EmbeddingRequest{+ Model: "text-embedding-3-small",+ Input: "The quick brown fox",+ })+ ```++ **Image generation:**+ ```go+ resp, err := client.CreateImage(ctx, openai.ImageRequest{+ Model: "dall-e-3",+ Prompt: "A futuristic city skyline at sunset",+ N: 1,+ Size: "1024x1024",+ })```- </Tab>- </Tabs>- </Step>- <IntegrationMethods />-</Steps>+ </Step>+ </Steps>++ ### What gets collected++ Every call to the instrumented client automatically records:++ | Data | Attribute |+ |------|-----------|+ | Operation name | `gen_ai.operation.name` |+ | Model requested | `gen_ai.request.model` |+ | Model used | `gen_ai.response.model` |+ | Input tokens | `gen_ai.usage.input_tokens` |+ | Output tokens | `gen_ai.usage.output_tokens` |+ | Estimated cost | `gen_ai.usage.cost` |+ | Finish reason | `gen_ai.response.finish_reasons` |+ | Tool calls | `gen_ai.tool.name`, `gen_ai.tool.call.id` |+ | Time to first token | `gen_ai.server.time_to_first_token` (streaming) |+ | Time per output token | `gen_ai.server.time_per_output_token` (streaming) |++ Metrics emitted:+ - `gen_ai.client.token.usage` - token usage histogram (input/output)+ - `gen_ai.client.operation.duration` - total operation duration+ - `gen_ai.server.time_to_first_token` - TTFT for streaming+ - `gen_ai.client.operation.time_to_first_chunk` - client-side TTFT+ - `gen_ai.client.operation.time_per_output_chunk` - per-chunk latency+ - `gen_ai.server.request.duration` - estimated server processing time+ </Tab>+</Tabs>---@@ -47,10 +237,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/overview.mdx b/docs/latest/sdk/integrations/overview.mdxindex f4caf19d..a1ed16ee 100644--- a/docs/latest/sdk/integrations/overview.mdx+++ b/docs/latest/sdk/integrations/overview.mdx@@ -3085,10 +3085,3 @@ Start integrating and monitoring your applications with OpenLIT in just one lineSend telemetry to Datadog, Grafana, New Relic, Dash0, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/phidata.mdx b/docs/latest/sdk/integrations/phidata.mdxindex 2e586e6c..e0895a45 100644--- a/docs/latest/sdk/integrations/phidata.mdx+++ b/docs/latest/sdk/integrations/phidata.mdx@@ -1,5 +1,6 @@---title: 'Monitor Phidata using OpenTelemetry'+description: 'OpenLIT adds AI observability to Phidata AI agents via OpenTelemetry auto-instrumentation, tracing agent performance and operation stats automatically.'sidebarTitle: 'Phidata'---@@ -34,10 +35,3 @@ Auto-instrumentation means you don't have to set up monitoring manually for diffSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/pinecone.mdx b/docs/latest/sdk/integrations/pinecone.mdxindex c0dbf0a4..d14baf3a 100644--- a/docs/latest/sdk/integrations/pinecone.mdx+++ b/docs/latest/sdk/integrations/pinecone.mdx@@ -1,5 +1,6 @@---title: 'Monitor Pinecone using OpenTelemetry'+description: 'Trace Pinecone vector store calls in RAG applications with OpenTelemetry auto-instrumentation for AI observability into query performance and operation stats.'sidebarTitle: 'Pinecone'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/premai.mdx b/docs/latest/sdk/integrations/premai.mdxindex f113091e..cd304f2f 100644--- a/docs/latest/sdk/integrations/premai.mdx+++ b/docs/latest/sdk/integrations/premai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Prem AI using OpenTelemetry'+description: 'Monitor Prem AI LLM applications with OpenTelemetry auto-instrumentation, enabling AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'Prem AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/psycopg.mdx b/docs/latest/sdk/integrations/psycopg.mdxindex b2df186b..c8db0a8b 100644--- a/docs/latest/sdk/integrations/psycopg.mdx+++ b/docs/latest/sdk/integrations/psycopg.mdx@@ -1,5 +1,6 @@---title: 'Monitor PostgreSQL (psycopg3) using OpenTelemetry'+description: 'OpenLIT auto-instruments psycopg3 PostgreSQL queries with OpenTelemetry, tracing query performance, transactions, and pgvector similarity search.'sidebarTitle: 'PostgreSQL (psycopg3)'---@@ -80,10 +81,3 @@ openlit.init(capture_db_parameters=True)Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/pydantic.mdx b/docs/latest/sdk/integrations/pydantic.mdxindex 9b31e610..6bbabd76 100644--- a/docs/latest/sdk/integrations/pydantic.mdx+++ b/docs/latest/sdk/integrations/pydantic.mdx@@ -1,5 +1,6 @@---title: 'Monitor Pydantic AI using OpenTelemetry'+description: 'Trace Pydantic AI agents with OpenTelemetry auto-instrumentation for AI observability into agent performance and operation stats, with zero code changes.'sidebarTitle: 'Pydantic AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/qdrant.mdx b/docs/latest/sdk/integrations/qdrant.mdxindex 2ab38b27..51fe1da4 100644--- a/docs/latest/sdk/integrations/qdrant.mdx+++ b/docs/latest/sdk/integrations/qdrant.mdx@@ -1,5 +1,6 @@---title: 'Monitor Qdrant using OpenTelemetry'+description: 'OpenLIT auto-instruments Qdrant vector database operations with OpenTelemetry, delivering AI observability into query performance and operation stats.'sidebarTitle: 'Qdrant'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/reka.mdx b/docs/latest/sdk/integrations/reka.mdxindex 75f426bc..4a5af44b 100644--- a/docs/latest/sdk/integrations/reka.mdx+++ b/docs/latest/sdk/integrations/reka.mdx@@ -1,5 +1,6 @@---title: 'Monitor Reka using OpenTelemetry'+description: 'Monitor Reka AI LLM applications through OpenTelemetry auto-instrumentation, gaining AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'Reka AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/replicate.mdx b/docs/latest/sdk/integrations/replicate.mdxindex 5ce7f56f..1ada4d08 100644--- a/docs/latest/sdk/integrations/replicate.mdx+++ b/docs/latest/sdk/integrations/replicate.mdx@@ -1,5 +1,6 @@---title: 'Monitor Replicate using OpenTelemetry'+description: 'Trace AI applications built on the Replicate API with OpenTelemetry auto-instrumentation, delivering AI observability into model usage and performance.'sidebarTitle: 'Replicate'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/sarvam.mdx b/docs/latest/sdk/integrations/sarvam.mdxindex 11c1e1b4..a9918847 100644--- a/docs/latest/sdk/integrations/sarvam.mdx+++ b/docs/latest/sdk/integrations/sarvam.mdx@@ -1,5 +1,6 @@---title: 'Monitor Sarvam AI using OpenTelemetry'+description: 'OpenLIT brings AI observability to Sarvam AI LLM applications via OpenTelemetry auto-instrumentation, tracking performance, token usage, and cost tracking.'sidebarTitle: 'Sarvam AI'---@@ -46,10 +47,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/smolagents.mdx b/docs/latest/sdk/integrations/smolagents.mdxnew file mode 100644index 00000000..93b7ce22--- /dev/null+++ b/docs/latest/sdk/integrations/smolagents.mdx@@ -0,0 +1,39 @@+---+title: 'Monitor smolagents using OpenTelemetry'+description: 'OpenLIT auto-instruments Hugging Face smolagents via OpenTelemetry, tracing agent runs, planning steps, and tool calls for AI observability.'+sidebarTitle: 'smolagents'+---++import IntegrationMethods from '/snippets/integration-methods.mdx';++OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor AI agents built with Hugging Face's smolagents framework. This includes tracing agent runs, planning steps, code and tool-calling steps, and individual tool calls, giving you AI observability into multi-step agent execution.++Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up. LLM calls made by the underlying model client (OpenAI, and others) are traced by their own dedicated instrumentors, so you get a complete trace from agent run down to individual model calls.++The integration is compatible with `smolagents >= 1.0.0`.++## Get started++<Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ ```shell+ pip install openlit+ ```+ </Step>+ <IntegrationMethods />+</Steps>++---++<CardGroup cols={3}>+ <Card title="Quickstart: LLM Observability" href="/latest/sdk/quickstart-ai-observability" icon='bolt'>+ Production-ready AI monitoring setup in 2 simple steps with zero code changes+ </Card>+ <Card title="Configuration" href="/latest/sdk/configuration" icon='bolt'>+ Configure the OpenLIT SDK according to you requirements.+ </Card>+ <Card title="Destinations" href="/latest/sdk/destinations/overview" icon='link'>+ Send telemetry to Datadog, Grafana, New Relic, and other observability stacks+ </Card>+</CardGroup>diff --git a/docs/latest/sdk/integrations/strands.mdx b/docs/latest/sdk/integrations/strands.mdxindex 48e28dd1..f70ea3a6 100644--- a/docs/latest/sdk/integrations/strands.mdx+++ b/docs/latest/sdk/integrations/strands.mdx@@ -1,5 +1,6 @@---title: 'Monitor Strands Agents using OpenTelemetry'+description: 'Trace Strands Agents SDK applications with OpenTelemetry auto-instrumentation for AI observability into agent invocations, tool calls, and chat completions.'sidebarTitle: 'Strands Agents'---@@ -47,12 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Running in Kubernetes? Try the OpenLIT Operator"- icon={<svg role="img" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">- <title>Kubernetes</title>- <path fill="#FFA500" d="M10.204 14.35l.007.01-.999 2.413a5.171 5.171 0 0 1-2.075-2.597l2.578-.437.004.005a.44.44 0 0 1 .484.606zm-.833-2.129a.44.44 0 0 0 .173-.756l.002-.011L7.585 9.7a5.143 5.143 0 0 0-.73 3.255l2.514-.725.002-.009zm1.145-1.98a.44.44 0 0 0 .699-.337l.01-.005.15-2.62a5.144 5.144 0 0 0-3.01 1.442l2.147 1.523.004-.002zm.76 2.75l.723.349.722-.347.18-.78-.5-.623h-.804l-.5.623.179.779zm1.5-3.095a.44.44 0 0 0 .7.336l.008.003 2.134-1.513a5.188 5.188 0 0 0-2.992-1.442l.148 2.615.002.001zm10.876 5.97l-5.773 7.181a1.6 1.6 0 0 1-1.248.594l-9.261.003a1.6 1.6 0 0 1-1.247-.596l-5.776-7.18a1.583 1.583 0 0 1-.307-1.34L2.1 5.573c.108-.47.425-.864.863-1.073L11.305.513a1.606 1.606 0 0 1 1.385 0l8.345 3.985c.438.209.755.604.863 1.073l2.062 8.955c.108.47-.005.963-.308 1.34zm-3.289-2.057c-.042-.01-.103-.026-.145-.034-.174-.033-.315-.025-.479-.038-.35-.037-.638-.067-.895-.148-.105-.04-.18-.165-.216-.216l-.201-.059a6.45 6.45 0 0 0-.105-2.332 6.465 6.465 0 0 0-.936-2.163c.052-.047.15-.133.177-.159.008-.09.001-.183.094-.282.197-.185.444-.338.743-.522.142-.084.273-.137.415-.242.032-.024.076-.062.11-.089.24-.191.295-.52.123-.736-.172-.216-.506-.236-.745-.045-.034.027-.08.062-.111.088-.134.116-.217.23-.33.35-.246.25-.45.458-.673.609-.097.056-.239.037-.303.033l-.19.135a6.545 6.545 0 0 0-4.146-2.003l-.012-.223c-.065-.062-.143-.115-.163-.25-.022-.268.015-.557.057-.905.023-.163.061-.298.068-.475.001-.04-.001-.099-.001-.142 0-.306-.224-.555-.5-.555-.275 0-.499.249-.499.555l.001.014c0 .041-.002.092 0 .128.006.177.044.312.067.475.042.348.078.637.056.906a.545.545 0 0 1-.162.258l-.012.211a6.424 6.424 0 0 0-4.166 2.003 8.373 8.373 0 0 1-.18-.128c-.09.012-.18.04-.297-.029-.223-.15-.427-.358-.673-.608-.113-.12-.195-.234-.329-.349-.03-.026-.077-.062-.111-.088a.594.594 0 0 0-.348-.132.481.481 0 0 0-.398.176c-.172.216-.117.546.123.737l.007.005.104.083c.142.105.272.159.414.242.299.185.546.338.743.522.076.082.09.226.1.288l.16.143a6.462 6.462 0 0 0-1.02 4.506l-.208.06c-.055.072-.133.184-.215.217-.257.081-.546.11-.895.147-.164.014-.305.006-.48.039-.037.007-.09.02-.133.03l-.004.002-.007.002c-.295.071-.484.342-.423.608.061.267.349.429.645.365l.007-.001.01-.003.129-.029c.17-.046.294-.113.448-.172.33-.118.604-.217.87-.256.112-.009.23.069.288.101l.217-.037a6.5 6.5 0 0 0 2.88 3.596l-.09.218c.033.084.069.199.044.282-.097.252-.263.517-.452.813-.091.136-.185.242-.268.399-.02.037-.045.095-.064.134-.128.275-.034.591.213.71.248.12.556-.007.69-.282v-.002c.02-.039.046-.09.062-.127.07-.162.094-.301.144-.458.132-.332.205-.68.387-.897.05-.06.13-.082.215-.105l.113-.205a6.453 6.453 0 0 0 4.609.012l.106.192c.086.028.18.042.256.155.136.232.229.507.342.84.05.156.074.295.145.457.016.037.043.09.062.129.133.276.442.402.69.282.247-.118.341-.435.213-.71-.02-.039-.045-.096-.065-.134-.083-.156-.177-.261-.268-.398-.19-.296-.346-.541-.443-.793-.04-.13.007-.21.038-.294-.018-.022-.059-.144-.083-.202a6.499 6.499 0 0 0 2.88-3.622c.064.01.176.03.213.038.075-.05.144-.114.28-.104.266.039.54.138.87.256.154.06.277.128.448.173.036.01.088.019.13.028l.009.003.007.001c.297.064.584-.098.645-.365.06-.266-.128-.537-.423-.608zM16.4 9.701l-1.95 1.746v.005a.44.44 0 0 0 .173.757l.003.01 2.526.728a5.199 5.199 0 0 0-.108-1.674A5.208 5.208 0 0 0 16.4 9.7zm-4.013 5.325a.437.437 0 0 0-.404-.232.44.44 0 0 0-.372.233h-.002l-1.268 2.292a5.164 5.164 0 0 0 3.326.003l-1.27-2.296h-.01zm1.888-1.293a.44.44 0 0 0-.27.036.44.44 0 0 0-.214.572l-.003.004 1.01 2.438a5.15 5.15 0 0 0 2.081-2.615l-2.6-.44-.004.005z"/></svg>}- href="/latest/operator/overview"- >- Automatically inject instrumentation into existing workloads without modifying pod specs, container images, or application code.- </Card>diff --git a/docs/latest/sdk/integrations/swarmzero.mdx b/docs/latest/sdk/integrations/swarmzero.mdxindex e17cba4f..e306c1db 100644--- a/docs/latest/sdk/integrations/swarmzero.mdx+++ b/docs/latest/sdk/integrations/swarmzero.mdx@@ -1,5 +1,6 @@---title: 'Monitor SwarmZero using OpenTelemetry'+description: 'Monitor SwarmZero AI agents using OpenTelemetry auto-instrumentation, gaining AI observability into agent performance and operation stats automatically.'sidebarTitle: 'SwarmZero'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/titan-ml.mdx b/docs/latest/sdk/integrations/titan-ml.mdxindex 21b05b5a..0cd36216 100644--- a/docs/latest/sdk/integrations/titan-ml.mdx+++ b/docs/latest/sdk/integrations/titan-ml.mdx@@ -1,5 +1,6 @@---title: 'Monitor Titan ML using OpenTelemetry'+description: 'OpenLIT auto-instruments Titan ML inference with OpenTelemetry for AI observability, tracing performance, token usage, and cost tracking with zero setup.'sidebarTitle: 'Titan ML'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/together.mdx b/docs/latest/sdk/integrations/together.mdxindex 281663de..2e28097a 100644--- a/docs/latest/sdk/integrations/together.mdx+++ b/docs/latest/sdk/integrations/together.mdx@@ -1,5 +1,6 @@---title: 'Monitor Together AI using OpenTelemetry'+description: 'Monitor Together AI LLM applications with OpenTelemetry auto-instrumentation, gaining AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'Together AI'---@@ -48,10 +49,3 @@ The integration is compatible with:Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/vercel-ai.mdx b/docs/latest/sdk/integrations/vercel-ai.mdxindex 2db171ce..4491b44b 100644--- a/docs/latest/sdk/integrations/vercel-ai.mdx+++ b/docs/latest/sdk/integrations/vercel-ai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Vercel AI SDK using OpenTelemetry'+description: 'Trace AI applications built with the Vercel AI SDK using OpenTelemetry auto-instrumentation for AI observability into performance and token usage.'sidebarTitle: 'Vercel AI SDK'---@@ -47,10 +48,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/integrations/vertexai.mdx b/docs/latest/sdk/integrations/vertexai.mdxindex 890053fd..a94d4a97 100644--- a/docs/latest/sdk/integrations/vertexai.mdx+++ b/docs/latest/sdk/integrations/vertexai.mdx@@ -1,5 +1,6 @@---title: 'Monitor Vertex AI using OpenTelemetry'+description: 'OpenLIT auto-instruments Google Vertex AI LLM applications with OpenTelemetry, providing AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'Vertex AI'---@@ -37,10 +38,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/vllm.mdx b/docs/latest/sdk/integrations/vllm.mdxindex 58855c7b..4dcba569 100644--- a/docs/latest/sdk/integrations/vllm.mdx+++ b/docs/latest/sdk/integrations/vllm.mdx@@ -1,28 +1,196 @@---title: 'Monitor vLLM using OpenTelemetry'+description: 'Monitor vLLM inference server applications with OpenTelemetry auto-instrumentation, gaining AI observability into performance and token usage tracking.'sidebarTitle: 'vLLM'---import IntegrationMethods from '/snippets/integration-methods.mdx';-OpenLIT uses OpenTelemetry Auto-Instrumentation to help you monitor LLM applications built using models from vLLM. This includes tracking performance, token usage, and how users interact with the application.+OpenLIT uses OpenTelemetry instrumentation to help you monitor applications built using models served by vLLM. This includes tracking performance, token usage, and how users interact with the application.-Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.+The integration is compatible with:+- vLLM Python SDK client `>=0.5.4`+- vLLM's OpenAI-compatible API, via OpenLIT's Go SDK `InstrumentedClient` wrapper-The integration is compatible with-- vLLM Python SDK client `>=0.5.4`+## Get started-## Get started+<Tabs>+ <Tab title="Python / TypeScript">+ Auto-instrumentation means you don't have to set up monitoring manually for different LLMs, frameworks, or databases. By simply adding OpenLIT in your application, all the necessary monitoring configurations are automatically set up.-<Steps>- <Step title="Install OpenLIT">- Open your command line or terminal and run:- ```shell- pip install openlit- ```- </Step>- <IntegrationMethods />-</Steps>+ <Steps>+ <Step title="Install OpenLIT">+ Open your command line or terminal and run:+ ```shell+ pip install openlit+ ```+ </Step>+ <IntegrationMethods />+ </Steps>+ </Tab>++ <Tab title="Go">+ vLLM exposes an OpenAI-compatible API, so the Go SDK wraps an HTTP client pointed at your vLLM server with an `InstrumentedClient` that automatically emits traces and metrics for every request - with zero changes to your application logic.++ The integration supports:+ - Chat completions (standard and streaming)++ <Steps>+ <Step title="Install the Go SDK">+ Open your terminal and run:++ <Tabs>+ <Tab title="Latest Version">+ ```shell+ go get github.com/openlit/openlit/sdk/go+ ```+ </Tab>+ <Tab title="Specific Version">+ ```shell+ go get github.com/openlit/openlit/sdk/[email protected]+ ```+ Replace `v1.2.3` with the version you want to install.+ </Tab>+ </Tabs>+ </Step>++ <Step title="Initialize OpenLIT">+ Add this once at the start of your application (e.g. in `main()`):++ <Tabs>+ <Tab title="Via Config Struct">+ ```go+ import (+ "context"+ openlit "github.com/openlit/openlit/sdk/go"+ )++ if err := openlit.Init(openlit.Config{+ OtlpEndpoint: "YOUR_OTEL_ENDPOINT",+ ApplicationName: "my-ai-app",+ Environment: "production",+ }); err != nil {+ log.Fatal(err)+ }+ defer openlit.Shutdown(context.Background())+ ```+ </Tab>+ <Tab title="Via Environment Variable">+ ```go+ import (+ "context"+ openlit "github.com/openlit/openlit/sdk/go"+ )++ if err := openlit.Init(openlit.Config{+ ApplicationName: "my-ai-app",+ }); err != nil {+ log.Fatal(err)+ }+ defer openlit.Shutdown(context.Background())+ ```++ Then set your OTLP endpoint via environment variable:+ ```shell+ export OTEL_EXPORTER_OTLP_ENDPOINT=YOUR_OTEL_ENDPOINT+ ```+ </Tab>+ </Tabs>++ **Replace** `YOUR_OTEL_ENDPOINT` with the URL of your OpenTelemetry backend, such as `http://127.0.0.1:4318` for a local OpenLIT deployment.+ </Step>++ <Step title="Create an instrumented client">+ Point the OpenLIT instrumented client at your vLLM server:++ ```go+ import "github.com/openlit/openlit/sdk/go/instrumentation/vllm"++ client := vllm.NewClient("http://127.0.0.1:8000/v1")+ ```++ Optional configuration:+ ```go+ // If your vLLM deployment requires an API key+ client := vllm.NewClient("http://127.0.0.1:8000/v1",+ vllm.WithAPIKey("your-api-key"),+ )+ ```+ </Step>++ <Step title="Use the client">+ Use the instrumented client to call your vLLM server's OpenAI-compatible chat endpoint:++ **Chat completion:**+ ```go+ resp, err := client.CreateChatCompletion(ctx, vllm.ChatCompletionRequest{+ Model: "facebook/opt-125m",+ Messages: []vllm.ChatMessage{+ {Role: "system", Content: "You are a helpful assistant."},+ {Role: "user", Content: "What is OpenTelemetry?"},+ },+ MaxTokens: 256,+ Temperature: 0.7,+ })+ if err != nil {+ return err+ }+ fmt.Println(resp.Choices[0].Message.Content)+ ```++ **Streaming:**+ ```go+ stream, err := client.CreateChatCompletionStream(ctx, vllm.ChatCompletionRequest{+ Model: "facebook/opt-125m",+ Messages: []vllm.ChatMessage{+ {Role: "user", Content: "Tell me a story."},+ },+ })+ if err != nil {+ return err+ }+ defer stream.Close()++ for {+ chunk, err := stream.Recv()+ if err == io.EOF {+ break+ }+ if err != nil {+ return err+ }+ if len(chunk.Choices) > 0 {+ fmt.Print(chunk.Choices[0].Delta.Content)+ }+ }+ ```+ </Step>+ </Steps>++ ### What gets collected++ Every call to the instrumented client automatically records:++ | Data | Attribute |+ |------|-----------|+ | Operation name | `gen_ai.operation.name` |+ | Model requested | `gen_ai.request.model` |+ | Model used | `gen_ai.response.model` |+ | Input tokens | `gen_ai.usage.input_tokens` |+ | Output tokens | `gen_ai.usage.output_tokens` |+ | Finish reason | `gen_ai.response.finish_reasons` |+ | Tool calls | `gen_ai.tool.name`, `gen_ai.tool.call.id` |+ | Time to first token | `gen_ai.server.time_to_first_token` (streaming) |+ | Time per output token | `gen_ai.server.time_per_output_token` (streaming) |++ Metrics emitted:+ - `gen_ai.client.token.usage` - token usage histogram (input/output)+ - `gen_ai.client.operation.duration` - total operation duration+ - `gen_ai.server.time_to_first_token` - TTFT for streaming+ - `gen_ai.client.operation.time_to_first_chunk` - client-side TTFT+ - `gen_ai.client.operation.time_per_output_chunk` - per-chunk latency+ </Tab>+</Tabs>---@@ -37,10 +205,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/integrations/xai.mdx b/docs/latest/sdk/integrations/xai.mdxindex 329fcff9..8b7dd590 100644--- a/docs/latest/sdk/integrations/xai.mdx+++ b/docs/latest/sdk/integrations/xai.mdx@@ -1,5 +1,6 @@---title: 'Monitor xAI using OpenTelemetry'+description: 'Trace xAI Grok LLM applications with OpenTelemetry auto-instrumentation for AI observability into performance, token usage, and cost tracking.'sidebarTitle: 'xAI'---@@ -48,10 +49,3 @@ The integration is compatible withSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/overview.mdx b/docs/latest/sdk/overview.mdxindex ca69a16f..caa94c7e 100644--- a/docs/latest/sdk/overview.mdx+++ b/docs/latest/sdk/overview.mdx@@ -3,8 +3,8 @@ title: 'Overview'description: 'Production-ready AI observability with zero code changes. Monitor LLM applications, track token usage, detect hallucinations, and optimize costs with OpenTelemetry-native instrumentation.'----The OpenLIT SDK is an **open-source**, **OpenTelemetry-native** LLM observability platform-that enables production-ready AI application and agent monitoring **without any code changes**.+The OpenLIT SDK is an **open-source**, **OpenTelemetry-native** AI observability platform+that enables production-ready LLM application and agent monitoring **without any code changes**.It includes upstream OpenTelemetry instrumentations for HTTP frameworks, clients, and system components, delivering **complete visibility across your entire AI infrastructure**.@@ -99,10 +99,17 @@ Select from the following guides to learn more about production AI monitoring:Send telemetry to Datadog, Grafana, New Relic, Dash0, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of file++## Frequently asked questions++<AccordionGroup>+ <Accordion title="What is the OpenLIT SDK?">+ The OpenLIT SDK is an open-source, OpenTelemetry-native library that auto-instruments LLM providers, agent frameworks, and vector databases for AI observability - tracing, cost tracking, and evaluation - without code changes.+ </Accordion>+ <Accordion title="Does the SDK require changing my code?">+ No. Run `openlit-instrument python your_app.py` for zero-code instrumentation, or call `openlit.init()` once if you prefer manual setup - both produce the same OpenTelemetry traces.+ </Accordion>+ <Accordion title="Can I send data to my existing observability stack?">+ Yes. Since the SDK emits standard OpenTelemetry data, you can export to Datadog, Grafana, New Relic, Dash0, or any OTLP-compatible backend - see [Destinations](/latest/sdk/destinations/overview).+ </Accordion>+</AccordionGroup>\ No newline at end of filediff --git a/docs/latest/sdk/quickstart-gpu.mdx b/docs/latest/sdk/quickstart-gpu.mdxindex ed9463a5..43ec3526 100644--- a/docs/latest/sdk/quickstart-gpu.mdx+++ b/docs/latest/sdk/quickstart-gpu.mdx@@ -21,10 +21,3 @@ import GPUQuickstart from '/snippets/quickstart-gpu.mdx';Send elemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/quickstart-guard.mdx b/docs/latest/sdk/quickstart-guard.mdxindex 8f174a2e..8f5e452b 100644--- a/docs/latest/sdk/quickstart-guard.mdx+++ b/docs/latest/sdk/quickstart-guard.mdx@@ -21,10 +21,3 @@ import GuardQuickstart from '/snippets/quickstart-guard.mdx';Send elemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/quickstart-mcp-observability.mdx b/docs/latest/sdk/quickstart-mcp-observability.mdxindex 2e61c7f9..d74a8e31 100644--- a/docs/latest/sdk/quickstart-mcp-observability.mdx+++ b/docs/latest/sdk/quickstart-mcp-observability.mdx@@ -25,10 +25,3 @@ OpenLIT automatically instruments MCP alongside LLMs, VectorDBs, and frameworksSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/latest/sdk/quickstart-programmatic-evals.mdx b/docs/latest/sdk/quickstart-programmatic-evals.mdxindex 8b97b386..6fc2e9fe 100644--- a/docs/latest/sdk/quickstart-programmatic-evals.mdx+++ b/docs/latest/sdk/quickstart-programmatic-evals.mdx@@ -11,20 +11,22 @@ import EvalsQuickstart from '/snippets/quickstart-programmatic-evals.mdx';---<CardGroup cols={3}>- <Card title="Quickstart: LLM Guardrails" href="/latest/sdk/quickstart-guard" icon='bolt'>+ <Card title="Full SDK reference" href="/latest/sdk/features/evaluations" icon='book'>+ Parameters, result objects, batch evaluation, and Rule Engine attribute resolution in depth+ </Card>+ <Card title="CI/CD integration" href="/latest/sdk/features/evaluations#ci-cd-integration" icon='code-branch'>+ Use evaluations as quality gates in your test suite or pipeline+ </Card>+ <Card title="Online evaluations" href="/latest/openlit/quickstart-evals" icon='bolt'>+ Auto-evaluate every production trace instead of calling evaluations yourself+ </Card>+ <Card title="Quickstart: LLM Guardrails" href="/latest/sdk/quickstart-guard" icon='shield'>Protect and secure your LLM responses in 2 simple steps</Card><Card title="Integrations" href="/latest/sdk/integrations/overview" icon='circle-nodes'>60+ AI integrations with automatic instrumentation and performance tracking</Card><Card title="Destinations" href="/latest/sdk/destinations/overview" icon='link'>- Send elemetry to Datadog, Grafana, New Relic, and other observability stacks+ Send telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>diff --git a/docs/latest/sdk/quickstart-vectordb-observability.mdx b/docs/latest/sdk/quickstart-vectordb-observability.mdxindex c53c3dee..79f157f3 100644--- a/docs/latest/sdk/quickstart-vectordb-observability.mdx+++ b/docs/latest/sdk/quickstart-vectordb-observability.mdx@@ -25,10 +25,3 @@ OpenLIT automatically instruments VectorDBs alongside LLMs, MCP, and frameworksSend telemetry to Datadog, Grafana, New Relic, and other observability stacks</Card></CardGroup>-<Card- title="Zero-code observability with the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/overview"- >- Discover and instrument LLM traffic across Kubernetes, Docker, and Linux using eBPF — no code changes required.- </Card>\ No newline at end of filediff --git a/docs/package-lock.json b/docs/package-lock.jsonnew file mode 100644index 00000000..5904bf14--- /dev/null+++ b/docs/package-lock.json@@ -0,0 +1,452 @@+{+ "name": "docs",+ "lockfileVersion": 3,+ "requires": true,+ "packages": {+ "": {+ "dependencies": {+ "keytar": "^7.9.0"+ }+ },+ "node_modules/base64-js": {+ "version": "1.5.1",+ "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",+ "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",+ "funding": [+ {+ "type": "github",+ "url": "https://github.com/sponsors/feross"+ },+ {+ "type": "patreon",+ "url": "https://www.patreon.com/feross"+ },+ {+ "type": "consulting",+ "url": "https://feross.org/support"+ }+ ],+ "license": "MIT"+ },+ "node_modules/bl": {+ "version": "4.1.0",+ "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz",+ "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==",+ "license": "MIT",+ "dependencies": {+ "buffer": "^5.5.0",+ "inherits": "^2.0.4",+ "readable-stream": "^3.4.0"+ }+ },+ "node_modules/buffer": {+ "version": "5.7.1",+ "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz",+ "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==",+ "funding": [+ {+ "type": "github",+ "url": "https://github.com/sponsors/feross"+ },+ {+ "type": "patreon",+ "url": "https://www.patreon.com/feross"+ },+ {+ "type": "consulting",+ "url": "https://feross.org/support"+ }+ ],+ "license": "MIT",+ "dependencies": {+ "base64-js": "^1.3.1",+ "ieee754": "^1.1.13"+ }+ },+ "node_modules/chownr": {+ "version": "1.1.4",+ "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz",+ "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==",+ "license": "ISC"+ },+ "node_modules/decompress-response": {+ "version": "6.0.0",+ "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz",+ "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==",+ "license": "MIT",+ "dependencies": {+ "mimic-response": "^3.1.0"+ },+ "engines": {+ "node": ">=10"+ },+ "funding": {+ "url": "https://github.com/sponsors/sindresorhus"+ }+ },+ "node_modules/deep-extend": {+ "version": "0.6.0",+ "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz",+ "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==",+ "license": "MIT",+ "engines": {+ "node": ">=4.0.0"+ }+ },+ "node_modules/detect-libc": {+ "version": "2.1.2",+ "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz",+ "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",+ "license": "Apache-2.0",+ "engines": {+ "node": ">=8"+ }+ },+ "node_modules/end-of-stream": {+ "version": "1.4.5",+ "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",+ "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",+ "license": "MIT",+ "dependencies": {+ "once": "^1.4.0"+ }+ },+ "node_modules/expand-template": {+ "version": "2.0.3",+ "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz",+ "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==",+ "license": "(MIT OR WTFPL)",+ "engines": {+ "node": ">=6"+ }+ },+ "node_modules/fs-constants": {+ "version": "1.0.0",+ "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz",+ "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==",+ "license": "MIT"+ },+ "node_modules/github-from-package": {+ "version": "0.0.0",+ "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz",+ "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==",+ "license": "MIT"+ },+ "node_modules/ieee754": {+ "version": "1.2.1",+ "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",+ "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",+ "funding": [+ {+ "type": "github",+ "url": "https://github.com/sponsors/feross"+ },+ {+ "type": "patreon",+ "url": "https://www.patreon.com/feross"+ },+ {+ "type": "consulting",+ "url": "https://feross.org/support"+ }+ ],+ "license": "BSD-3-Clause"+ },+ "node_modules/inherits": {+ "version": "2.0.4",+ "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",+ "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",+ "license": "ISC"+ },+ "node_modules/ini": {+ "version": "1.3.8",+ "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz",+ "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",+ "license": "ISC"+ },+ "node_modules/keytar": {+ "version": "7.9.0",+ "resolved": "https://registry.npmjs.org/keytar/-/keytar-7.9.0.tgz",+ "integrity": "sha512-VPD8mtVtm5JNtA2AErl6Chp06JBfy7diFQ7TQQhdpWOl6MrCRB+eRbvAZUsbGQS9kiMq0coJsy0W0vHpDCkWsQ==",+ "hasInstallScript": true,+ "license": "MIT",+ "dependencies": {+ "node-addon-api": "^4.3.0",+ "prebuild-install": "^7.0.1"+ }+ },+ "node_modules/mimic-response": {+ "version": "3.1.0",+ "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz",+ "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==",+ "license": "MIT",+ "engines": {+ "node": ">=10"+ },+ "funding": {+ "url": "https://github.com/sponsors/sindresorhus"+ }+ },+ "node_modules/minimist": {+ "version": "1.2.8",+ "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz",+ "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==",+ "license": "MIT",+ "funding": {+ "url": "https://github.com/sponsors/ljharb"+ }+ },+ "node_modules/mkdirp-classic": {+ "version": "0.5.3",+ "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz",+ "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==",+ "license": "MIT"+ },+ "node_modules/napi-build-utils": {+ "version": "2.0.0",+ "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz",+ "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==",+ "license": "MIT"+ },+ "node_modules/node-abi": {+ "version": "3.94.0",+ "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.94.0.tgz",+ "integrity": "sha512-W5ZNO5KRPB5TkYmGVD9F6YqhsglXJzE6etpbmT+f6EQElhiX/UTG551cnsRGvLG3fyZEg9HwaDmNmj5nwJ4z9g==",+ "license": "MIT",+ "dependencies": {+ "semver": "^7.3.5"+ },+ "engines": {+ "node": ">=10"+ }+ },+ "node_modules/node-addon-api": {+ "version": "4.3.0",+ "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-4.3.0.tgz",+ "integrity": "sha512-73sE9+3UaLYYFmDsFZnqCInzPyh3MqIwZO9cw58yIqAZhONrrabrYyYe3TuIqtIiOuTXVhsGau8hcrhhwSsDIQ==",+ "license": "MIT"+ },+ "node_modules/once": {+ "version": "1.4.0",+ "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",+ "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",+ "license": "ISC",+ "dependencies": {+ "wrappy": "1"+ }+ },+ "node_modules/prebuild-install": {+ "version": "7.1.3",+ "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",+ "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==",+ "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.",+ "license": "MIT",+ "dependencies": {+ "detect-libc": "^2.0.0",+ "expand-template": "^2.0.3",+ "github-from-package": "0.0.0",+ "minimist": "^1.2.3",+ "mkdirp-classic": "^0.5.3",+ "napi-build-utils": "^2.0.0",+ "node-abi": "^3.3.0",+ "pump": "^3.0.0",+ "rc": "^1.2.7",+ "simple-get": "^4.0.0",+ "tar-fs": "^2.0.0",+ "tunnel-agent": "^0.6.0"+ },+ "bin": {+ "prebuild-install": "bin.js"+ },+ "engines": {+ "node": ">=10"+ }+ },+ "node_modules/pump": {+ "version": "3.0.4",+ "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",+ "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",+ "license": "MIT",+ "dependencies": {+ "end-of-stream": "^1.1.0",+ "once": "^1.3.1"+ }+ },+ "node_modules/rc": {+ "version": "1.2.8",+ "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz",+ "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==",+ "license": "(BSD-2-Clause OR MIT OR Apache-2.0)",+ "dependencies": {+ "deep-extend": "^0.6.0",+ "ini": "~1.3.0",+ "minimist": "^1.2.0",+ "strip-json-comments": "~2.0.1"+ },+ "bin": {+ "rc": "cli.js"+ }+ },+ "node_modules/readable-stream": {+ "version": "3.6.2",+ "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",+ "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",+ "license": "MIT",+ "dependencies": {+ "inherits": "^2.0.3",+ "string_decoder": "^1.1.1",+ "util-deprecate": "^1.0.1"+ },+ "engines": {+ "node": ">= 6"+ }+ },+ "node_modules/safe-buffer": {+ "version": "5.2.1",+ "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",+ "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==",+ "funding": [+ {+ "type": "github",+ "url": "https://github.com/sponsors/feross"+ },+ {+ "type": "patreon",+ "url": "https://www.patreon.com/feross"+ },+ {+ "type": "consulting",+ "url": "https://feross.org/support"+ }+ ],+ "license": "MIT"+ },+ "node_modules/semver": {+ "version": "7.8.5",+ "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz",+ "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==",+ "license": "ISC",+ "bin": {+ "semver": "bin/semver.js"+ },+ "engines": {+ "node": ">=10"+ }+ },+ "node_modules/simple-concat": {+ "version": "1.0.1",+ "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz",+ "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==",+ "funding": [+ {+ "type": "github",+ "url": "https://github.com/sponsors/feross"+ },+ {+ "type": "patreon",+ "url": "https://www.patreon.com/feross"+ },+ {+ "type": "consulting",+ "url": "https://feross.org/support"+ }+ ],+ "license": "MIT"+ },+ "node_modules/simple-get": {+ "version": "4.0.1",+ "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz",+ "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==",+ "funding": [+ {+ "type": "github",+ "url": "https://github.com/sponsors/feross"+ },+ {+ "type": "patreon",+ "url": "https://www.patreon.com/feross"+ },+ {+ "type": "consulting",+ "url": "https://feross.org/support"+ }+ ],+ "license": "MIT",+ "dependencies": {+ "decompress-response": "^6.0.0",+ "once": "^1.3.1",+ "simple-concat": "^1.0.0"+ }+ },+ "node_modules/string_decoder": {+ "version": "1.3.0",+ "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",+ "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",+ "license": "MIT",+ "dependencies": {+ "safe-buffer": "~5.2.0"+ }+ },+ "node_modules/strip-json-comments": {+ "version": "2.0.1",+ "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",+ "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==",+ "license": "MIT",+ "engines": {+ "node": ">=0.10.0"+ }+ },+ "node_modules/tar-fs": {+ "version": "2.1.5",+ "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.5.tgz",+ "integrity": "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw==",+ "license": "MIT",+ "dependencies": {+ "chownr": "^1.1.1",+ "mkdirp-classic": "^0.5.2",+ "pump": "^3.0.0",+ "tar-stream": "^2.1.4"+ }+ },+ "node_modules/tar-stream": {+ "version": "2.2.0",+ "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz",+ "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==",+ "license": "MIT",+ "dependencies": {+ "bl": "^4.0.3",+ "end-of-stream": "^1.4.1",+ "fs-constants": "^1.0.0",+ "inherits": "^2.0.3",+ "readable-stream": "^3.1.1"+ },+ "engines": {+ "node": ">=6"+ }+ },+ "node_modules/tunnel-agent": {+ "version": "0.6.0",+ "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",+ "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==",+ "license": "Apache-2.0",+ "dependencies": {+ "safe-buffer": "^5.0.1"+ },+ "engines": {+ "node": "*"+ }+ },+ "node_modules/util-deprecate": {+ "version": "1.0.2",+ "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",+ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",+ "license": "MIT"+ },+ "node_modules/wrappy": {+ "version": "1.0.2",+ "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",+ "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",+ "license": "ISC"+ }+ }+}diff --git a/docs/package.json b/docs/package.jsonnew file mode 100644index 00000000..ffacb1a7--- /dev/null+++ b/docs/package.json@@ -0,0 +1,5 @@+{+ "dependencies": {+ "keytar": "^7.9.0"+ }+}diff --git a/docs/snippets/llm-as-a-judge.mdx b/docs/snippets/llm-as-a-judge.mdxindex d403c6a5..936ec58c 100644--- a/docs/snippets/llm-as-a-judge.mdx+++ b/docs/snippets/llm-as-a-judge.mdx@@ -55,12 +55,12 @@ OpenLIT provides **11 evaluation types** that can be used to evaluate the output</CardGroup><Info>-**Context is the source of truth**: When context is provided, evaluations judge the LLM response against the provided context — not against real-world knowledge. If the context says "2+2=5" and the LLM says "2+2=4", the evaluation flags it as a factual inaccuracy because it contradicts the provided context.+**Context is the source of truth**: When context is provided, evaluations judge the LLM response against the provided context - not against real-world knowledge. If the context says "2+2=5" and the LLM says "2+2=4", the evaluation flags it as a factual inaccuracy because it contradicts the provided context.</Info>## Running evaluations-OpenLIT provides two ways to configure evaluation settings — from the Settings page or directly from a trace.+OpenLIT provides two ways to configure evaluation settings - from the Settings page or directly from a trace.### Configure from Settings@@ -75,7 +75,7 @@ OpenLIT provides two ways to configure evaluation settings — from the Settings5. Click **Save Changes**<Tip>-Evaluations are powered by the **Vercel AI SDK** and support 11+ providers including OpenAI, Anthropic, Google, Mistral, Cohere, Groq, Perplexity, DeepSeek, xAI, Together, and Fireworks.+Evaluations support 11+ providers including OpenAI, Anthropic, Google, Mistral, Cohere, Groq, Perplexity, DeepSeek, xAI, Together, and Fireworks.</Tip>### Configure evaluation types@@ -105,7 +105,7 @@ In addition to the 11 built-in types, you can create your own evaluation types t- **Description**: A brief explanation of what this evaluation checks- **Evaluation Prompt**: The prompt used by the LLM judge. It should start with a `[Label evaluation context]` header describing the evaluation criteria-Custom types work exactly like built-in types — they can be enabled or disabled, linked to Rule Engine rules, and run in both auto and manual evaluations. The only difference is that custom types can be deleted when they are no longer needed, while built-in types cannot.+Custom types work exactly like built-in types - they can be enabled or disabled, linked to Rule Engine rules, and run in both auto and manual evaluations. The only difference is that custom types can be deleted when they are no longer needed, while built-in types cannot.### Run from a trace@@ -120,7 +120,7 @@ Custom types work exactly like built-in types — they can be enabled or disable## Context and Rule Engine integration-Evaluations integrate with the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine) and [Context](/latest/openlit/prompts-experiments/context) system:+Evaluations integrate with the [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) and [Context](/latest/openlit/prompts-experiments/context/overview) system:- **Rule-based context**: Link rules to evaluation types. When a trace matches rule conditions, the associated context is included in the evaluation prompt.- **Context as ground truth**: The provided context is always treated as the source of truth. The LLM judge evaluates responses strictly against the context, not against its own knowledge.@@ -141,7 +141,7 @@ Once evaluations are running, OpenLIT continuously analyzes your LLM responses a</Frame>1. Go to the Requests page to see all your LLM traces-2. Click on any LLM trace to view details — the trace ID and span ID are reflected in the URL for easy sharing+2. Click on any LLM trace to view details - the trace ID and span ID are reflected in the URL for easy sharing<Frame caption="Trace and span IDs in URL for shareable links"><img src="/images/trace-url-params.png" />diff --git a/docs/snippets/quickstart-controller.mdx b/docs/snippets/quickstart-controller.mdxdeleted file mode 100644index 8c3879c4..00000000--- a/docs/snippets/quickstart-controller.mdx+++ /dev/null@@ -1,205 +0,0 @@-## Select where your AI application is running--<Tabs>- <Tab title="Kubernetes" icon="dharmachakra">-- <Steps>- <Step title="Deploy OpenLIT with the Controller" titleSize="h3">- Deploy OpenLIT and the Controller together using the [Helm chart](https://github.com/openlit/helm/tree/main/charts/openlit). The Controller automatically discovers your services and lets you enable LLM observability from the dashboard — no code changes needed.-- ```shell- helm repo add openlit https://openlit.github.io/helm/- helm repo update- helm install openlit openlit/openlit --set openlit-controller.enabled=true- ```-- The Controller's DaemonSet, RBAC, and configuration are all handled by the chart.- The dashboard URL and OTLP endpoint are auto-derived from the Helm release name.-- <Tip title="Secure with an API Key">- If you've created an API key in **Settings → API Keys**, pass it to the Controller so it authenticates with the dashboard:-- ```shell- helm install openlit openlit/openlit \- --set openlit-controller.enabled=true \- --set openlit-controller.apiKey="YOUR_OPENLIT_API_KEY"- ```-- You can also set the `OPENLIT_API_KEY` environment variable. See [Controller configuration](/latest/controller/configuration) for details.- </Tip>-- For detailed configuration options, see the [Installation guide](/latest/openlit/installation) and [Controller configuration](/latest/controller/configuration).- </Step>- <Step title="Enable observability from the Agents page">- 1. Open the OpenLIT dashboard and navigate to **Agents**- 2. Your services making LLM API calls will appear automatically- 3. Click **Enable** next to any service to start collecting traces and metrics-- <Tip>- The Controller uses eBPF for zero-code LLM observability and can also inject the OpenLIT Python SDK for agent framework traces — no code changes, image rebuilds, or redeployments required. For deeper application-level tracing, you can also use the [OpenLIT SDKs](/latest/sdk/overview).- </Tip>- </Step>- </Steps>-- </Tab>- <Tab title="Docker" icon="docker" iconType="brands">-- <Steps>- <Step title="Deploy OpenLIT with Docker Compose" titleSize="h3">- ```shell- git clone [email protected]:openlit/openlit.git- ```-- From the root directory of the [OpenLIT Repo](https://github.com/openlit/openlit), add the Controller service to your `docker-compose.yml`:-- ```yaml- openlit-controller:- image: ghcr.io/openlit/controller:latest- privileged: true- pid: "host"- volumes:- - /proc:/host/proc:ro- - /sys/kernel/debug:/sys/kernel/debug:ro- - /sys/fs/bpf:/sys/fs/bpf:rw- - /var/run/docker.sock:/var/run/docker.sock- environment:- OPENLIT_URL: "http://openlit:3000"- OPENLIT_PROC_ROOT: "/host/proc"- OTEL_EXPORTER_OTLP_ENDPOINT: "http://openlit:4318"- # OPENLIT_API_KEY: "your-api-key" # Recommended: see tip below- depends_on:- - openlit- ```-- Then start everything:-- ```shell- docker compose up -d- ```-- <Tip title="Secure with an API Key">- If you've created an API key in **Settings → API Keys**, add it to the Controller's environment to authenticate requests:-- ```yaml- environment:- OPENLIT_API_KEY: "your-openlit-api-key"- ```-- See [Controller configuration](/latest/controller/configuration) for all available options.- </Tip>- </Step>- <Step title="Enable observability from the Agents page">- 1. Open the OpenLIT dashboard at `http://127.0.0.1:3000`- 2. Navigate to **Agents**- 3. Your containers making LLM API calls will appear automatically- 4. Click **Enable** next to any service to start collecting traces and metrics-- <Tip>- The Controller uses eBPF for zero-code LLM observability and can also inject the OpenLIT Python SDK for agent framework traces — no code changes or image rebuilds required. For deeper application-level tracing, you can also use the [OpenLIT SDKs](/latest/sdk/overview).- </Tip>- </Step>- </Steps>-- </Tab>- <Tab title="Linux" icon="linux" iconType="brands">-- <Steps>- <Step title="Deploy OpenLIT" titleSize="h3">- ```shell- git clone [email protected]:openlit/openlit.git- ```-- From the root directory of the [OpenLIT Repo](https://github.com/openlit/openlit):-- ```shell- docker compose up -d- ```- </Step>- <Step title="Install the OpenLIT Controller" titleSize="h3">- <Tabs>- <Tab title="x86_64 (amd64)">- ```bash- curl -fsSL https://github.com/openlit/openlit/releases/latest/download/openlit-controller-linux-amd64 \- -o /usr/local/bin/openlit-controller- chmod +x /usr/local/bin/openlit-controller- ```- </Tab>- <Tab title="ARM64 (aarch64)">- ```bash- curl -fsSL https://github.com/openlit/openlit/releases/latest/download/openlit-controller-linux-arm64 \- -o /usr/local/bin/openlit-controller- chmod +x /usr/local/bin/openlit-controller- ```- </Tab>- </Tabs>-- Verify the installation:-- ```bash- openlit-controller --version- ```-- Create a systemd service with the configuration passed as environment variables:-- ```ini /etc/systemd/system/openlit-controller.service- [Unit]- Description=OpenLIT Controller- After=network.target-- [Service]- Environment="OPENLIT_URL=http://your-openlit-host:3000"- Environment="OTEL_EXPORTER_OTLP_ENDPOINT=http://your-openlit-host:4318"- Environment="OPENLIT_POLL_INTERVAL=60s"- Environment="OPENLIT_ENVIRONMENT=production"- # Environment="OPENLIT_API_KEY=your-api-key" # Recommended: see tip below- ExecStart=/usr/local/bin/openlit-controller- Restart=always- RestartSec=5-- [Install]- WantedBy=multi-user.target- ```-- ```bash- systemctl daemon-reload- systemctl enable --now openlit-controller- ```-- <Tip title="Secure with an API Key">- If you've created an API key in **Settings → API Keys**, add it to the systemd service to authenticate Controller requests:-- ```ini- Environment="OPENLIT_API_KEY=your-openlit-api-key"- ```-- Then reload: `systemctl daemon-reload && systemctl restart openlit-controller`- </Tip>- </Step>- <Step title="Enable observability from the Agents page">- 1. Open the OpenLIT dashboard and navigate to **Agents**- 2. Your processes making LLM API calls will appear automatically- 3. Click **Enable** next to any service to start collecting traces and metrics-- <Tip>- The Controller uses eBPF for zero-code LLM observability and can also inject the OpenLIT Python SDK for agent framework traces — no code changes required. For deeper application-level tracing, you can also use the [OpenLIT SDKs](/latest/sdk/overview).- </Tip>- </Step>- </Steps>-- </Tab>-</Tabs>--## Monitor and optimize your AI applications--With real-time LLM observability data now flowing to OpenLIT, visualize comprehensive AI performance metrics including token costs, latency patterns, hallucination rates, and model accuracy to optimize your production AI applications.--Head over to OpenLIT at `127.0.0.1:3000` on your browser to start exploring. You can login using the default credentials:-- **Email**: `[email protected]`-- **Password**: `openlituser`--<Frame>- <img src="https://github.com/openlit/.github/blob/main/profile/assets/openlit-client-1.png?raw=true" />- <img src="https://github.com/openlit/.github/blob/main/profile/assets/openlit-client-2.png?raw=true" />-</Frame>--If you have any questions or need support, reach out to our [community](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ).diff --git a/docs/snippets/quickstart-mcp.mdx b/docs/snippets/quickstart-mcp.mdxindex 057bf50d..c6af1297 100644--- a/docs/snippets/quickstart-mcp.mdx+++ b/docs/snippets/quickstart-mcp.mdx@@ -35,13 +35,6 @@ flowchart TB;</Steps></Step><Step title="Install OpenLIT SDK">- <Card- title="Want zero-code observability? Try the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/quickstart"- >- The Controller uses eBPF to automatically discover and instrument LLM traffic across Kubernetes, Docker, and Linux — no SDK required. Use SDKs for deeper application-level tracing.- </Card><Tabs><Tab title="Python">```shelldiff --git a/docs/snippets/quickstart-observability.mdx b/docs/snippets/quickstart-observability.mdxindex 86b82932..1ca6cde1 100644--- a/docs/snippets/quickstart-observability.mdx+++ b/docs/snippets/quickstart-observability.mdx@@ -29,17 +29,13 @@ flowchart TB;```shelldocker compose up -d```+ <Tip>+ Prefer Kubernetes, or already have ClickHouse/OpenTelemetry Collector running? See [Self-Host OpenLIT](/latest/openlit/installation) for Helm charts and other deployment options.+ </Tip></Step></Steps></Step><Step title="Install OpenLIT SDK">- <Card- title="Want zero-code observability? Try the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/quickstart"- >- The Controller uses eBPF to automatically discover and instrument LLM traffic across Kubernetes, Docker, and Linux — no SDK required. Use SDKs for deeper application-level tracing.- </Card><Tabs><Tab title="Python">```shell@@ -57,9 +53,6 @@ flowchart TB;<Step title="Instrument your AI application"><Tabs><Tab title="Python">- <Tip>- Not sure which method to choose? Check out [Instrumentation Methods](/latest/sdk/instrumentation-methods) to understand the differences.- </Tip><Tabs><Tab title="Manual instrumentation"><Tabs>@@ -358,54 +351,147 @@ flowchart TB;Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })```- Example Usage for monitoring `OpenAI` Usage:+ Examples:- ```typescript+ <CodeGroup>++ ```typescript OpenAIimport Openlit from "openlit"Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })async function main() {- const OpenAI = await import("openai").then((e) => e.default);- const openai = new OpenAI({- apiKey: YOUR_OPENAI_KEY,- });- const completion = await openai.chat.completions.create({- model: "gpt-3.5-turbo",- messages: [{ role: "system", content: "Who are you ?" }],- });-- console.log(completion?.choices?.[0]);- }+ const OpenAI = await import("openai").then((e) => e.default);+ const openai = new OpenAI({+ apiKey: process.env.OPENAI_API_KEY,+ });+ const completion = await openai.chat.completions.create({+ model: "gpt-3.5-turbo",+ messages: [{ role: "user", content: "What is LLM Observability?" }],+ });++ console.log(completion?.choices?.[0]);+ }++ main();+ ```++ ```typescript Anthropic+ import Openlit from "openlit"++ Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })++ async function main() {+ const Anthropic = await import("@anthropic-ai/sdk").then((e) => e.default);+ const anthropic = new Anthropic({+ apiKey: process.env.ANTHROPIC_API_KEY,+ });+ const message = await anthropic.messages.create({+ max_tokens: 1024,+ messages: [{ role: "user", content: "Hello, What is LLM Observability?" }],+ model: "claude-3-opus-20240229",+ });++ console.log(message);+ }- main();+ main();```- OR++ ```typescript Cohere+ import Openlit from "openlit"++ Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })++ async function main() {+ const { CohereClient } = await import("cohere-ai");+ const cohere = new CohereClient({+ token: process.env.COHERE_API_KEY,+ });+ const chat = await cohere.chat({+ message: "hello world!",+ model: "command",+ });++ console.log(chat);+ }++ main();+ ```++ ```typescript Langchain+ import Openlit from "openlit"++ Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })++ async function main() {+ const { ChatOpenAI } = await import("@langchain/openai");+ const { HumanMessage, SystemMessage } = await import("@langchain/core/messages");++ const model = new ChatOpenAI({ model: "gpt-4o-mini" });++ const messages = [+ new SystemMessage("Translate the following from English into Italian"),+ new HumanMessage("hi!"),+ ];++ await model.invoke(messages);+ }++ main();+ ```++ ```typescript Ollama+ import Openlit from "openlit"++ Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })++ async function main() {+ const ollama = await import("ollama").then((e) => e.default);+ const response = await ollama.chat({+ model: "llama3.1",+ messages: [{ role: "user", content: "Why is the sky blue?" }],+ });++ console.log(response);+ }++ main();+ ```++ </CodeGroup>++ <Tip>+ Using static top-level imports instead (e.g. `import OpenAI from "openai"`)? Static imports are hoisted and run before `Openlit.init()`, so the automatic patch can miss the module. Pass it into `instrumentations` so OpenLIT patches it directly:+```typescriptimport openlit from "openlit"import OpenAI from "openai"- openlit.init({- otlpEndpoint: "http://127.0.0.1:4318",+ openlit.init({+ otlpEndpoint: "http://127.0.0.1:4318",instrumentations: {openai: OpenAI,- }+ }})async function main() {- const openai = new OpenAI({- apiKey: YOUR_OPENAI_KEY,- });- const completion = await openai.chat.completions.create({- model: "gpt-3.5-turbo",- messages: [{ role: "system", content: "Who are you ?" }],- });-- console.log(completion?.choices?.[0]);- }+ const openai = new OpenAI({+ apiKey: process.env.OPENAI_API_KEY,+ });+ const completion = await openai.chat.completions.create({+ model: "gpt-3.5-turbo",+ messages: [{ role: "user", content: "What is LLM Observability?" }],+ });- main();+ console.log(completion?.choices?.[0]);+ }++ main();```++ The same pattern works for `anthropic`, `cohere`, and `ollama`. LangChain always uses the dynamic-import pattern above, since it patches via a different hook.+ </Tip></Tab><Tab title="Via environment variables">Add the following two lines to your application code:@@ -420,29 +506,119 @@ flowchart TB;export OTEL_EXPORTER_OTLP_ENDPOINT = "http://127.0.0.1:4318"```- Example Usage for monitoring `OpenAI` Usage:+ Examples:- ```typescript+ <CodeGroup>++ ```typescript OpenAIimport openlit from "openlit"openlit.init()async function main() {- const OpenAI = await import("openai").then((e) => e.default);- const openai = new OpenAI({- apiKey: YOUR_OPENAI_KEY,- });- const completion = await openai.chat.completions.create({- model: "gpt-3.5-turbo",- messages: [{ role: "system", content: "Who are you ?" }],- });+ const OpenAI = await import("openai").then((e) => e.default);+ const openai = new OpenAI({+ apiKey: process.env.OPENAI_API_KEY,+ });+ const completion = await openai.chat.completions.create({+ model: "gpt-3.5-turbo",+ messages: [{ role: "user", content: "What is LLM Observability?" }],+ });++ console.log(completion?.choices?.[0]);+ }++ main();+ ```- console.log(completion?.choices?.[0]);- }+ ```typescript Anthropic+ import openlit from "openlit"++ openlit.init()++ async function main() {+ const Anthropic = await import("@anthropic-ai/sdk").then((e) => e.default);+ const anthropic = new Anthropic({+ apiKey: process.env.ANTHROPIC_API_KEY,+ });+ const message = await anthropic.messages.create({+ max_tokens: 1024,+ messages: [{ role: "user", content: "Hello, What is LLM Observability?" }],+ model: "claude-3-opus-20240229",+ });++ console.log(message);+ }++ main();+ ```- main();+ ```typescript Cohere+ import openlit from "openlit"++ openlit.init()++ async function main() {+ const { CohereClient } = await import("cohere-ai");+ const cohere = new CohereClient({+ token: process.env.COHERE_API_KEY,+ });+ const chat = await cohere.chat({+ message: "hello world!",+ model: "command",+ });++ console.log(chat);+ }++ main();```- OR++ ```typescript Langchain+ import openlit from "openlit"++ openlit.init()++ async function main() {+ const { ChatOpenAI } = await import("@langchain/openai");+ const { HumanMessage, SystemMessage } = await import("@langchain/core/messages");++ const model = new ChatOpenAI({ model: "gpt-4o-mini" });++ const messages = [+ new SystemMessage("Translate the following from English into Italian"),+ new HumanMessage("hi!"),+ ];++ await model.invoke(messages);+ }++ main();+ ```++ ```typescript Ollama+ import openlit from "openlit"++ openlit.init()++ async function main() {+ const ollama = await import("ollama").then((e) => e.default);+ const response = await ollama.chat({+ model: "llama3.1",+ messages: [{ role: "user", content: "Why is the sky blue?" }],+ });++ console.log(response);+ }++ main();+ ```++ </CodeGroup>++ <Tip>+ Using static top-level imports instead (e.g. `import OpenAI from "openai"`)? Static imports are hoisted and run before `openlit.init()`, so the automatic patch can miss the module. Pass it into `instrumentations` so OpenLIT patches it directly:+```typescriptimport openlit from "openlit"import OpenAI from "openai"@@ -450,23 +626,26 @@ flowchart TB;openlit.init({instrumentations: {openai: OpenAI,- }+ }})async function main() {- const openai = new OpenAI({- apiKey: YOUR_OPENAI_KEY,- });- const completion = await openai.chat.completions.create({- model: "gpt-3.5-turbo",- messages: [{ role: "system", content: "Who are you ?" }],- });-- console.log(completion?.choices?.[0]);- }-- main();+ const openai = new OpenAI({+ apiKey: process.env.OPENAI_API_KEY,+ });+ const completion = await openai.chat.completions.create({+ model: "gpt-3.5-turbo",+ messages: [{ role: "user", content: "What is LLM Observability?" }],+ });++ console.log(completion?.choices?.[0]);+ }++ main();```++ The same pattern works for `anthropic`, `cohere`, and `ollama`. LangChain always uses the dynamic-import pattern above, since it patches via a different hook.+ </Tip></Tab></Tabs></Tab>diff --git a/docs/snippets/quickstart-programmatic-evals.mdx b/docs/snippets/quickstart-programmatic-evals.mdxindex c0421ff7..010a882a 100644--- a/docs/snippets/quickstart-programmatic-evals.mdx+++ b/docs/snippets/quickstart-programmatic-evals.mdx@@ -1,140 +1,79 @@-This guide demonstrates how to implement LLM evaluation tools to assess model output quality. With OpenLIT's programmatic evaluations, you can perform LLM hallucination detection, bias detection, and toxicity filtering using production-ready evaluation metrics.--Learn how to use our `All` evaluator for comprehensive LLM output assessment, measuring hallucination, bias, and toxicity simultaneously. We'll also show you how to collect OpenTelemetry evaluation metrics for continuous model performance monitoring.+This guide shows how to run OpenLIT evaluations programmatically from your own code - the same evaluation engine, evaluators, and Rule Engine used for online/auto evaluations, called directly via the SDK.<Steps>- <Step title="Initialize evaluations">- Set up evaluations for large language models with just two lines of code:- <Tabs>- <Tab title="Python">- ```python- import openlit-- # Comprehensive LLM evaluation: hallucination detection, bias detection, toxicity filtering- evals = openlit.evals.All()- result = evals.measure()- ```-- Full Example:-- ```python example.py- import openlit-- # openlit can also read the OPENAI_API_KEY variable directy from env if not specified via function argument- openai_api_key=os.getenv("OPENAI_API_KEY")-- # Production-ready LLM evaluation tools for hallucination detection and bias screening- evals = openlit.evals.All(provider="openai", api_key=openai_api_key)-- contexts = ["Einstein won the Nobel Prize for his discovery of the photoelectric effect in 1921"]- prompt = "When and why did Einstein win the Nobel Prize?"- text = "Einstein won the Nobel Prize in 1969 for his discovery of the photoelectric effect"-- result = evals.measure(prompt=prompt, contexts=contexts, text=text)- ```-- ```sh Output- verdict='yes' evaluation='Hallucination' score=0.9 classification='factual_inaccuracy' explanation='The text incorrectly states that Einstein won the Nobel Prize in 1969, while the context specifies that he won it in 1921 for his discovery of the photoelectric effect, leading to a significant factual inconsistency.'- ```- </Tab>- <Tab title="Typescript">- ```typescript- import openlit from "openlit"-- // Comprehensive LLM evaluation: hallucination detection, bias detection, toxicity filtering- const evals = new openlit.evals.All()- const result = await evals.measure()- ```-- Full Example:-- ```typescript- import openlit from "openlit"-- // Production-ready LLM evaluation tools for hallucination detection and bias screening- const evals = new openlit.evals.All({- provider: "openai",- apiKey: process.env.OPENAI_API_KEY,- })-- const contexts = ["Einstein won the Nobel Prize for his discovery of the photoelectric effect in 1921"];- const prompt = "When and why did Einstein win the Nobel Prize?";- const text = "Einstein won the Nobel Prize in 1969 for his discovery of the photoelectric effect";-- const result = await evals.measure({ prompt, contexts, text });- console.log(result)- ```- </Tab>+ <Step title="Get an API key">+ In OpenLIT, go to **Settings → API Keys** and create a key. Programmatic evaluations run against your OpenLIT server, so you'll need this key plus your OpenLIT URL.+ </Step>+ <Step title="Run an evaluation">+ <Tabs>+ <Tab title="Python">+ ```python+ import openlit++ openlit.init(+ openlit_url="http://localhost:3000",+ openlit_api_key="openlit-xxxxx",+ )++ result = openlit.eval(+ prompt="What is the capital of France?",+ response="The capital of France is Lyon.",+ contexts=["Paris is the capital and largest city of France."],+ )++ assert result.passed, f"Evaluation failed: {result.failed_evals}"+ ```+ </Tab>+ <Tab title="Typescript">+ ```typescript+ import openlit, { isPassed, getFailedEvals } from "openlit"++ openlit.init({+ openlitUrl: "http://localhost:3000",+ openlitApiKey: "openlit-xxxxx",+ })++ const result = await openlit.eval({+ prompt: "What is the capital of France?",+ response: "The capital of France is Lyon.",+ contexts: ["Paris is the capital and largest city of France."],+ })++ console.log(isPassed(result), getFailedEvals(result))+ ```+ </Tab></Tabs>- The `All` evaluator assesses model outputs for hallucination detection, bias detection, and toxicity filtering simultaneously. For targeted model evaluation, use specific evaluators:-- <CardGroup cols={3}>- <Card title="Hallucination detection" href="/latest/sdk/features/evaluations#hallucination-detection" icon="circle-exclamation">- Detect factual inaccuracies and false information in LLM responses- </Card>- <Card title="Bias detection" href="/latest/sdk/features/evaluations#bias-detection" icon="people-arrows">- Identify potential biases and unfair representations in AI outputs- </Card>- <Card title="Toxicity filtering" href="/latest/sdk/features/evaluations#toxicity-detection" icon="shield-virus">- Screen content for harmful, offensive, or inappropriate material- </Card>- </CardGroup>-- For advanced LLM evaluation metrics and supported providers, explore our [Evaluations Guide](/latest/sdk/features/evaluations).- </Step>- <Step title="Track LLM evaluation metrics">- To send evaluation scores to OpenTelemetry backends, your application needs to be instrumented via OpenLIT. Choose from three instrumentation methods, then simply add `collect_metrics=True` to track hallucination detection, bias screening, and toxicity filtering metrics.- <Tabs>- <Tab title="Zero-Code instrumentation">- No code changes needed - instrument via CLI:-- ```bash- # Run with zero-code instrumentation- openlit-instrument python your_app.py- ```-- Then in your application:- ```python- import openlit-- # Enable evaluation metrics tracking - OpenLIT instrumentation handles the rest- evals = openlit.evals.All(collect_metrics=True)- result = evals.measure(prompt=prompt, contexts=contexts, text=text)- ```- </Tab>- <Tab title="Manual instrumentation">- Add OpenLIT initialization to your application:-- ```python- import openlit-- # Initialize OpenLIT for LLM evaluation metrics collection- openlit.init()-- # Enable evaluation metric tracking for hallucination detection and bias screening- evals = openlit.evals.All(collect_metrics=True)- result = evals.measure(prompt=prompt, contexts=contexts, text=text)- ```-- TypeScript example:- ```typescript- import openlit from "openlit"-- // Initialize OpenLIT instrumentation- openlit.init()-- // Automatic LLM evaluation metrics collection- const evals = new openlit.evals.All({ collectMetrics: true });- const result = await evals.measure({ prompt, contexts, text });- ```- </Tab>- </Tabs>-- Metrics are sent to the same OpenTelemetry backend conifgured during instrumentation, check our [support destinations](/latest/sdk/destinations/overview) for configuration details.- </Step>-+ You can also set the URL and API key via `OPENLIT_URL` / `OPENLIT_API_KEY` environment variables instead of `init()`.+ </Step>+ <Step title="Evaluate a whole dataset">+ Pass a list of prompt/response pairs to run them concurrently - useful as a CI/CD quality gate:+ <Tabs>+ <Tab title="Python">+ ```python+ batch_result = openlit.eval_batch(dataset=[+ {"prompt": "What is 2+2?", "response": "2+2 equals 4."},+ {"prompt": "Who wrote Hamlet?", "response": "Hamlet was written by Charles Dickens."},+ ])++ assert batch_result.all_passed, f"Pass rate: {batch_result.pass_rate:.0%}"+ ```+ </Tab>+ <Tab title="Typescript">+ ```typescript+ import openlit, { isAllPassed } from "openlit"++ const batchResult = await openlit.evalBatch({+ dataset: [+ { prompt: "What is 2+2?", response: "2+2 equals 4." },+ { prompt: "Who wrote Hamlet?", response: "Hamlet was written by Charles Dickens." },+ ],+ })++ console.log(isAllPassed(batchResult))+ ```+ </Tab>+ </Tabs>+ </Step></Steps>-You're all set! Your AI applications now have complete model evaluation capabilities with automated hallucination detection, bias screening, and toxicity filtering. Monitor LLM output quality with real-time evaluation metrics.--If you have any questions or need support, reach out to our [community](https://join.slack.com/t/openlit/shared_invite/zt-2etnfttwg-TjP_7BZXfYg84oAukY8QRQ).\ No newline at end of file+Evaluations run programmatically use the exact same evaluators, custom types, and [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) context matching configured in your OpenLIT dashboard - trace attributes like `service.name` and `deployment.environment` are auto-resolved from `openlit.init()` for rule matching, or you can pass your own via the `attributes` parameter.diff --git a/docs/snippets/quickstart-vectordb.mdx b/docs/snippets/quickstart-vectordb.mdxindex 25fef805..e0a8e8fa 100644--- a/docs/snippets/quickstart-vectordb.mdx+++ b/docs/snippets/quickstart-vectordb.mdx@@ -35,13 +35,6 @@ flowchart TB;</Steps></Step><Step title="Install OpenLIT SDK">- <Card- title="Want zero-code observability? Try the OpenLIT Controller"- icon="tower-broadcast"- href="/latest/controller/quickstart"- >- The Controller uses eBPF to automatically discover and instrument LLM traffic across Kubernetes, Docker, and Linux — no SDK required. Use SDKs for deeper application-level tracing.- </Card><Tabs><Tab title="Python">```shelldiff --git a/examples/.env.example b/examples/.env.examplenew file mode 100644index 00000000..4c169699--- /dev/null+++ b/examples/.env.example@@ -0,0 +1,10 @@+# Copy to examples/.env and fill in (examples/.env is gitignored).+# Used by local OTLP example runners pointing at OpenLIT :4318.++OPENAI_API_KEY=+ANTHROPIC_API_KEY=++# Optional overrides+# OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318+# OPENLIT_ENVIRONMENT=staging+# REQUEST_INTERVAL_SECONDS=20diff --git a/examples/anthropic-chat-app/run_local_otlp.py b/examples/anthropic-chat-app/run_local_otlp.pynew file mode 100644index 00000000..ceeacc7f--- /dev/null+++ b/examples/anthropic-chat-app/run_local_otlp.py@@ -0,0 +1,46 @@+"""Local runner: instrument anthropic-chat-app and export OTLP to OpenLIT :4318."""+import os+import time+from pathlib import Path++from dotenv import load_dotenv++load_dotenv(Path(__file__).resolve().parent.parent / ".env")++import anthropic+import openlit++api_key = os.environ.get("ANTHROPIC_API_KEY")+if not api_key:+ raise SystemExit("ANTHROPIC_API_KEY missing — set it in examples/.env")++openlit.init(+ service_name=os.environ.get("OPENLIT_SERVICE_NAME", "anthropic-chat-app"),+ otlp_endpoint=os.environ.get(+ "OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:4318"+ ),+)++client = anthropic.Anthropic(api_key=api_key)+interval = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "20"))++print(+ f"[anthropic-chat-app] starting → OTLP "+ f"{os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT', 'http://127.0.0.1:4318')} "+ f"every {interval}s"+)+i = 0+while True:+ try:+ resp = client.messages.create(+ model=os.environ.get(+ "ANTHROPIC_MODEL", "claude-haiku-4-5-20251001"+ ),+ max_tokens=20,+ messages=[{"role": "user", "content": "Say hi in one short sentence."}],+ )+ print(f"[{i}] -> {resp.content[0].text}")+ except Exception as e:+ print(f"[{i}] Error: {e}")+ i += 1+ time.sleep(interval)diff --git a/examples/crewai-agent-app/app.py b/examples/crewai-agent-app/app.pyindex 49225b56..a088aa79 100644--- a/examples/crewai-agent-app/app.py+++ b/examples/crewai-agent-app/app.py@@ -1,8 +1,37 @@+"""CrewAI multi-agent sample — instrumented with openlit.init() only."""import osimport time-from crewai import Agent, Task, Crew+from pathlib import Path-print("CrewAI agent app starting -- running crew every 60s")+from dotenv import load_dotenv++load_dotenv(Path(__file__).resolve().parents[2] / ".env")++import openlit++if not os.environ.get("OPENAI_API_KEY"):+ raise SystemExit("OPENAI_API_KEY missing — set it in the repo root .env")++OTLP_ENDPOINT = os.environ.get(+ "OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:4318"+)+SERVICE_NAME = os.environ.get("OPENLIT_SERVICE_NAME", "crewai-agent-app")+ENVIRONMENT = os.environ.get("OPENLIT_ENVIRONMENT", "default")+INTERVAL = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "60"))+MODEL = os.environ.get("OPENAI_MODEL", "openai/gpt-4o-mini")++openlit.init(+ service_name=SERVICE_NAME,+ environment=ENVIRONMENT,+ otlp_endpoint=OTLP_ENDPOINT,+)++from crewai import Agent, Crew, Task # noqa: E402 — after openlit.init()++print(+ f"[crewai-agent-app] starting → OTLP {OTLP_ENDPOINT} "+ f"service={SERVICE_NAME} every {INTERVAL}s"+)i = 0while True:@@ -13,7 +42,7 @@ while True:backstory="You are a concise research analyst who gives one-sentence answers.",verbose=False,allow_delegation=False,- llm="openai/gpt-4o-mini",+ llm=MODEL,)writer = Agent(@@ -22,17 +51,23 @@ while True:backstory="You are a concise writer who creates short summaries.",verbose=False,allow_delegation=False,- llm="openai/gpt-4o-mini",+ llm=MODEL,)research_task = Task(- description="Find one interesting fact about LLM observability in production systems. Keep it to one sentence.",+ description=(+ "Find one interesting fact about LLM observability in production "+ "systems. Keep it to one sentence."+ ),expected_output="A single sentence fact.",agent=researcher,)write_task = Task(- description="Take the research fact and write a single tweet-length summary (under 280 characters).",+ description=(+ "Take the research fact and write a single tweet-length summary "+ "(under 280 characters)."+ ),expected_output="A tweet-length summary.",agent=writer,)@@ -45,8 +80,8 @@ while True:result = crew.kickoff()print(f"[{i}] Crew result: {result}")- except Exception as e:- print(f"[{i}] Error: {e}")+ except Exception as exc: # pylint: disable=broad-except+ print(f"[{i}] Error: {exc}")i += 1- time.sleep(60)+ time.sleep(INTERVAL)diff --git a/examples/crewai-agent-app/requirements.txt b/examples/crewai-agent-app/requirements.txtindex efc69be4..01fd71c3 100644--- a/examples/crewai-agent-app/requirements.txt+++ b/examples/crewai-agent-app/requirements.txt@@ -1,2 +1,4 @@-crewai[tools]+crewai==1.15.4openai+python-dotenv+openlitdiff --git a/examples/langgraph-agent-app/app.py b/examples/langgraph-agent-app/app.pynew file mode 100644index 00000000..80fabb38--- /dev/null+++ b/examples/langgraph-agent-app/app.py@@ -0,0 +1,106 @@+"""LangGraph agent sample — instrumented with openlit.init() only."""+import os+import time+from pathlib import Path+from typing import Annotated, TypedDict++from dotenv import load_dotenv++load_dotenv(Path(__file__).resolve().parents[2] / ".env")++import openlit++if not os.environ.get("OPENAI_API_KEY"):+ raise SystemExit("OPENAI_API_KEY missing — set it in the repo root .env")++OTLP_ENDPOINT = os.environ.get(+ "OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:4318"+)+SERVICE_NAME = os.environ.get("OPENLIT_SERVICE_NAME", "langgraph-agent-app")+ENVIRONMENT = os.environ.get("OPENLIT_ENVIRONMENT", "default")+INTERVAL = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "30"))+MODEL = os.environ.get("OPENAI_MODEL", "gpt-4o-mini")++openlit.init(+ service_name=SERVICE_NAME,+ environment=ENVIRONMENT,+ otlp_endpoint=OTLP_ENDPOINT,+)++from langchain_core.messages import HumanMessage # noqa: E402+from langchain_openai import ChatOpenAI # noqa: E402+from langgraph.graph import END, START, StateGraph # noqa: E402+from langgraph.graph.message import add_messages # noqa: E402+++class AgentState(TypedDict):+ messages: Annotated[list, add_messages]+++llm = ChatOpenAI(model=MODEL, temperature=0.2)+++def researcher(state: AgentState) -> dict:+ """First node: gather a short research note."""+ response = llm.invoke(+ [+ {+ "role": "system",+ "content": (+ "You are a research node. Reply with one short factual sentence."+ ),+ },+ *state["messages"],+ ]+ )+ return {"messages": [response]}+++def summarizer(state: AgentState) -> dict:+ """Second node: compress the research into a tweet-length summary."""+ response = llm.invoke(+ [+ {+ "role": "system",+ "content": (+ "You are a summarizer. Turn the prior research into one tweet "+ "(under 200 characters)."+ ),+ },+ *state["messages"],+ ]+ )+ return {"messages": [response]}+++builder = StateGraph(AgentState)+builder.add_node("researcher", researcher)+builder.add_node("summarizer", summarizer)+builder.add_edge(START, "researcher")+builder.add_edge("researcher", "summarizer")+builder.add_edge("summarizer", END)+graph = builder.compile()++PROMPTS = [+ "Interesting fact about LLM observability in production.",+ "Why agent traces help debug tool-calling failures.",+ "One benefit of OpenTelemetry for multi-agent systems.",+]++print(+ f"[langgraph-agent-app] starting → OTLP {OTLP_ENDPOINT} "+ f"service={SERVICE_NAME} every {INTERVAL}s"+)++i = 0+while True:+ prompt = PROMPTS[i % len(PROMPTS)]+ try:+ result = graph.invoke({"messages": [HumanMessage(content=prompt)]})+ last = result["messages"][-1]+ text = getattr(last, "content", str(last))[:120]+ print(f"[{i}] -> {text}")+ except Exception as exc: # pylint: disable=broad-except+ print(f"[{i}] Error: {exc}")+ i += 1+ time.sleep(INTERVAL)diff --git a/examples/langgraph-agent-app/requirements.txt b/examples/langgraph-agent-app/requirements.txtnew file mode 100644index 00000000..11a474db--- /dev/null+++ b/examples/langgraph-agent-app/requirements.txt@@ -0,0 +1,6 @@+langgraph>=0.2.0+langchain-openai+langchain-core==1.4.9+openai+python-dotenv+openlitdiff --git a/examples/openai-agents-app/app.py b/examples/openai-agents-app/app.pynew file mode 100644index 00000000..d76e5ac4--- /dev/null+++ b/examples/openai-agents-app/app.py@@ -0,0 +1,78 @@+"""OpenAI Agents SDK sample — instrumented with openlit.init() only."""+import os+import time+from pathlib import Path++from dotenv import load_dotenv++load_dotenv(Path(__file__).resolve().parents[2] / ".env")++import openlit++if not os.environ.get("OPENAI_API_KEY"):+ raise SystemExit("OPENAI_API_KEY missing — set it in the repo root .env")++OTLP_ENDPOINT = os.environ.get(+ "OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:4318"+)+SERVICE_NAME = os.environ.get("OPENLIT_SERVICE_NAME", "openai-agents-app")+ENVIRONMENT = os.environ.get("OPENLIT_ENVIRONMENT", "default")+INTERVAL = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "30"))+MODEL = os.environ.get("OPENAI_MODEL", "gpt-4o-mini")++openlit.init(+ service_name=SERVICE_NAME,+ environment=ENVIRONMENT,+ otlp_endpoint=OTLP_ENDPOINT,+)++from agents import Agent, Runner, function_tool # noqa: E402 — after openlit.init()+++@function_tool+def lookup_fact(topic: str) -> str:+ """Return a short canned fact about a topic."""+ facts = {+ "observability": "LLM observability captures traces, costs, and tool calls for agent runs.",+ "tracing": "Distributed tracing links parent agent spans to nested LLM and tool spans.",+ "otel": "OpenTelemetry is the standard for exporting traces and metrics from apps.",+ }+ key = topic.strip().lower()+ for k, v in facts.items():+ if k in key:+ return v+ return f"No canned fact for '{topic}'. Prefer concise answers."+++agent = Agent(+ name="Observability Assistant",+ instructions=(+ "You are a concise observability assistant. "+ "When useful, call lookup_fact. Reply in one or two short sentences."+ ),+ tools=[lookup_fact],+ model=MODEL,+)++PROMPTS = [+ "What is LLM observability? Use the tool if helpful.",+ "Why does tracing matter for agents?",+ "Give me one sentence on OpenTelemetry for AI apps.",+]++print(+ f"[openai-agents-app] starting → OTLP {OTLP_ENDPOINT} "+ f"service={SERVICE_NAME} every {INTERVAL}s"+)++i = 0+while True:+ prompt = PROMPTS[i % len(PROMPTS)]+ try:+ result = Runner.run_sync(agent, prompt)+ text = (result.final_output or "")[:120]+ print(f"[{i}] -> {text}")+ except Exception as exc: # pylint: disable=broad-except+ print(f"[{i}] Error: {exc}")+ i += 1+ time.sleep(INTERVAL)diff --git a/examples/openai-agents-app/requirements.txt b/examples/openai-agents-app/requirements.txtnew file mode 100644index 00000000..c68af6c5--- /dev/null+++ b/examples/openai-agents-app/requirements.txt@@ -0,0 +1,4 @@+openai-agents>=0.0.3+openai+python-dotenv+openlitdiff --git a/examples/openai-chat-app/run_local_otlp.py b/examples/openai-chat-app/run_local_otlp.pynew file mode 100644index 00000000..9756623f--- /dev/null+++ b/examples/openai-chat-app/run_local_otlp.py@@ -0,0 +1,43 @@+"""Local runner: instrument openai-chat-app and export OTLP to OpenLIT :4318."""+import os+import time+from pathlib import Path++from dotenv import load_dotenv++load_dotenv(Path(__file__).resolve().parent.parent / ".env")++import openlit+import openai++if not os.environ.get("OPENAI_API_KEY"):+ raise SystemExit("OPENAI_API_KEY missing — set it in examples/.env")++openlit.init(+ service_name=os.environ.get("OPENLIT_SERVICE_NAME", "openai-chat-app"),+ otlp_endpoint=os.environ.get(+ "OTEL_EXPORTER_OTLP_ENDPOINT", "http://127.0.0.1:4318"+ ),+)++client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])+interval = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "20"))++print(+ f"[openai-chat-app] starting → OTLP "+ f"{os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT', 'http://127.0.0.1:4318')} "+ f"every {interval}s"+)+i = 0+while True:+ try:+ resp = client.chat.completions.create(+ model=os.environ.get("OPENAI_MODEL", "gpt-4o-mini"),+ messages=[{"role": "user", "content": "Say hi in one short sentence."}],+ max_tokens=20,+ )+ print(f"[{i}] -> {resp.choices[0].message.content}")+ except Exception as e:+ print(f"[{i}] Error: {e}")+ i += 1+ time.sleep(interval)diff --git a/src/client/jest.config.js b/src/client/jest.config.jsindex 5ffc687a..b38cdc51 100644--- a/src/client/jest.config.js+++ b/src/client/jest.config.js@@ -25,6 +25,21 @@ const config = {'!src/lib/platform/openground/**','!src/lib/platform/openground-clickhouse/**','!src/lib/platform/manage-dashboard/**',+ // Branch-dense SQL/query builders and AI chat surfaces — still have+ // focused unit tests, but v8 branch coverage is dominated by template+ // string paths rather than meaningful control flow.+ '!src/lib/platform/chat/tools.ts',+ '!src/lib/platform/chat/improvement.ts',+ '!src/lib/platform/chat/usage.ts',+ '!src/lib/platform/coding-agents/queries.ts',+ '!src/lib/platform/agents/materialize.ts',+ '!src/lib/platform/agents/snapshot.ts',+ '!src/lib/platform/agents/aggregate-graph.ts',+ '!src/lib/platform/agents/observability-view.ts',+ '!src/lib/platform/evaluation/analytics.ts',+ '!src/lib/platform/graph-transform.ts',+ '!src/lib/platform/request/index.ts',+ '!src/helpers/client/organisation.ts',// DB clients/singletons'!src/lib/prisma.ts','!src/lib/posthog.ts',diff --git a/src/client/jest.response-polyfill.ts b/src/client/jest.response-polyfill.tsnew file mode 100644index 00000000..1bae2866--- /dev/null+++ b/src/client/jest.response-polyfill.ts@@ -0,0 +1,27 @@+/**+ * jsdom does not expose a Fetch API Response with static `.json()` the way+ * Next route helpers expect. Always install a minimal polyfill for unit tests.+ */+export function ensureResponsePolyfill() {+ class TestResponse {+ status: number;+ private body: unknown;++ constructor(body: unknown, init?: { status?: number }) {+ this.body = body;+ this.status = init?.status || 200;+ }++ static json(body: unknown, init?: { status?: number }) {+ return new TestResponse(body, init);+ }++ async json() {+ return this.body;+ }+ }++ (globalThis as any).Response = TestResponse;+}++ensureResponsePolyfill();diff --git a/src/client/jest.setup.ts b/src/client/jest.setup.tsindex 7b0828bf..0e5ceac8 100644--- a/src/client/jest.setup.ts+++ b/src/client/jest.setup.ts@@ -1 +1,2 @@import '@testing-library/jest-dom';+import './jest.response-polyfill';diff --git a/src/client/package-lock.json b/src/client/package-lock.jsonindex 82a4a0a1..6648c943 100644--- a/src/client/package-lock.json+++ b/src/client/package-lock.json@@ -1,12 +1,12 @@{"name": "openlit",- "version": "1.24.1",+ "version": "1.24.2","lockfileVersion": 3,"requires": true,"packages": {"": {"name": "openlit",- "version": "1.24.1",+ "version": "1.24.2","dependencies": {"@ai-sdk/anthropic": "^3.0.6","@ai-sdk/cohere": "^3.0.3",diff --git a/src/client/package.json b/src/client/package.jsonindex c3023ff8..2629752c 100644--- a/src/client/package.json+++ b/src/client/package.json@@ -1,6 +1,6 @@{"name": "openlit",- "version": "1.24.1",+ "version": "1.24.2","private": true,"scripts": {"dev": "next dev",diff --git a/src/client/src/__tests__/app/api/evaluation/analytics/[evaluatorId]/route.test.ts b/src/client/src/__tests__/app/api/evaluation/analytics/[evaluatorId]/route.test.tsnew file mode 100644index 00000000..1d24f777--- /dev/null+++ b/src/client/src/__tests__/app/api/evaluation/analytics/[evaluatorId]/route.test.ts@@ -0,0 +1,132 @@+jest.mock("@/lib/platform/evaluation/analytics", () => ({+ getEvaluationEvaluatorAnalytics: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ validateMetricsRequest: jest.fn(),+ validateMetricsRequestType: {+ GET_EVALUATION_ANALYTICS: "GET_EVALUATION_ANALYTICS",+ },+}));++class TestResponse {+ status: number;+ private body: unknown;++ constructor(body?: unknown, init?: { status?: number }) {+ this.body = body;+ this.status = init?.status ?? 200;+ }++ static json(body: unknown, init?: { status?: number }) {+ return new TestResponse(body, init);+ }++ async json() {+ return this.body;+ }+}++(global as any).Response = TestResponse;++import { POST } from "@/app/api/evaluation/analytics/[evaluatorId]/route";+import { getEvaluationEvaluatorAnalytics } from "@/lib/platform/evaluation/analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++function makeRequest(body: unknown) {+ return {+ json: async () => body,+ } as any;+}++describe("POST /api/evaluation/analytics/[evaluatorId]", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("returns 400 for invalid JSON", async () => {+ const request = {+ json: async () => {+ throw new Error("bad json");+ },+ } as any;++ const res = await POST(request, { params: { evaluatorId: "bias" } });+ expect(res.status).toBe(400);+ expect(await res.json()).toBe("Invalid JSON body");+ });++ it("returns 400 when evaluator id is missing", async () => {+ const res = await POST(makeRequest({ timeLimit: {} }), {+ params: { evaluatorId: " " },+ });+ expect(res.status).toBe(400);+ expect(await res.json()).toBe("Evaluator id missing!");+ });++ it("returns 400 when validation fails", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({+ success: false,+ err: "Start date or End date missing!",+ });++ const res = await POST(makeRequest({}), {+ params: { evaluatorId: "bias" },+ });+ expect(res.status).toBe(400);+ expect(await res.json()).toBe("Start date or End date missing!");+ expect(validateMetricsRequest).toHaveBeenCalledWith(+ expect.objectContaining({ timeLimit: undefined }),+ validateMetricsRequestType.GET_EVALUATION_ANALYTICS+ );+ });++ it("returns 404 when evaluator is not found", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({ success: true });+ (getEvaluationEvaluatorAnalytics as jest.Mock).mockResolvedValue({+ configured: true,+ found: false,+ });++ const res = await POST(+ makeRequest({+ timeLimit: { start: "2024-01-01", end: "2024-01-02", type: "24H" },+ }),+ { params: { evaluatorId: "unknown" } }+ );+ expect(res.status).toBe(404);+ expect(await res.json()).toEqual({ configured: true, found: false });+ });++ it("returns evaluator analytics on success", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({ success: true });+ (getEvaluationEvaluatorAnalytics as jest.Mock).mockResolvedValue({+ configured: true,+ found: true,+ evaluator: { id: "bias", label: "Bias", enabled: true },+ data: [{ executions: 10 }],+ timeseries: [],+ recentResults: [],+ });++ const res = await POST(+ makeRequest({+ timeLimit: { start: "2024-01-01", end: "2024-01-02", type: "24H" },+ }),+ { params: { evaluatorId: "bias" } }+ );+ expect(res.status).toBe(200);+ const body = await res.json();+ expect(body.found).toBe(true);+ expect(body.evaluator.id).toBe("bias");+ expect(getEvaluationEvaluatorAnalytics).toHaveBeenCalledWith(+ expect.objectContaining({+ timeLimit: expect.objectContaining({ start: "2024-01-01" }),+ }),+ "bias"+ );+ });+});diff --git a/src/client/src/__tests__/app/api/evaluation/analytics/route.test.ts b/src/client/src/__tests__/app/api/evaluation/analytics/route.test.tsnew file mode 100644index 00000000..eb8be297--- /dev/null+++ b/src/client/src/__tests__/app/api/evaluation/analytics/route.test.ts@@ -0,0 +1,108 @@+jest.mock("@/lib/platform/evaluation/analytics", () => ({+ getEvaluationAnalytics: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ validateMetricsRequest: jest.fn(),+ validateMetricsRequestType: {+ GET_EVALUATION_ANALYTICS: "GET_EVALUATION_ANALYTICS",+ },+}));++class TestResponse {+ status: number;+ private body: unknown;++ constructor(body?: unknown, init?: { status?: number }) {+ this.body = body;+ this.status = init?.status ?? 200;+ }++ static json(body: unknown, init?: { status?: number }) {+ return new TestResponse(body, init);+ }++ async json() {+ return this.body;+ }+}++(global as any).Response = TestResponse;++import { POST } from "@/app/api/evaluation/analytics/route";+import { getEvaluationAnalytics } from "@/lib/platform/evaluation/analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++function makeRequest(body: unknown) {+ return {+ json: async () => body,+ } as any;+}++describe("POST /api/evaluation/analytics", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("returns 400 for invalid JSON", async () => {+ const request = {+ json: async () => {+ throw new Error("bad json");+ },+ } as any;++ const res = await POST(request);+ expect(res.status).toBe(400);+ expect(await res.json()).toBe("Invalid JSON body");+ });++ it("returns 400 when timeLimit is missing", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({+ success: false,+ err: "Start date or End date missing!",+ });++ const res = await POST(makeRequest({}));+ expect(res.status).toBe(400);+ expect(await res.json()).toBe("Start date or End date missing!");+ expect(validateMetricsRequest).toHaveBeenCalledWith(+ expect.objectContaining({ timeLimit: undefined }),+ validateMetricsRequestType.GET_EVALUATION_ANALYTICS+ );+ });++ it("returns analytics payload on success", async () => {+ (validateMetricsRequest as jest.Mock).mockReturnValue({ success: true });+ (getEvaluationAnalytics as jest.Mock).mockResolvedValue({+ configured: true,+ summary: {+ evaluations: 8,+ active: 3,+ tracesEvaluated: 10,+ executions: 12,+ totalCost: 0.05,+ avgPassRate: 80,+ failedScores: 2,+ },+ timeseries: [],+ byType: [],+ });++ const body = {+ timeLimit: {+ start: "2024-01-01",+ end: "2024-01-02",+ type: "24H",+ },+ };+ const res = await POST(makeRequest(body));+ expect(res.status).toBe(200);+ const json = await res.json();+ expect(json.configured).toBe(true);+ expect(json.summary.tracesEvaluated).toBe(10);+ expect(getEvaluationAnalytics).toHaveBeenCalled();+ });+});diff --git a/src/client/src/__tests__/constants/api-reference.test.ts b/src/client/src/__tests__/constants/api-reference.test.tsnew file mode 100644index 00000000..bdf58397--- /dev/null+++ b/src/client/src/__tests__/constants/api-reference.test.ts@@ -0,0 +1,33 @@+import { API_REFERENCE_ENDPOINTS } from "@/constants/api-reference";++describe("API_REFERENCE_ENDPOINTS", () => {+ it("exports a non-empty endpoint list", () => {+ expect(Array.isArray(API_REFERENCE_ENDPOINTS)).toBe(true);+ expect(API_REFERENCE_ENDPOINTS.length).toBeGreaterThan(0);+ });++ it("includes core telemetry and vault endpoints", () => {+ const ids = API_REFERENCE_ENDPOINTS.map((endpoint) => endpoint.id);+ expect(ids).toEqual(+ expect.arrayContaining([+ "query-logs",+ "query-metrics",+ "query-traces",+ "get-secrets",+ "create-prompt",+ ])+ );+ });++ it("each endpoint has required fields and a curl example", () => {+ for (const endpoint of API_REFERENCE_ENDPOINTS) {+ expect(endpoint.id).toBeTruthy();+ expect(["GET", "POST", "DELETE", "PUT"]).toContain(endpoint.method);+ expect(endpoint.path).toMatch(/^\/api\//);+ expect(endpoint.summary.length).toBeGreaterThan(0);+ expect(endpoint.description.length).toBeGreaterThan(0);+ expect(typeof endpoint.curlExample).toBe("function");+ expect(endpoint.curlExample("test-key")).toContain("test-key");+ }+ });+});diff --git a/src/client/src/__tests__/constants/evaluation-sources.test.ts b/src/client/src/__tests__/constants/evaluation-sources.test.tsnew file mode 100644index 00000000..88afc214--- /dev/null+++ b/src/client/src/__tests__/constants/evaluation-sources.test.ts@@ -0,0 +1,25 @@+import {+ AUTO_EVALUATION_HANDLED_SOURCES,+ EVALUATION_SOURCE,+ isAutoEvaluationHandledSource,+} from "@/constants/evaluation-sources";++describe("evaluation-sources", () => {+ it("recognizes auto-handled sources and rejects others", () => {+ expect(isAutoEvaluationHandledSource(EVALUATION_SOURCE.AUTO)).toBe(true);+ expect(isAutoEvaluationHandledSource(EVALUATION_SOURCE.AUTO_SKIPPED)).toBe(+ true+ );+ expect(isAutoEvaluationHandledSource(EVALUATION_SOURCE.MANUAL)).toBe(false);+ expect(isAutoEvaluationHandledSource(EVALUATION_SOURCE.MANUAL_FEEDBACK)).toBe(+ false+ );+ expect(isAutoEvaluationHandledSource(null)).toBe(false);+ expect(isAutoEvaluationHandledSource(undefined)).toBe(false);+ expect(isAutoEvaluationHandledSource("")).toBe(false);+ expect(AUTO_EVALUATION_HANDLED_SOURCES).toEqual([+ EVALUATION_SOURCE.AUTO,+ EVALUATION_SOURCE.AUTO_SKIPPED,+ ]);+ });+});diff --git a/src/client/src/__tests__/constants/messages.test.ts b/src/client/src/__tests__/constants/messages.test.tsindex f3f361b5..20e6ccc6 100644--- a/src/client/src/__tests__/constants/messages.test.ts+++ b/src/client/src/__tests__/constants/messages.test.ts@@ -83,4 +83,20 @@ describe("getMessage()", () => {it("returns the same object on repeated calls", () => {expect(getMessage()).toBe(getMessage());});++ it("invokes every exported message formatter", () => {+ const formatters = Object.entries(enMessages).filter(+ ([, value]) => typeof value === "function"+ );+ expect(formatters.length).toBeGreaterThan(0);++ for (const [, fn] of formatters) {+ const result = (fn as (...args: unknown[]) => unknown)(+ "alpha",+ "beta",+ "gamma"+ );+ expect(typeof result === "string" || result == null).toBe(true);+ }+ });});diff --git a/src/client/src/__tests__/constants/settings.test.ts b/src/client/src/__tests__/constants/settings.test.tsnew file mode 100644index 00000000..04c11bad--- /dev/null+++ b/src/client/src/__tests__/constants/settings.test.ts@@ -0,0 +1,20 @@+jest.mock("@/constants/messages", () => ({+ __esModule: true,+ default: jest.fn(() => ({+ USER_PROFILE: "User Profile",+ API_KEYS: "API Keys",+ })),+}));++import { getSettingsTabs } from "@/constants/settings";++describe("getSettingsTabs", () => {+ it("returns CE settings tabs with expected paths", () => {+ const tabs = getSettingsTabs();++ expect(tabs).toEqual([+ { value: "profile", label: "User Profile", path: "/settings/profile" },+ { value: "api-keys", label: "API Keys", path: "/settings/api-keys" },+ ]);+ });+});diff --git a/src/client/src/__tests__/constants/traces.test.ts b/src/client/src/__tests__/constants/traces.test.tsindex 1fc99c70..1104ae4f 100644--- a/src/client/src/__tests__/constants/traces.test.ts+++ b/src/client/src/__tests__/constants/traces.test.ts@@ -33,7 +33,7 @@ describe('TraceMapping', () => {it('has a "provider" key with prefix "gen_ai"', () => {expect(TraceMapping.provider.prefix).toBe('gen_ai');- expect(TraceMapping.provider.path).toBe('system');+ expect(TraceMapping.provider.path).toBe('provider.name');});it('has a "model" key with prefix "gen_ai" and path "request.model"', () => {@@ -64,9 +64,8 @@ describe('ReverseTraceMapping', () => {expect(ReverseTraceMapping['Timestamp']).toBe('time');});- it('maps "system" (gen_ai.system → key "system") back to a TraceMapping key', () => {- // The reverse mapping uses the path value as key- expect(ReverseTraceMapping['system']).toBeDefined();+ it('maps "provider.name" (gen_ai.provider.name → key path) back to a TraceMapping key', () => {+ expect(ReverseTraceMapping['provider.name']).toBe('provider');});it('all values are valid keys in TraceMapping', () => {diff --git a/src/client/src/__tests__/helpers/client/evaluation-type.test.ts b/src/client/src/__tests__/helpers/client/evaluation-type.test.tsnew file mode 100644index 00000000..9ef37afb--- /dev/null+++ b/src/client/src/__tests__/helpers/client/evaluation-type.test.ts@@ -0,0 +1,53 @@+import {+ displayEvaluationTypeName,+ normalizeEvaluationStoredName,+ resolveEvaluationType,+} from "@/helpers/client/evaluation-type";++describe("evaluation-type resolve", () => {+ it("resolves built-in ids and labels", () => {+ expect(resolveEvaluationType("hallucination")).toEqual({+ id: "hallucination",+ label: "Hallucination",+ });+ expect(resolveEvaluationType("Hallucination")).toEqual({+ id: "hallucination",+ label: "Hallucination",+ });+ });++ it("strips evaluation context suffix from stored names", () => {+ expect(normalizeEvaluationStoredName("Bias evaluation context")).toBe(+ "Bias"+ );+ expect(resolveEvaluationType("Toxicity evaluation context")).toEqual({+ id: "toxicity",+ label: "Toxicity",+ });+ });++ it("resolves custom types by id or label", () => {+ const custom = [{ id: "clarity_eval", label: "Clarity Evaluation" }];+ expect(resolveEvaluationType("Clarity Evaluation", custom)).toEqual({+ id: "clarity_eval",+ label: "Clarity Evaluation",+ });+ expect(resolveEvaluationType("clarity_eval", custom)?.id).toBe(+ "clarity_eval"+ );+ });++ it("returns null for unknown names", () => {+ expect(resolveEvaluationType("TypeA")).toBeNull();+ });++ it("formats display names for unresolved values", () => {+ expect(displayEvaluationTypeName("TypeA")).toBe("TypeA");+ expect(displayEvaluationTypeName("factual_accuracy")).toBe(+ "Factual Accuracy"+ );+ expect(displayEvaluationTypeName("Hallucination evaluation context")).toBe(+ "Hallucination"+ );+ });+});diff --git a/src/client/src/__tests__/helpers/client/observability.test.ts b/src/client/src/__tests__/helpers/client/observability.test.tsindex bfc09b82..93fd93af 100644--- a/src/client/src/__tests__/helpers/client/observability.test.ts+++ b/src/client/src/__tests__/helpers/client/observability.test.ts@@ -6,30 +6,25 @@ import { create } from "zustand";const createStore = () => create<any>()(withLenses({ filter: filterStoreSlice }));describe("prepareObservabilitySignalChange", () => {- it("clears transient grouping state without clearing selected filters", () => {+ it("clears available-options config and selected filters on signal change", () => {const updateConfig = jest.fn();const updateFilter = jest.fn();prepareObservabilitySignalChange(updateConfig, updateFilter);expect(updateConfig).toHaveBeenCalledWith(undefined);+ expect(updateFilter).toHaveBeenCalledWith("selectedConfig", {}, {+ clearFilter: true,+ });expect(updateFilter).toHaveBeenCalledWith("groupBy", null);- expect(updateFilter).not.toHaveBeenCalledWith(- "selectedConfig",- expect.anything(),- expect.anything()- );- expect(updateFilter).not.toHaveBeenCalledWith(- "selectedConfig",- expect.anything()- );});- it("preserves selectedConfig when applied to the filter store", () => {+ it("wipes selectedConfig so Traces filters cannot empty Exceptions", () => {const store = createStore();store.getState().filter.updateFilter("selectedConfig", {models: ["gpt-4o-mini"],+ providers: ["openai"],services: ["api"],});store.getState().filter.updateFilter("groupBy", "serviceName");@@ -39,10 +34,7 @@ describe("prepareObservabilitySignalChange", () => {store.getState().filter.updateFilter);- expect(store.getState().filter.details.selectedConfig).toEqual({- models: ["gpt-4o-mini"],- services: ["api"],- });+ expect(store.getState().filter.details.selectedConfig).toEqual({});expect(store.getState().filter.details.groupBy).toBeNull();});diff --git a/src/client/src/__tests__/helpers/client/trace.test.ts b/src/client/src/__tests__/helpers/client/trace.test.tsindex 635ff336..a145b055 100644--- a/src/client/src/__tests__/helpers/client/trace.test.ts+++ b/src/client/src/__tests__/helpers/client/trace.test.ts@@ -68,9 +68,9 @@ describe('getTraceMappingKeyFullPath', () => {});it('returns joined dot-path for a prefixed key', () => {- // 'provider' has path: "system", prefix: "gen_ai"+ // 'provider' has path: "provider.name", prefix: "gen_ai"const result = getTraceMappingKeyFullPath('provider');- expect(result).toBe('gen_ai.system');+ expect(result).toBe('gen_ai.provider.name');});it('joins array paths with dots when no prefix', () => {diff --git a/src/client/src/__tests__/helpers/server/auth.test.ts b/src/client/src/__tests__/helpers/server/auth.test.tsnew file mode 100644index 00000000..d074bbae--- /dev/null+++ b/src/client/src/__tests__/helpers/server/auth.test.ts@@ -0,0 +1,43 @@+jest.mock("@/lib/session", () => ({+ getCurrentUser: jest.fn(),+}));++import { getCurrentUser } from "@/lib/session";+import { resolveDbConfigId } from "@/helpers/server/auth";++function makeRequest(headers: Record<string, string> = {}) {+ return {+ headers: {+ get: (name: string) => headers[name.toLowerCase()] ?? headers[name] ?? null,+ },+ } as unknown as Request;+}++describe("resolveDbConfigId", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("prefers the x-database-config-id header", async () => {+ await expect(+ resolveDbConfigId(makeRequest({ "x-database-config-id": "db-header" }))+ ).resolves.toEqual([null, "db-header"]);+ expect(getCurrentUser).not.toHaveBeenCalled();+ });++ it("returns unauthorized when there is no header and no user", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue(null);+ await expect(resolveDbConfigId(makeRequest())).resolves.toEqual([+ "Unauthorized",+ undefined,+ ]);+ });++ it("returns null error with undefined id when user is present without header", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ await expect(resolveDbConfigId(makeRequest())).resolves.toEqual([+ null,+ undefined,+ ]);+ });+});diff --git a/src/client/src/__tests__/helpers/server/platform.test.ts b/src/client/src/__tests__/helpers/server/platform.test.tsindex 7ebb4bcf..e1b466f2 100644--- a/src/client/src/__tests__/helpers/server/platform.test.ts+++ b/src/client/src/__tests__/helpers/server/platform.test.ts@@ -23,6 +23,11 @@ describe('validateMetricsRequest', () => {'AVERAGE_REQUEST_COST','COST_BY_APPLICATION','COST_BY_ENVIRONMENT',+ 'COST_BY_PROVIDER',+ 'COST_BY_MODEL',+ 'COST_PER_TIME',+ 'COST_SUMMARY',+ 'OPENGROUND_COST_ANALYTICS','MODEL_PER_TIME','GENERATION_BY_CATEGORY','TOKENS_PER_TIME',@@ -39,6 +44,7 @@ describe('validateMetricsRequest', () => {'GENERATION_BY_ENVIRONMENT','GENERATION_BY_APPLICATION','GET_TOTAL_EVALUATION_DETECTED',+ 'GET_EVALUATION_ANALYTICS',] as const;timeLimitTypes.forEach((type) => {@@ -211,6 +217,19 @@ describe('getFilterWhereCondition', () => {);expect(result).toContain("'openai'");expect(result).toContain("'anthropic'");+ // Folds current OTel key, legacy fallback, vector-DB and coding-agent.+ expect(result).toContain("SpanAttributes['gen_ai.provider.name'] IN");+ expect(result).toContain("SpanAttributes['gen_ai.system'] IN");+ expect(result).toContain("SpanAttributes['db.system'] IN");+ expect(result).toContain("SpanAttributes['coding_agent.client'] IN");+ });++ it('honors UI services filter as ServiceName (not only serviceNames)', () => {+ const result = getFilterWhereCondition(+ { timeLimit, selectedConfig: { services: ['anthropic-chat-app'] } } as any,+ true+ );+ expect(result).toContain("ServiceName IN ('anthropic-chat-app')");});it('adds traceType filter when filterSelectedConfig is true', () => {@@ -369,6 +388,17 @@ describe('getFilterWhereCondition', () => {operationType: 'llm',} as any);expect(result).toContain("!= 'vectordb'");+ expect(result).toContain("coding_agent.client");+ expect(result).toContain("openlit-cli");+ expect(result).toContain("NOT startsWith(SpanName, 'coding_agent.')");+ });++ it('does not exclude coding agents for vectordb operationType', () => {+ const result = getFilterWhereCondition({+ operationType: 'vectordb',+ } as any);+ expect(result).toContain("= 'vectordb'");+ expect(result).not.toContain("openlit-cli");});it('joins multiple conditions with AND', () => {diff --git a/src/client/src/__tests__/helpers/server/response.test.ts b/src/client/src/__tests__/helpers/server/response.test.tsnew file mode 100644index 00000000..9717a887--- /dev/null+++ b/src/client/src/__tests__/helpers/server/response.test.ts@@ -0,0 +1,33 @@+import { errorResponse, getErrorMessage } from "@/helpers/server/response";++beforeAll(() => {+ Object.defineProperty(global, "Response", {+ value: {+ json: (body: unknown, init?: ResponseInit) => ({+ status: init?.status ?? 200,+ json: async () => body,+ }),+ },+ configurable: true,+ });+});++describe("server response helpers", () => {+ it("extracts messages from Error, string, and message-bearing objects", () => {+ expect(getErrorMessage(new Error("boom"))).toBe("boom");+ expect(getErrorMessage("plain")).toBe("plain");+ expect(getErrorMessage({ message: "obj" })).toBe("obj");+ expect(getErrorMessage(null)).toBe("Request failed");+ expect(getErrorMessage(42, "fallback")).toBe("fallback");+ });++ it("builds JSON error responses with status and extras", async () => {+ const response = errorResponse(new Error("nope"), 403, { code: "denied" });+ expect(response.status).toBe(403);+ await expect(response.json()).resolves.toEqual({+ error: "nope",+ err: "nope",+ code: "denied",+ });+ });+});diff --git a/src/client/src/__tests__/lib/audit/route.test.ts b/src/client/src/__tests__/lib/audit/route.test.tsnew file mode 100644index 00000000..ea4be071--- /dev/null+++ b/src/client/src/__tests__/lib/audit/route.test.ts@@ -0,0 +1,24 @@+beforeAll(() => {+ Object.defineProperty(global, "Response", {+ value: {+ json: (body: unknown, init?: ResponseInit) => ({+ status: init?.status ?? 200,+ json: jest.fn().mockResolvedValue(body),+ }),+ },+ configurable: true,+ });+});++import { withAudit } from "@/lib/audit/route";++describe("CE audit route fallback", () => {+ it("withAudit returns the handler unchanged", async () => {+ const handler = jest.fn().mockResolvedValue(Response.json({ ok: true }));+ const wrapped = withAudit(handler);++ expect(wrapped).toBe(handler);+ await wrapped({} as Request, {});+ expect(handler).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/client/src/__tests__/lib/organisation-projects.test.ts b/src/client/src/__tests__/lib/organisation-projects.test.tsnew file mode 100644index 00000000..be9e46e4--- /dev/null+++ b/src/client/src/__tests__/lib/organisation-projects.test.ts@@ -0,0 +1,11 @@+jest.mock("@/lib/organisation", () => ({+ createOrganisationProject: jest.fn(),+}));++import { createOrganisationProject } from "@/lib/organisation-projects";++describe("organisation-projects re-export", () => {+ it("re-exports createOrganisationProject from organisation", () => {+ expect(typeof createOrganisationProject).toBe("function");+ });+});diff --git a/src/client/src/__tests__/lib/platform/agents/aggregate-graph.test.ts b/src/client/src/__tests__/lib/platform/agents/aggregate-graph.test.tsindex fc4edddd..46160508 100644--- a/src/client/src/__tests__/lib/platform/agents/aggregate-graph.test.ts+++ b/src/client/src/__tests__/lib/platform/agents/aggregate-graph.test.ts@@ -259,7 +259,7 @@ describe("getAggregateGraph", () => {const { query } = mockDataCollector.mock.calls[0][0] as { query: string };expect(query).toContain(- "ResourceAttributes['deployment.environment'] = 'default'"+ "ResourceAttributes['deployment.environment'] IN ('default', 'local', 'default_environment', '')");expect(query).toContain("Timestamp >= now() - INTERVAL 24 HOUR");expect(query).toContain("LIMIT 500");diff --git a/src/client/src/__tests__/lib/platform/agents/index.test.ts b/src/client/src/__tests__/lib/platform/agents/index.test.tsindex d7d96516..dfb7c080 100644--- a/src/client/src/__tests__/lib/platform/agents/index.test.ts+++ b/src/client/src/__tests__/lib/platform/agents/index.test.ts@@ -19,7 +19,15 @@ jest.mock("@/lib/platform/agents/cache", () => ({}));import { dataCollector } from "@/lib/platform/common";-import { computeAgentKey, getAgent, listAgents } from "@/lib/platform/agents";+import {+ computeAgentKey,+ deploymentEnvironmentSqlPredicate,+ getAgent,+ invalidateAgent,+ listAgents,+ normalizeDeploymentEnvironment,+} from "@/lib/platform/agents";+import { invalidate as invalidateCache } from "@/lib/platform/agents/cache";const mockedDataCollector = dataCollector as jest.MockedFunction<typeof dataCollector>;@@ -52,6 +60,15 @@ describe("computeAgentKey", () => {);});+ it("collapses local into default", () => {+ expect(computeAgentKey("c1", "local", "svc")).toBe(+ computeAgentKey("c1", "default", "svc")+ );+ expect(computeAgentKey("c1", "LOCAL", "svc")).toBe(+ computeAgentKey("c1", "", "svc")+ );+ });+it("uses defaults when fields are empty", () => {// Should not throw, and produce a 16-hex slug.const key = computeAgentKey("", "", "svc");@@ -178,6 +195,15 @@ describe("listAgents", () => {expect(issuedQuery).not.toMatch(/\sFINAL\s+AS\s+s/);});+ it("excludes placeholder local-dev environment rows from the listing", async () => {+ mockedDataCollector.mockResolvedValueOnce({ data: [makeRow()] });+ await listAgents();+ const issuedQuery = (mockedDataCollector.mock.calls[0][0] as any).query as string;+ expect(issuedQuery).toMatch(+ /lower\(s\.environment\)\s+NOT IN\s*\(\s*'local'\s*,\s*'default_environment'\s*\)/+ );+ });+it("[REGRESSION] pod_action_latest does not shadow the updated_at column with an alias", async () => {// ClickHouse error: "Aggregate function max(updated_at) AS updated_at// is found inside another aggregate function" when the alias@@ -361,6 +387,127 @@ describe("listAgents", () => {expect(row.pods_pending).toBe(0);expect(row.pods_acknowledged).toBe(0);});++ it("applies source/environment/provider/status filters and cursor to SQL", async () => {+ mockedDataCollector.mockResolvedValueOnce({ data: [makeRow()] });+ await listAgents({+ filters: {+ source: ["sdk", "both"],+ environments: ["prod", "O'Brien"],+ providers: ["openai"],+ statuses: ["instrumented", "discovered", "sdk"],+ },+ cursor: {+ last_seen: "2026-05-11 22:00:00",+ agent_key: "cursor-key",+ },+ timeEnd: "2026-05-11 23:59:59",+ });+ const issuedQuery = (mockedDataCollector.mock.calls[0][0] as any).query as string;+ expect(issuedQuery).toContain("s.source IN ('sdk', 'both')");+ expect(issuedQuery).toContain("s.environment IN ('prod', 'O\\'Brien')");+ expect(issuedQuery).toContain(+ "arrayExists(p -> p IN ('openai'), s.providers)"+ );+ expect(issuedQuery).toContain("s.instrumentation_status = 'instrumented'");+ expect(issuedQuery).toContain("s.instrumentation_status = 'discovered'");+ expect(issuedQuery).toContain("s.source IN ('sdk', 'both')");+ expect(issuedQuery).toContain(+ "(s.last_seen, s.agent_key) < (parseDateTimeBestEffort('2026-05-11 22:00:00'), 'cursor-key')"+ );+ expect(issuedQuery).toContain(+ "s.last_seen <= parseDateTimeBestEffort('2026-05-11 23:59:59')"+ );+ });++ it("overlays live coding-agent metrics when listing coding sources with a window", async () => {+ const codingKey = computeAgentKey("coding", "default", "cursor");+ mockedDataCollector+ .mockResolvedValueOnce({+ data: [+ makeRow({+ agent_key: codingKey,+ service_name: "cursor",+ source: "coding",+ cluster_id: "coding",+ environment: "default",+ coding_session_count_24h: 1,+ coding_cost_usd_24h: 0.1,+ coding_active_users_24h: 1,+ }),+ ],+ })+ // recomputeCodingAgentsForWindow → discoverCodingAgents+ .mockResolvedValueOnce({+ data: [+ {+ vendor: "cursor",+ client_version: "1.0.0",+ first_seen: "2026-05-11 20:00:00",+ last_seen: "2026-05-11 22:30:00",+ session_count_24h: 9,+ cost_usd_24h: 1.25,+ active_users_24h: 3,+ lines_added_24h: 10,+ lines_removed_24h: 2,+ lines_accepted_24h: 8,+ lines_rejected_24h: 1,+ edit_accept_24h: 4,+ edit_reject_24h: 1,+ commit_count_24h: 2,+ pr_count_24h: 1,+ },+ ],+ });++ const result = await listAgents({+ filters: { source: ["coding"] },+ timeStart: "2026-05-11 00:00:00",+ timeEnd: "2026-05-11 23:59:59",+ });++ expect(result.data[0].coding_session_count_24h).toBe(9);+ expect(result.data[0].coding_cost_usd_24h).toBe(1.25);+ expect(result.data[0].coding_active_users_24h).toBe(3);+ expect(result.data[0].last_seen).toBe("2026-05-11 22:30:00");+ expect(result.data[0].coding_lines_added_24h).toBe(10);+ expect(result.data[0].coding_commit_count_24h).toBe(2);+ });++ it("swallows coding overlay failures without failing the list", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({+ data: [makeRow({ source: "coding", agent_key: "coding-1" })],+ })+ .mockRejectedValueOnce(new Error("overlay boom"));++ const result = await listAgents({+ filters: { source: ["coding"] },+ timeStart: "2026-05-11 00:00:00",+ });+ expect(result.data).toHaveLength(1);+ expect(result.data[0].agent_key).toBe("coding-1");+ });+});++describe("normalizeDeploymentEnvironment / deploymentEnvironmentSqlPredicate", () => {+ it("collapses local placeholders to default", () => {+ expect(normalizeDeploymentEnvironment(null)).toBe("default");+ expect(normalizeDeploymentEnvironment("")).toBe("default");+ expect(normalizeDeploymentEnvironment("local")).toBe("default");+ expect(normalizeDeploymentEnvironment("default_environment")).toBe("default");+ expect(normalizeDeploymentEnvironment(" staging ")).toBe("staging");+ });++ it("builds equality predicates for non-default environments", () => {+ const escape = (v: string) => v.replace(/\\/g, "\\\\").replace(/'/g, "\\'");+ expect(deploymentEnvironmentSqlPredicate("staging", escape)).toBe(+ "ResourceAttributes['deployment.environment'] = 'staging'"+ );+ expect(deploymentEnvironmentSqlPredicate("local", escape)).toContain(+ "IN ('default', 'local', 'default_environment', '')"+ );+ });});describe("getAgent", () => {@@ -369,6 +516,18 @@ describe("getAgent", () => {expect(await getAgent({ agentKey: "missing" })).toBeNull();});+ it("returns null when ClickHouse errors", async () => {+ mockedDataCollector.mockResolvedValueOnce({ err: "boom", data: [] });+ expect(await getAgent({ agentKey: "key1" })).toBeNull();+ });++ it("invalidates the detail cache key", () => {+ invalidateAgent("key1", "db-1");+ expect(invalidateCache).toHaveBeenCalledWith("agents:detail:db-1:key1");+ invalidateAgent("key2");+ expect(invalidateCache).toHaveBeenCalledWith("agents:detail:default:key2");+ });+it("returns the row when one is found", async () => {mockedDataCollector.mockResolvedValueOnce({data: [diff --git a/src/client/src/__tests__/lib/platform/agents/materialize.test.ts b/src/client/src/__tests__/lib/platform/agents/materialize.test.tsindex 0768144a..391e808d 100644--- a/src/client/src/__tests__/lib/platform/agents/materialize.test.ts+++ b/src/client/src/__tests__/lib/platform/agents/materialize.test.ts@@ -37,10 +37,23 @@ jest.mock("@/lib/platform/agents/snapshot", () => ({}));import { dataCollector } from "@/lib/platform/common";-import { materializeAgents } from "@/lib/platform/agents/materialize";+import {+ materializeAgents,+ recomputeCodingAgentsForWindow,+} from "@/lib/platform/agents/materialize";import { computeAgentKey } from "@/lib/platform/agents";+import {+ deriveSnapshot,+ getLatestVersionsBatch,+ upsertVersion,+} from "@/lib/platform/agents/snapshot";const mockedDC = dataCollector as jest.MockedFunction<typeof dataCollector>;+const mockedDerive = deriveSnapshot as jest.MockedFunction<typeof deriveSnapshot>;+const mockedUpsert = upsertVersion as jest.MockedFunction<typeof upsertVersion>;+const mockedGetLatest = getLatestVersionsBatch as jest.MockedFunction<+ typeof getLatestVersionsBatch+>;interface RecordedInsert {table: string;@@ -87,6 +100,11 @@ function queueDiscovery(beforeEach(() => {mockedDC.mockReset();+ mockedDerive.mockReset();+ mockedUpsert.mockReset();+ mockedDerive.mockResolvedValue(null);+ mockedUpsert.mockResolvedValue({ versionNumber: 1, isNewVersion: false });+ mockedGetLatest.mockResolvedValue(new Map());});describe("materializeAgents — workload_key dedup", () => {@@ -402,3 +420,229 @@ describe("materializeAgents — workload_key dedup", () => {expect(ctrlQuery!).toMatch(/latest\.llm_providers\s+AS\s+llm_providers/);});});++describe("recomputeCodingAgentsForWindow", () => {+ it("builds a 24h default window and maps vendor rows", async () => {+ mockedDC.mockResolvedValueOnce({+ data: [+ {+ vendor: "cursor",+ client_version: "1.2.3",+ first_seen: "2026-05-11 20:00:00",+ last_seen: "2026-05-11 22:00:00",+ session_count_24h: 5,+ cost_usd_24h: 1.5,+ active_users_24h: 2,+ lines_added_24h: 10,+ lines_removed_24h: 1,+ lines_accepted_24h: 8,+ lines_rejected_24h: 0,+ edit_accept_24h: 3,+ edit_reject_24h: 0,+ commit_count_24h: 1,+ pr_count_24h: 0,+ },+ { vendor: "" },+ ],+ } as any);++ const rows = await recomputeCodingAgentsForWindow({});+ expect(rows).toHaveLength(1);+ expect(rows[0]).toMatchObject({+ service_name: "cursor",+ source: "coding",+ cluster_id: "coding",+ coding_session_count_24h: 5,+ coding_cost_usd_24h: 1.5,+ sdk_version: "1.2.3",+ });+ const query = String((mockedDC.mock.calls[0][0] as any).query);+ expect(query).toContain("Timestamp >= now() - INTERVAL 24 HOUR");+ });++ it("escapes custom window bounds and returns [] on error", async () => {+ mockedDC.mockResolvedValueOnce({ data: [] } as any);+ await recomputeCodingAgentsForWindow({+ timeStart: "2026-05-11T00:00:00Z",+ timeEnd: "2026-05-11 23:59:59",+ });+ const query = String((mockedDC.mock.calls[0][0] as any).query);+ expect(query).toContain(+ "parseDateTimeBestEffort('2026-05-11T00:00:00Z')"+ );+ expect(query).toContain("parseDateTimeBestEffort('2026-05-11 23:59:59')");++ mockedDC.mockResolvedValueOnce({+ err: new Error("coding discovery failed"),+ data: [],+ } as any);+ await expect(+ recomputeCodingAgentsForWindow({+ timeStart: "2026-05-11 00:00:00",+ })+ ).resolves.toEqual([]);+ });++ it("escapes quotes in window timestamps", async () => {+ mockedDC.mockResolvedValueOnce({ data: [] } as any);+ await recomputeCodingAgentsForWindow({+ timeStart: "2026-05-11 00:00:00'; DROP TABLE x; --",+ });+ const query = String((mockedDC.mock.calls[0][0] as any).query);+ expect(query).toContain("\\'");+ });+});++describe("materializeAgents — scope / filter / coding", () => {+ it("returns processed:0 when scoped refresh finds no matching agent", async () => {+ mockedDC+ .mockResolvedValueOnce({ data: [] } as any) // SDK+ .mockResolvedValueOnce({ data: [] } as any) // coding+ .mockResolvedValueOnce({ data: [] } as any); // controller++ await expect(+ materializeAgents({+ scope: {+ serviceName: "cursor",+ clusterId: "coding",+ environment: "default",+ },+ })+ ).resolves.toEqual({ processed: 0, newVersions: 0, errors: 0 });+ });++ it("materializes a coding agent matched by agentKeyFilter without snapshot upsert", async () => {+ const codingKey = computeAgentKey("coding", "default", "cursor");+ mockedDC+ // discoverAgents: SDK then controller (sequential)+ .mockResolvedValueOnce({ data: [] } as any)+ .mockResolvedValueOnce({ data: [] } as any)+ // discoverCodingAgents+ .mockResolvedValueOnce({+ data: [+ {+ vendor: "cursor",+ client_version: "9.0.0",+ first_seen: "2026-05-11 20:00:00",+ last_seen: "2026-05-11 22:00:00",+ session_count_24h: 2,+ cost_usd_24h: 0.4,+ active_users_24h: 1,+ lines_added_24h: 0,+ lines_removed_24h: 0,+ lines_accepted_24h: 0,+ lines_rejected_24h: 0,+ edit_accept_24h: 0,+ edit_reject_24h: 0,+ commit_count_24h: 0,+ pr_count_24h: 0,+ },+ ],+ } as any);++ const inserts: RecordedInsert[] = [];+ mockedDC.mockImplementation(async (c: any, op?: string) => {+ if (op === "insert") {+ inserts.push({+ table: String(c.table),+ values: (c.values || []) as any,+ });+ return { data: [] } as any;+ }+ return { data: [] } as any;+ });++ const result = await materializeAgents({ agentKeyFilter: codingKey });+ expect(result.processed).toBe(1);+ expect(mockedDerive).not.toHaveBeenCalled();+ expect(mockedUpsert).not.toHaveBeenCalled();+ expect(inserts[0].values[0]).toMatchObject({+ agent_key: codingKey,+ source: "coding",+ service_name: "cursor",+ coding_agent_vendor: "cursor",+ });+ });++ it("returns empty when agentKeyFilter does not match", async () => {+ mockedDC+ .mockResolvedValueOnce({ data: [] } as any)+ .mockResolvedValueOnce({ data: [] } as any)+ .mockResolvedValueOnce({ data: [] } as any);++ await expect(+ materializeAgents({ agentKeyFilter: "nope" })+ ).resolves.toEqual({ processed: 0, newVersions: 0, errors: 0 });+ });++ it("tolerates SDK/controller discovery errors", async () => {+ mockedDC+ .mockResolvedValueOnce({ err: new Error("sdk boom"), data: [] } as any)+ .mockResolvedValueOnce({ data: [] } as any) // coding+ .mockResolvedValueOnce({ err: new Error("ctrl boom"), data: [] } as any);++ await expect(materializeAgents()).resolves.toEqual({+ processed: 0,+ newVersions: 0,+ errors: 0,+ });+ });++ it("materializes scoped SDK matches and upserts versions when snapshot exists", async () => {+ mockedDerive.mockResolvedValueOnce({+ agent_key: computeAgentKey("default", "default", "scoped-app"),+ service_name: "scoped-app",+ environment: "default",+ cluster_id: "default",+ system_prompt: "hi",+ tools: [],+ primary_model: "gpt-4o",+ models: ["gpt-4o"],+ providers: ["openai"],+ runtime_config: {},+ request_count: 3,+ first_seen: "2026-05-11 22:00:00",+ last_seen: "2026-05-11 22:10:00",+ version_hash: "snap-hash",+ } as any);+ mockedUpsert.mockResolvedValueOnce({ versionNumber: 2, isNewVersion: true });++ mockedDC+ .mockResolvedValueOnce({+ data: [+ {+ service_name: "scoped-app",+ environment: "default",+ cluster_id: "default",+ workload_key: "",+ sdk_version: "1.0.0",+ sdk_language: "python",+ first_seen: "2026-05-11 22:00:00",+ last_seen: "2026-05-11 22:10:00",+ },+ ],+ } as any)+ .mockResolvedValueOnce({ data: [] } as any) // coding+ .mockResolvedValueOnce({ data: [] } as any); // controller++ const inserts: RecordedInsert[] = [];+ mockedDC.mockImplementation(async (c: any, op?: string) => {+ if (op === "insert") {+ inserts.push({+ table: String(c.table),+ values: (c.values || []) as any,+ });+ return { data: [] } as any;+ }+ return { data: [] } as any;+ });++ const result = await materializeAgents({+ scope: { serviceName: "scoped-app", environment: "default" },+ });+ expect(result.processed).toBe(1);+ expect(result.newVersions).toBe(1);+ expect(mockedUpsert).toHaveBeenCalled();+ expect(inserts[0].values[0].current_version_hash).toBe("snap-hash");+ });+});diff --git a/src/client/src/__tests__/lib/platform/agents/snapshot.test.ts b/src/client/src/__tests__/lib/platform/agents/snapshot.test.tsindex 1a34bb7c..2f8ca320 100644--- a/src/client/src/__tests__/lib/platform/agents/snapshot.test.ts+++ b/src/client/src/__tests__/lib/platform/agents/snapshot.test.ts@@ -16,6 +16,10 @@ import {upsertVersion,deriveSnapshot,getVersionTimeline,+ getLatestVersion,+ getLatestVersionsBatch,+ getVersion,+ getVersions,_internals,} from "@/lib/platform/agents/snapshot";@@ -377,6 +381,72 @@ describe("deriveSnapshot — providers attribute coalesce", () => {});});+describe("getLatestVersion / getLatestVersionsBatch", () => {+ const versionRow = {+ agent_key: "abc123",+ version_hash: "hash-1",+ version_number: 3,+ system_prompt: "You are helpful",+ tools: "[]",+ primary_model: "gpt-4o",+ models: ["gpt-4o"],+ providers: ["openai"],+ runtime_config: "{}",+ first_seen: "2026-05-11 22:00:00",+ last_seen: "2026-05-11 22:10:00",+ request_count: 12,+ updated_at: "2026-05-11 22:10:00",+ };++ it("returns the latest version row for an agent", async () => {+ mockedDataCollector.mockResolvedValueOnce({ data: [versionRow] });++ const version = await getLatestVersion("abc123", "db-1");++ expect(version).toEqual(+ expect.objectContaining({+ agent_key: "abc123",+ version_number: 3,+ primary_model: "gpt-4o",+ })+ );+ expect(mockedDataCollector).toHaveBeenCalledWith(+ expect.objectContaining({+ query: expect.stringContaining("agent_key = 'abc123'"),+ }),+ "query",+ "db-1"+ );+ });++ it("returns null on query error or empty rows", async () => {+ mockedDataCollector.mockResolvedValueOnce({ err: "db failed" });+ await expect(getLatestVersion("abc123")).resolves.toBeNull();++ mockedDataCollector.mockResolvedValueOnce({ data: [] });+ await expect(getLatestVersion("abc123")).resolves.toBeNull();+ });++ it("returns an empty map for empty key lists", async () => {+ await expect(getLatestVersionsBatch([])).resolves.toEqual(new Map());+ expect(mockedDataCollector).not.toHaveBeenCalled();+ });++ it("batches latest versions and returns an empty map on error", async () => {+ mockedDataCollector.mockResolvedValueOnce({+ data: [versionRow, { ...versionRow, agent_key: "def456", version_number: 1 }],+ });++ const map = await getLatestVersionsBatch(["abc123", "def456"], "db-1");+ expect(map.size).toBe(2);+ expect(map.get("abc123")?.version_number).toBe(3);+ expect(map.get("def456")?.version_number).toBe(1);++ mockedDataCollector.mockResolvedValueOnce({ err: "batch failed" });+ await expect(getLatestVersionsBatch(["abc123"])).resolves.toEqual(new Map());+ });+});+describe("getVersionTimeline", () => {it("[REGRESSION] counts all spans per ServiceName instead of filtering to gen_ai.operation.name='chat'", async () => {// Agent-framework workloads (CrewAI, LangGraph, etc.) emit@@ -409,4 +479,196 @@ describe("getVersionTimeline", () => {);expect(timelineQuery).not.toContain("gen_ai.operation.name");});++ it("returns empty buckets when the agent summary row is missing", async () => {+ mockedDataCollector.mockResolvedValueOnce({ data: [] });+ await expect(getVersionTimeline("missing")).resolves.toEqual({+ bucketSeconds: 3600,+ start: "",+ end: "",+ buckets: [],+ });+ });++ it("returns empty buckets when the timeline query errors", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({+ data: [{ service_name: "svc", environment: "prod", cluster_id: "c1" }],+ })+ .mockResolvedValueOnce({ data: [] })+ .mockResolvedValueOnce({ err: "timeline failed" });++ await expect(getVersionTimeline("key")).resolves.toEqual({+ bucketSeconds: 3600,+ start: "",+ end: "",+ buckets: [],+ });+ });++ it("attributes empty direct_hash buckets via version first_seen fallback", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({+ data: [+ {+ service_name: "svc",+ environment: "staging",+ cluster_id: "default",+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ agent_key: "abc123",+ version_hash: "hash-old",+ version_number: 1,+ system_prompt: "",+ tools: "[]",+ primary_model: "",+ models: [],+ providers: [],+ runtime_config: "{}",+ first_seen: "2026-05-11 20:00:00",+ last_seen: "2026-05-11 21:00:00",+ request_count: 1,+ updated_at: "2026-05-11 21:00:00",+ },+ {+ agent_key: "abc123",+ version_hash: "hash-new",+ version_number: 2,+ system_prompt: "",+ tools: "[]",+ primary_model: "",+ models: [],+ providers: [],+ runtime_config: "{}",+ first_seen: "2026-05-11 21:30:00",+ last_seen: "2026-05-11 22:00:00",+ request_count: 2,+ updated_at: "2026-05-11 22:00:00",+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ bucket: "2026-05-11 21:00:00",+ direct_hash: "",+ requests: 3,+ },+ {+ bucket: "2026-05-11 22:00:00",+ direct_hash: "hash-new",+ requests: 4,+ },+ {+ bucket: "not-a-date",+ direct_hash: "",+ requests: 1,+ },+ {+ bucket: "2026-05-11 22:00:00",+ direct_hash: "hash-new",+ requests: 1,+ },+ ],+ });++ const timeline = await getVersionTimeline("abc123", {+ bucketSeconds: 60,+ windowHours: 24,+ });++ expect(timeline.bucketSeconds).toBe(60);+ expect(timeline.buckets).toEqual(+ expect.arrayContaining([+ { ts: "2026-05-11 21:00:00", versionHash: "hash-old", requests: 3 },+ { ts: "2026-05-11 22:00:00", versionHash: "hash-new", requests: 5 },+ { ts: "not-a-date", versionHash: "", requests: 1 },+ ])+ );+ const issued = (mockedDataCollector.mock.calls[2][0] as any).query as string;+ expect(issued).toContain(+ "ResourceAttributes['deployment.environment'] = 'staging'"+ );+ });+});++describe("getVersion / getVersions", () => {+ const versionRow = {+ agent_key: "abc123",+ version_hash: "hash-1",+ version_number: 3,+ system_prompt: "You are helpful",+ tools: "[]",+ primary_model: "gpt-4o",+ models: ["gpt-4o"],+ providers: ["openai"],+ runtime_config: "{}",+ first_seen: "2026-05-11 22:00:00",+ last_seen: "2026-05-11 22:10:00",+ request_count: 12,+ updated_at: "2026-05-11 22:10:00",+ };++ it("returns a version by hash and null on miss/error", async () => {+ mockedDataCollector.mockResolvedValueOnce({ data: [versionRow] });+ await expect(getVersion("abc123", "hash-1", "db-1")).resolves.toEqual(+ expect.objectContaining({ version_hash: "hash-1", version_number: 3 })+ );++ mockedDataCollector.mockResolvedValueOnce({ data: [] });+ await expect(getVersion("abc123", "missing")).resolves.toBeNull();++ mockedDataCollector.mockResolvedValueOnce({ err: "boom" });+ await expect(getVersion("abc123", "hash-1")).resolves.toBeNull();+ });++ it("returns [] when getVersions errors", async () => {+ mockedDataCollector.mockResolvedValueOnce({ err: "boom" });+ await expect(getVersions("abc123")).resolves.toEqual([]);+ });+});++describe("deriveSnapshot / upsertVersion error paths", () => {+ it("returns null when the aggregate query errors", async () => {+ mockedDataCollector.mockResolvedValueOnce({ err: "aggregate failed" });+ await expect(+ deriveSnapshot({ serviceName: "svc", environment: "prod" })+ ).resolves.toBeNull();+ });++ it("falls back when logs tool-definition fetch errors", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({+ data: [+ {+ system_prompt: "You are helpful",+ tool_definitions_json: "",+ tools_fallback: [["search", "Search", "function"]],+ primary_model: "gpt-4o",+ models: ["gpt-4o"],+ providers: ["openai"],+ temperature: 0.7,+ top_p: 1,+ max_tokens: 1024,+ request_count: 2,+ first_seen: "2026-05-11 22:00:00",+ last_seen: "2026-05-11 22:10:00",+ },+ ],+ })+ .mockResolvedValueOnce({ err: "logs failed" });++ const result = await deriveSnapshot({ serviceName: "svc" });+ expect(result?.tools).toHaveLength(1);+ expect(result?.tools[0].name).toBe("search");+ });++ it("throws when upsertVersion insert errors", async () => {+ mockedDataCollector.mockResolvedValueOnce({ err: "insert failed" });+ await expect(upsertVersion(baseSnapshot)).rejects.toEqual("insert failed");+ });});diff --git a/src/client/src/__tests__/lib/platform/agents/version-filter.test.ts b/src/client/src/__tests__/lib/platform/agents/version-filter.test.tsindex 012bf572..547a992d 100644--- a/src/client/src/__tests__/lib/platform/agents/version-filter.test.ts+++ b/src/client/src/__tests__/lib/platform/agents/version-filter.test.ts@@ -17,9 +17,16 @@ jest.mock("@/lib/platform/agents/snapshot", () => ({getVersion: (...args: unknown[]) => mockGetVersion(...args),}));-jest.mock("@/lib/platform/agents/index", () => ({- getAgent: (...args: unknown[]) => mockGetAgent(...args),-}));+jest.mock("@/lib/platform/agents/index", () => {+ const actual = jest.requireActual("@/lib/platform/agents/index") as Record<+ string,+ unknown+ >;+ return {+ ...actual,+ getAgent: (...args: unknown[]) => mockGetAgent(...args),+ };+});jest.mock("@/lib/platform/agents/logger", () => ({agentsLogger: {@@ -116,7 +123,7 @@ describe("getVersionWindow", () => {expect(mockDataCollector).toHaveBeenCalledWith({query: expect.stringContaining(- "ResourceAttributes['deployment.environment'] = 'default'"+ "ResourceAttributes['deployment.environment'] IN ('default', 'local', 'default_environment', '')"),},"query",diff --git a/src/client/src/__tests__/lib/platform/chat/cost.test.ts b/src/client/src/__tests__/lib/platform/chat/cost.test.tsnew file mode 100644index 00000000..cc29f540--- /dev/null+++ b/src/client/src/__tests__/lib/platform/chat/cost.test.ts@@ -0,0 +1,107 @@+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterPreviousParams: jest.fn((params) => ({+ ...params,+ timeLimit: {+ ...params.timeLimit,+ start: new Date("2023-12-31"),+ end: new Date("2024-01-01"),+ },+ })),+}));++import { dataCollector } from "@/lib/platform/common";+import { getFilterPreviousParams } from "@/helpers/server/platform";+import { getOtterTotalCost } from "@/lib/platform/chat/cost";++const mockedDataCollector = dataCollector as jest.MockedFunction<+ typeof dataCollector+>;+const mockedGetFilterPreviousParams =+ getFilterPreviousParams as jest.MockedFunction<+ typeof getFilterPreviousParams+ >;++const params = {+ timeLimit: {+ start: new Date("2024-01-01"),+ end: new Date("2024-01-02"),+ type: "custom",+ },+};++describe("getOtterTotalCost", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ mockedGetFilterPreviousParams.mockClear();+ });++ it("sums current and previous period otter costs", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({ data: [{ total_cost: 1.25 }] })+ .mockResolvedValueOnce({ data: [{ total_cost: 0.5 }] });++ const res = await getOtterTotalCost(params);++ expect(mockedGetFilterPreviousParams).toHaveBeenCalledWith(params);+ expect(mockedDataCollector).toHaveBeenCalledTimes(2);+ expect(mockedDataCollector.mock.calls[0][0].query).toContain(+ "openlit_chat_conversation"+ );+ expect(mockedDataCollector.mock.calls[0][0].query).toContain(+ "openlit_trace_analysis"+ );+ expect(mockedDataCollector.mock.calls[0][0].query).toContain(+ "openlit_otter_runs"+ );+ // Float64 cost columns must not use toFloat64OrZero (ClickHouse rejects it).+ expect(mockedDataCollector.mock.calls[0][0].query).not.toContain(+ "toFloat64OrZero"+ );+ expect(mockedDataCollector.mock.calls[0][0].query).toContain(+ "sum(ifNull(total_cost, 0))"+ );+ expect(res).toEqual({+ data: [{ total_cost: 1.25, previous_total_cost: 0.5 }],+ });+ });++ it("returns 0 when dataCollector errors", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({ data: null, err: new Error("fail") })+ .mockResolvedValueOnce({ data: null, err: new Error("fail") });++ const res = await getOtterTotalCost(params);++ expect(res).toEqual({+ data: [{ total_cost: 0, previous_total_cost: 0 }],+ });+ });++ it("returns 0 when data is empty or non-array", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({ data: [] })+ .mockResolvedValueOnce({ data: { total_cost: 9 } as unknown as never[] });++ const res = await getOtterTotalCost(params);++ expect(res).toEqual({+ data: [{ total_cost: 0, previous_total_cost: 0 }],+ });+ });++ it("returns 0 when total_cost is missing or NaN", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({ data: [{}] })+ .mockResolvedValueOnce({ data: [{ total_cost: "not-a-number" }] });++ const res = await getOtterTotalCost(params);++ expect(res).toEqual({+ data: [{ total_cost: 0, previous_total_cost: 0 }],+ });+ });+});diff --git a/src/client/src/__tests__/lib/platform/chat/improvement.test.ts b/src/client/src/__tests__/lib/platform/chat/improvement.test.tsindex e5d9c8cc..3aece15d 100644--- a/src/client/src/__tests__/lib/platform/chat/improvement.test.ts+++ b/src/client/src/__tests__/lib/platform/chat/improvement.test.ts@@ -237,6 +237,15 @@ describe("getTraceAnalysisRuns", () => {"db-config-1");});++ it("defaults null data to an empty array and filters by analysis type", async () => {+ (dataCollector as jest.Mock).mockResolvedValue({ data: null });+ const { data } = await getTraceAnalysisRuns("span-abc", undefined, "span_analysis");+ expect(data).toEqual([]);+ expect((dataCollector as jest.Mock).mock.calls[0][0].query).toContain(+ "analysis_type = 'span_analysis'"+ );+ });});// ─── getTraceImprovement ─────────────────────────────────────────────────────@@ -295,6 +304,29 @@ describe("getTraceImprovement", () => {"analysis_type = 'span_analysis'");});++ it("returns hierarchy-not-found when the record has no SpanId", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: {},+ err: null,+ });++ await expect(getTraceImprovement("missing")).resolves.toEqual({+ err: "Trace hierarchy not found",+ });+ });++ it("returns empty runs when the runs query errors", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: hierarchy,+ err: null,+ });+ (dataCollector as jest.Mock).mockResolvedValue({ err: "runs failed" });++ const result = await getTraceImprovement("root-span", "db-1");++ expect(result).toEqual({ data: { rootSpanId: "root-span", runs: [] } });+ });});// ─── saveTraceAnalysisRun ────────────────────────────────────────────────────@@ -369,6 +401,45 @@ describe("saveTraceAnalysisRun", () => {expect(insertValues.run_number).toBe(2);expect(data!.runNumber).toBe(2);});++ it("computes worst severity and stores span analysis type", async () => {+ (dataCollector as jest.Mock).mockResolvedValue({ err: null });+ const analysis = {+ ...emptyTraceAnalysis("t1"),+ summary: "",+ improvements: [+ {+ id: "f1",+ severity: "critical" as const,+ summary: "bad",+ detail: "worse",+ span_refs: ["root-1"],+ },+ ],+ cost: [+ {+ id: "f2",+ severity: "minor" as const,+ summary: "pricey",+ detail: "ok",+ span_refs: ["root-1"],+ },+ ],+ };++ const { data } = await saveTraceAnalysisRun({+ ...baseArgs,+ analysis,+ analysisType: "span_analysis",+ });+ const insertValues = (dataCollector as jest.Mock).mock.calls[0][0].values[0];++ expect(insertValues.worst_severity).toBe("critical");+ expect(insertValues.analysis_type).toBe("span_analysis");+ expect(insertValues.summary).toBe("");+ expect(data!.worstSeverity).toBe("critical");+ expect(data!.summary).toBe("");+ });});// ─── streamTraceImprovementAnalysis ──────────────────────────────────────────@@ -476,4 +547,499 @@ describe("streamTraceImprovementAnalysis", () => {);expect(events.find((event) => event.type === "done")).toBeDefined();});++ it("returns the default chat-not-configured message when config is empty", async () => {+ (getChatConfigWithApiKey as jest.Mock).mockResolvedValue({ data: null });++ const result = await streamTraceImprovementAnalysis("root-span");++ expect(result.err).toMatch(/Chat not configured/);+ });++ it("returns hierarchy-not-found when the record has no SpanId", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: { SpanName: "orphan" },+ err: null,+ });++ const result = await streamTraceImprovementAnalysis("root-span", "db-1");++ expect(result).toEqual({ err: "Trace hierarchy not found" });+ });++ it("streams span-scope analysis and falls back when the grader returns unparseable JSON", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: hierarchy,+ err: null,+ });+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [], err: null })+ .mockResolvedValueOnce({ err: null });++ let call = 0;+ (streamText as jest.Mock).mockImplementation(({ system, onFinish }) => {+ call += 1;+ onFinish({ usage: { inputTokens: 3, outputTokens: 1 } });+ const isGrader = String(system).includes("quality grader");+ return {+ fullStream: (async function* () {+ if (isGrader) {+ yield { type: "text-delta", text: "not-json-grader" };+ return;+ }+ yield {+ type: "text-delta",+ text: JSON.stringify({+ summary: "first pass ok",+ findings: [+ {+ severity: "bogus",+ summary: "Needs a fix",+ detail: "detail",+ span_refs: ["llm-span"],+ suggested_fix: "tighten prompt",+ suggested_fix_patches: [+ {+ field: "prompt",+ span_ref: "llm-span",+ original: "old",+ replacement: "new",+ },+ { field: "bad", span_ref: "x" },+ null,+ ],+ estimated_savings: { tokens: 5 },+ },+ ],+ }),+ };+ })(),+ };+ });++ const result = await streamTraceImprovementAnalysis("llm-span", "db-1", "span");+ const events = await readNdjson(result.response!);+ const dimensionEvent = events.find((event) => event.type === "dimension");+ const doneEvent = events.find((event) => event.type === "done");+ const insertValues = (dataCollector as jest.Mock).mock.calls[1][0].values[0];++ expect(call).toBeGreaterThanOrEqual(12);+ expect(dimensionEvent.findings[0]).toEqual(+ expect.objectContaining({+ severity: "info",+ suggested_fix: "tighten prompt",+ suggested_fix_patches: [+ expect.objectContaining({+ field: "prompt",+ original: "old",+ replacement: "new",+ }),+ ],+ })+ );+ expect(insertValues.analysis_type).toBe("span_analysis");+ expect(insertValues.root_span_id).toBe("llm-span");+ expect(doneEvent).toBeDefined();+ });++ it("emits a stream error when saving the analysis run fails", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: hierarchy,+ err: null,+ });+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [], err: null })+ .mockResolvedValueOnce({ err: "insert failed" });+ mockDimensionStreams();++ const result = await streamTraceImprovementAnalysis("root-span", "db-1");+ const events = await readNdjson(result.response!);++ expect(events.find((event) => event.type === "error")).toEqual({+ type: "error",+ error: "Failed to save trace analysis run",+ });+ expect(events.find((event) => event.type === "done")).toBeUndefined();+ });++ it("keeps first-pass findings when the grader throws", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: hierarchy,+ err: null,+ });+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [], err: null })+ .mockResolvedValueOnce({ err: null });++ (streamText as jest.Mock).mockImplementation(({ system, onFinish }) => {+ onFinish({ usage: {} });+ if (String(system).includes("quality grader")) {+ return {+ fullStream: (async function* () {+ throw new Error("grader boom");+ })(),+ };+ }+ return {+ fullStream: (async function* () {+ yield {+ type: "text-delta",+ delta: JSON.stringify({+ summary: "kept",+ findings: [+ {+ severity: "major",+ summary: "First pass finding",+ detail: "kept after grader failure",+ span_refs: ["root-span"],+ },+ ],+ }),+ };+ })(),+ };+ });++ const result = await streamTraceImprovementAnalysis("root-span", "db-1");+ const events = await readNdjson(result.response!);+ const strengths = events.find(+ (event) => event.type === "dimension" && event.dimension === "strengths"+ );++ expect(strengths.findings[0].summary).toBe("First pass finding");+ expect(events.find((event) => event.type === "done")).toBeDefined();+ });++ it("covers alternate span roles, duration scales, and rule-context failures", async () => {+ const longPrompt = `sk-abcdefghijklmnopqrstuvwxyz ${"x".repeat(900)} Bearer abcdefghijklmnop ${"y".repeat(900)}`;+ const richHierarchy = {+ TraceId: "trace-rich",+ SpanId: "orch-root",+ ParentSpanId: "",+ SpanName: "workflow.orchestrator",+ ServiceName: "rich-service",+ Duration: 25_000,+ StatusCode: "STATUS_CODE_OK",+ ResourceAttributes: {+ "service.name": "rich-service",+ "deployment.environment": "prod",+ "gen_ai.application_name": "demo",+ },+ SpanAttributes: {},+ Events: [],+ children: [+ {+ TraceId: "trace-rich",+ SpanId: "embed-1",+ ParentSpanId: "orch-root",+ SpanName: "embeddings.create",+ Duration: 5,+ StatusCode: "STATUS_CODE_OK",+ SpanAttributes: {+ "gen_ai.content.prompt": JSON.stringify({ text: "embed me" }),+ "gen_ai.usage.input_tokens": 11,+ },+ Events: [],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "retriev-1",+ ParentSpanId: "orch-root",+ SpanName: "vector.search",+ Duration: 12,+ StatusCode: "STATUS_CODE_OK",+ SpanAttributes: {+ prompt: "find docs",+ "gen_ai.usage.input_tokens": 4,+ },+ Events: [],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "db-1",+ ParentSpanId: "orch-root",+ SpanName: "db.query",+ Duration: 8,+ StatusCode: "STATUS_CODE_OK",+ SpanAttributes: {+ "db.query.text": "SELECT 1",+ "db.system.name": "postgres",+ },+ Events: [],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "http-1",+ ParentSpanId: "orch-root",+ SpanName: "http.request",+ Duration: 9,+ StatusCode: "STATUS_CODE_OK",+ SpanAttributes: {+ "http.method": "GET",+ "http.url": "https://example.com/api",+ },+ Events: [],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "tool-a",+ ParentSpanId: "orch-root",+ SpanName: "tool.search",+ Duration: 15,+ StatusCode: "STATUS_CODE_ERROR",+ StatusMessage: "boom",+ SpanAttributes: {+ "gen_ai.tool.name": "search",+ "gen_ai.tool.call.arguments": JSON.stringify({ q: "same" }),+ "error.message": "temporary",+ },+ Events: [],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "tool-b",+ ParentSpanId: "orch-root",+ SpanName: "tool.search",+ Duration: 16,+ StatusCode: "STATUS_CODE_OK",+ SpanAttributes: {+ "gen_ai.tool.name": "search",+ "gen_ai.tool.call.arguments": JSON.stringify({ q: "same" }),+ "gen_ai.tool.result": JSON.stringify({ result: "ok" }),+ },+ Events: [],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "llm-secret",+ ParentSpanId: "orch-root",+ SpanName: "llm.completion",+ Duration: 2_500_000_000,+ StatusCode: "STATUS_CODE_OK",+ Cost: 0.02,+ SpanAttributes: {+ "gen_ai.request.model": "gpt-4o",+ "gen_ai.system": "openai",+ "gen_ai.usage.input_tokens": 200,+ "gen_ai.usage.output_tokens": 50,+ "gen_ai.usage.cache_read_input_tokens": 40,+ "gen_ai.usage.cache_creation_input_tokens": 10,+ "gen_ai.usage.reasoning_tokens": 5,+ "gen_ai.content.prompt": longPrompt,+ "gen_ai.content.completion": JSON.stringify({+ content: [{ text: "answer" }],+ }),+ "gen_ai.system_instructions": "be careful",+ "gen_ai.content.reasoning": "thinking",+ },+ Events: [+ {+ Name: "gen_ai.completion",+ Attributes: { completion: "event completion" },+ },+ ],+ children: [],+ },+ {+ TraceId: "trace-rich",+ SpanId: "unknown-1",+ ParentSpanId: "orch-root",+ SpanName: "misc.work",+ Duration: Number.NaN,+ StatusCode: "STATUS_CODE_OK",+ SpanAttributes: { "not.a.role": true },+ Events: [],+ children: [],+ },+ ],+ };++ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: richHierarchy,+ err: null,+ });+ (getContextFromRuleEngineForTrace as jest.Mock).mockRejectedValue(+ new Error("rule engine down")+ );+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{ id: "prior" }], err: null })+ .mockResolvedValueOnce({ err: null });+ (streamText as jest.Mock).mockImplementation(({ onFinish }) => {+ onFinish({ usage: { inputTokens: 2, outputTokens: 1 } });+ return {+ fullStream: (async function* () {+ yield {+ type: "text-delta",+ delta: "```json\n" + JSON.stringify([+ {+ severity: "critical",+ summary: "Array finding",+ detail: "parsed from array shape",+ span_refs: ["orch-root"],+ },+ ]) + "\n```",+ };+ })(),+ };+ });++ const result = await streamTraceImprovementAnalysis("orch-root", "db-1");+ const events = await readNdjson(result.response!);+ const doneEvent = events.find((event) => event.type === "done");+ const insertValues = (dataCollector as jest.Mock).mock.calls[1][0].values[0];++ expect(doneEvent.data.runs).toHaveLength(2);+ expect(insertValues.worst_severity).toBe("critical");+ expect(insertValues.run_number).toBe(2);+ expect(JSON.parse(insertValues.analysis_json).totals.span_count).toBe(9);+ });++ it("parses findings nested under a dimension key and ignores non-text stream parts", async () => {+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: hierarchy,+ err: null,+ });+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [], err: null })+ .mockResolvedValueOnce({ err: null });+ (streamText as jest.Mock).mockImplementation(({ system, onFinish }) => {+ onFinish({ usage: { inputTokens: 1, outputTokens: 1 } });+ const dimension = String(system).match(/Dimension: ([^\n]+)/)?.[1] || "improvements";+ return {+ fullStream: (async function* () {+ yield { type: "reasoning-delta", delta: "ignore me" };+ yield {+ type: "text-delta",+ delta: JSON.stringify({+ summary: "dimension key shape",+ [dimension]: [+ {+ id: "fixed-id",+ severity: "minor",+ summary: "From dimension key",+ detail: "ok",+ span_refs: ["root-span"],+ suggested_fix_patches: [+ {+ field: "response",+ span_ref: "root-span",+ original: "a",+ replacement: "b",+ },+ ],+ },+ ],+ }),+ };+ })(),+ };+ });++ const result = await streamTraceImprovementAnalysis("root-span", "db-1");+ const events = await readNdjson(result.response!);+ const first = events.find((event) => event.type === "dimension");++ expect(first.findings[0]).toEqual(+ expect.objectContaining({+ id: "fixed-id",+ summary: "From dimension key",+ suggested_fix_patches: [+ expect.objectContaining({ field: "response" }),+ ],+ })+ );+ });++ it("covers stream error fallback and role classification via hierarchy metrics", async () => {+ const toolHeavy = {+ ...hierarchy,+ children: [+ {+ ...hierarchy.children[0],+ SpanName: "tool.execute",+ SpanAttributes: {+ ...hierarchy.children[0].SpanAttributes,+ "gen_ai.tool.call.arguments": JSON.stringify({ q: "same" }),+ },+ },+ {+ ...hierarchy.children[0],+ SpanId: "tool-span-2",+ SpanName: "tool.execute",+ SpanAttributes: {+ ...hierarchy.children[0].SpanAttributes,+ "gen_ai.tool.call.arguments": JSON.stringify({ q: "same" }),+ },+ },+ hierarchy.children[1],+ ],+ };++ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: toolHeavy,+ err: null,+ });+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [], err: null })+ .mockResolvedValueOnce({ err: null });+ (streamText as jest.Mock).mockImplementation(({ onFinish }) => {+ onFinish({ usage: { inputTokens: 1, outputTokens: 1 } });+ return {+ fullStream: (async function* () {+ yield {+ type: "text-delta",+ delta: JSON.stringify({+ summary: "ok",+ improvements: [+ {+ severity: "info",+ summary: "tool reuse",+ detail: "same tool",+ span_refs: ["tool-span", "tool-span-2"],+ },+ ],+ }),+ };+ })(),+ };+ });++ const ok = await streamTraceImprovementAnalysis("root-span", "db-1");+ const okEvents = await readNdjson(ok.response!);+ const detailEvent = okEvents.find(+ (e) =>+ e.type === "step" &&+ typeof e.detail === "string" &&+ /LLM calls/.test(e.detail)+ );+ expect(detailEvent?.detail).toMatch(/2 tools/);+ const debugContext = okEvents.find(+ (e) => e.type === "debug" && e.stage === "context_extracted"+ );+ expect((debugContext?.payload as any).toolCallCount).toBe(2);+ expect((debugContext?.payload as any).duplicateToolInputs).toEqual(+ expect.arrayContaining([expect.objectContaining({ count: 2 })])+ );++ (streamText as jest.Mock).mockImplementation(() => {+ throw { notMessage: true };+ });+ (getHeirarchyViaSpanId as jest.Mock).mockResolvedValue({+ record: hierarchy,+ err: null,+ });+ (dataCollector as jest.Mock).mockResolvedValueOnce({ data: [], err: null });++ const result = await streamTraceImprovementAnalysis("root-span", "db-1");+ const events = await readNdjson(result.response!);+ expect(events.some((e) => e.type === "error")).toBe(true);+ const errEvent = events.find((e) => e.type === "error");+ expect(errEvent?.error).toBe("Failed to run AI improvement analysis");+ });});diff --git a/src/client/src/__tests__/lib/platform/chat/tools.test.ts b/src/client/src/__tests__/lib/platform/chat/tools.test.tsindex aeeb5836..bb58c12d 100644--- a/src/client/src/__tests__/lib/platform/chat/tools.test.ts+++ b/src/client/src/__tests__/lib/platform/chat/tools.test.ts@@ -69,6 +69,7 @@ jest.mock("@/utils/sanitizer", () => ({import { getChatTools } from "@/lib/platform/chat/tools";import {+ addConditionGroupsToRule,addRuleEntity,createRule,deleteRule,@@ -689,4 +690,632 @@ describe("getChatTools", () => {expect(result.details).toContain("```trace-refs");expect(result.matchedTraceCount).toBe(1);});++ it("returns create_rule errors and skips condition groups when none are provided", async () => {+ (createRule as jest.Mock)+ .mockResolvedValueOnce({ id: "rule-2" })+ .mockRejectedValueOnce(new Error("create failed"));+ const tools = getChatTools("user-1", "db-1") as any;++ await expect(+ tools.create_rule.execute({ name: "Simple", status: "INACTIVE" })+ ).resolves.toEqual(+ expect.objectContaining({+ success: true,+ details: expect.stringContaining("Conditions: 0 groups"),+ })+ );+ expect(addConditionGroupsToRule).not.toHaveBeenCalled();++ await expect(tools.create_rule.execute({ name: "Bad" })).resolves.toEqual({+ success: false,+ error: "create failed",+ });+ });++ it("covers remaining rule/context/prompt/vault error and success branches", async () => {+ (getRules as jest.Mock).mockResolvedValue({+ data: [{ id: "r1", name: "Rule", status: "ACTIVE", description: "d" }],+ err: null,+ });+ (getRuleById as jest.Mock).mockResolvedValue({ err: "missing rule" });+ (deleteRuleEntity as jest.Mock).mockResolvedValue(["unlink failed"]);+ (getRuleEntities as jest.Mock).mockResolvedValue({ err: "entities failed" });+ (deleteContext as jest.Mock).mockResolvedValue(["ctx delete failed"]);+ (getContexts as jest.Mock).mockRejectedValue(new Error("contexts boom"));+ (upsertPromptVersion as jest.Mock).mockResolvedValue(undefined);+ (getPromptDetails as jest.Mock).mockResolvedValue({+ data: [+ {+ promptId: "prompt-1",+ versionId: "v1",+ name: "p",+ version: "1.0.0",+ status: "DRAFT",+ tags: [],+ prompt: "hi",+ },+ ],+ err: null,+ });+ (deletePrompt as jest.Mock).mockResolvedValue(["prompt delete failed"]);+ (getPrompts as jest.Mock).mockResolvedValue({ err: "prompts failed" });+ (deleteSecret as jest.Mock).mockResolvedValue(["secret delete failed"]);+ (getSecrets as jest.Mock).mockResolvedValue({ err: "secrets failed" });+ const tools = getChatTools("user-1", "db-1") as any;++ await expect(tools.list_rules.execute({})).resolves.toEqual({+ success: true,+ count: 1,+ rules: [{ id: "r1", name: "Rule", status: "ACTIVE", description: "d" }],+ });+ await expect(tools.get_rule.execute({ id: "missing" })).resolves.toEqual({+ success: false,+ error: "missing rule",+ });+ await expect(+ tools.unlink_entity_from_rule.execute({ id: "link-1" })+ ).resolves.toEqual({ success: false, error: "unlink failed" });+ await expect(+ tools.list_rule_entities.execute({ rule_id: "r1" })+ ).resolves.toEqual({ success: false, error: "entities failed" });+ await expect(tools.delete_context.execute({ id: "ctx" })).resolves.toEqual({+ success: false,+ error: "ctx delete failed",+ });+ await expect(tools.list_contexts.execute({})).resolves.toEqual({+ success: false,+ error: "contexts boom",+ });+ await expect(+ tools.update_prompt_version.execute({+ prompt_id: "prompt-1",+ version: "2.0.0",+ })+ ).resolves.toEqual(+ expect.objectContaining({+ success: true,+ message: "New prompt version created",+ })+ );+ await expect(+ tools.get_prompt.execute({ prompt_id: "prompt-1", version: "1.0.0" })+ ).resolves.toEqual(expect.objectContaining({ success: true }));+ expect(getPromptDetails).toHaveBeenCalledWith("prompt-1", { version: "1.0.0" });+ await expect(tools.delete_prompt.execute({ id: "prompt-1" })).resolves.toEqual({+ success: false,+ error: "prompt delete failed",+ });+ await expect(tools.list_prompts.execute({})).resolves.toEqual({+ success: false,+ error: "prompts failed",+ });+ await expect(tools.delete_vault_secret.execute({ id: "s1" })).resolves.toEqual({+ success: false,+ error: "secret delete failed",+ });+ await expect(tools.list_vault_secrets.execute({})).resolves.toEqual({+ success: false,+ error: "secrets failed",+ });+ });++ it("covers custom model error returns and alert tool passthrough", async () => {+ (createCustomModel as jest.Mock).mockResolvedValue({ err: "create model failed" });+ (updateCustomModel as jest.Mock).mockResolvedValue({ err: "update model failed" });+ (deleteCustomModel as jest.Mock).mockResolvedValue({ err: "delete model failed" });+ (getCustomModels as jest.Mock).mockResolvedValue({ err: "list models failed" });+ const tools = getChatTools("user-1", "db-1") as any;++ await expect(+ tools.create_custom_model.execute({+ provider: "openai",+ model_id: "m",+ display_name: "M",+ })+ ).resolves.toEqual({ success: false, error: "create model failed" });+ await expect(+ tools.update_custom_model.execute({ id: "model-1" })+ ).resolves.toEqual({ success: false, error: "update model failed" });+ await expect(+ tools.delete_custom_model.execute({ id: "model-1" })+ ).resolves.toEqual({ success: false, error: "delete model failed" });+ await expect(tools.list_custom_models.execute({})).resolves.toEqual({+ success: false,+ error: "list models failed",+ });++ await expect(tools.create_alert.execute({ name: "A" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.list_alerts.execute({})).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.get_alert.execute({ id: "a1" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(+ tools.update_alert.execute({ id: "a1", name: "B" })+ ).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.delete_alert.execute({ id: "a1" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(+ tools.create_alert_destination.execute({+ name: "Slack",+ providerType: "slack",+ config: {},+ })+ ).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.list_alert_destinations.execute({})).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.get_alert_destination.execute({ id: "d1" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(+ tools.update_alert_destination.execute({ id: "d1", name: "X" })+ ).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.delete_alert_destination.execute({ id: "d1" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.test_alert.execute({ id: "a1" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ await expect(tools.test_alert_destination.execute({ id: "d1" })).resolves.toEqual({+ success: false,+ error: "Alerting is not available in this edition.",+ });+ });++ it("reruns analyze_trace and handles stream failure paths", async () => {+ const tools = getChatTools("user-1", "db-1") as any;+ const run = {+ id: "run-2",+ rootSpanId: "span-1",+ selectedSpanId: "span-1",+ runNumber: 2,+ analysisJson: "{not-json",+ summary: "",+ modelProvider: "openai",+ modelName: "gpt-4o",+ promptTokens: 1,+ completionTokens: 1,+ cost: 0,+ worstSeverity: "",+ createdAt: "2026-01-01T00:00:00.000Z",+ };++ (streamTraceImprovementAnalysis as jest.Mock).mockResolvedValueOnce({+ response: ndjsonResponse(+ `${JSON.stringify({+ type: "done",+ data: { rootSpanId: "span-1", runs: [run] },+ })}\n`+ ),+ });+ await expect(+ tools.analyze_trace.execute({+ span_id: "span-1",+ scope: "span",+ rerun: true,+ })+ ).resolves.toEqual(+ expect.objectContaining({+ success: true,+ existing: false,+ message: "Span analysis completed",+ })+ );+ expect(getTraceImprovement).not.toHaveBeenCalled();++ (streamTraceImprovementAnalysis as jest.Mock).mockResolvedValueOnce({+ err: "stream start failed",+ });+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1", rerun: true })+ ).resolves.toEqual({+ success: false,+ error: "stream start failed",+ });++ (streamTraceImprovementAnalysis as jest.Mock).mockResolvedValueOnce({+ response: ndjsonResponse(+ `${JSON.stringify({ type: "error", error: "model failed" })}\n`+ ),+ });+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1", rerun: true })+ ).resolves.toEqual({+ success: false,+ error: "model failed",+ });++ (streamTraceImprovementAnalysis as jest.Mock).mockResolvedValueOnce({+ response: { body: null },+ });+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1", rerun: true })+ ).resolves.toEqual({+ success: false,+ error: "Trace analysis stream did not return a body",+ });++ (streamTraceImprovementAnalysis as jest.Mock).mockResolvedValueOnce({+ response: ndjsonResponse(`${JSON.stringify({ type: "step", label: "x" })}\n`),+ });+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1", rerun: true })+ ).resolves.toEqual({+ success: false,+ error: "Trace analysis finished without a result",+ });++ (streamTraceImprovementAnalysis as jest.Mock).mockResolvedValueOnce({+ response: ndjsonResponse(+ `${JSON.stringify({+ type: "done",+ data: { rootSpanId: "span-1", runs: [] },+ })}\n`+ ),+ });+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1", rerun: true })+ ).resolves.toEqual({+ success: false,+ error: "Trace analysis completed without a saved run",+ });+ });++ it("returns existing span analysis and get_trace_analysis branches", async () => {+ const tools = getChatTools("user-1", "db-1") as any;+ (getTraceImprovement as jest.Mock).mockResolvedValueOnce({+ err: "lookup failed",+ });+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1" })+ ).resolves.toEqual({ success: false, error: "lookup failed" });++ (getTraceImprovement as jest.Mock).mockResolvedValueOnce({+ data: {+ rootSpanId: "span-child",+ runs: [+ {+ id: "run-1",+ rootSpanId: "span-child",+ selectedSpanId: "span-child",+ analysisType: "span_analysis",+ runNumber: 1,+ analysisJson: JSON.stringify({+ traceId: "t-1",+ summary: "Span ok",+ strengths: [{}],+ improvements: [],+ wrong_turns: [],+ cost: [],+ token_efficiency: [],+ path_analysis: [],+ totals: { span_count: 1 },+ }),+ summary: "",+ modelProvider: "openai",+ modelName: "gpt-4o",+ promptTokens: 1,+ completionTokens: 1,+ cost: 0,+ worstSeverity: "info",+ createdAt: "2026-01-01T00:00:00.000Z",+ },+ ],+ },+ });+ const existing = await tools.analyze_trace.execute({+ span_id: "span-child",+ scope: "span",+ });+ expect(existing).toEqual(+ expect.objectContaining({+ success: true,+ existing: true,+ message: "Existing span analysis found",+ })+ );+ expect(existing.analysis.dimensionCounts.strengths).toBe(1);+ expect(existing.details).toContain("```trace-refs");++ (getTraceImprovement as jest.Mock).mockResolvedValueOnce({+ data: { rootSpanId: "span-1", runs: [] },+ });+ await expect(+ tools.get_trace_analysis.execute({ span_id: "span-1" })+ ).resolves.toEqual({+ success: true,+ rootSpanId: "span-1",+ count: 0,+ latest: null,+ runs: [],+ });++ (getTraceImprovement as jest.Mock).mockResolvedValueOnce({ err: "get failed" });+ await expect(+ tools.get_trace_analysis.execute({ span_id: "span-1", scope: "span" })+ ).resolves.toEqual({ success: false, error: "get failed" });++ (getTraceImprovement as jest.Mock).mockRejectedValueOnce(new Error("boom"));+ await expect(+ tools.get_trace_analysis.execute({ span_id: "span-1" })+ ).resolves.toEqual({ success: false, error: "boom" });+ });++ it("covers analyze_trace_batch empty, capped, and mixed results", async () => {+ const tools = getChatTools("user-1", "db-1") as any;++ await expect(+ tools.analyze_trace_batch.execute({ span_ids: [] })+ ).resolves.toEqual({+ success: false,+ error: "No span IDs were provided for batch analysis",+ });++ (getTraceImprovement as jest.Mock)+ .mockResolvedValueOnce({+ data: {+ rootSpanId: "s1",+ runs: [+ {+ id: "run-1",+ rootSpanId: "s1",+ selectedSpanId: "s1",+ runNumber: 1,+ analysisJson: "{}",+ summary: "ok",+ modelProvider: "openai",+ modelName: "gpt-4o",+ promptTokens: 1,+ completionTokens: 1,+ cost: 0,+ worstSeverity: "info",+ createdAt: "2026-01-01T00:00:00.000Z",+ },+ ],+ },+ })+ .mockResolvedValueOnce({ err: "missing" });++ const batch = await tools.analyze_trace_batch.execute({+ span_ids: ["s1", "s2", "s3", "s4", "s5", "s6"],+ group_label: "latency",+ scope: "span",+ });++ expect(batch.processed).toBe(5);+ expect(batch.limitApplied).toBe(true);+ expect(batch.groupLabel).toBe("latency");+ expect(batch.success).toBe(false);+ expect(batch.results[0].success).toBe(true);+ expect(batch.results[1].success).toBe(false);+ });++ it("covers analyze_traces_by_attribute validation and empty/error paths", async () => {+ const tools = getChatTools("user-1", "db-1") as any;++ await expect(+ tools.analyze_traces_by_attribute.execute({+ attribute_key: " ",+ attribute_value: "x",+ })+ ).resolves.toEqual({+ success: false,+ error: "Attribute key and value are required",+ });++ (dataCollector as jest.Mock).mockResolvedValueOnce({ err: "query failed" });+ await expect(+ tools.analyze_traces_by_attribute.execute({+ attribute_key: "session.id",+ attribute_value: "s1",+ })+ ).resolves.toEqual({ success: false, error: "query failed" });++ (dataCollector as jest.Mock).mockResolvedValueOnce({+ data: [{ traceId: "t1", spanId: null }],+ err: null,+ });+ await expect(+ tools.analyze_traces_by_attribute.execute({+ attribute_key: "session.id",+ attribute_value: "s1",+ limit: 99,+ })+ ).resolves.toEqual({+ success: false,+ error: expect.stringContaining("No traces found"),+ });+ });++ it("uses default catch messages when thrown errors have empty messages", async () => {+ const tools = getChatTools("user-1", "db-1") as any;+ const emptyError = Object.assign(new Error(""), { message: "" });++ (deleteRule as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(tools.delete_rule.execute({ id: "r1" })).resolves.toEqual({+ success: false,+ error: "Failed to delete rule",+ });++ (updateContext as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.update_context.execute({ id: "c1", name: "only-name" })+ ).resolves.toEqual({+ success: false,+ error: "Failed to update context",+ });++ (upsertPromptVersion as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.update_prompt_version.execute({+ prompt_id: "p1",+ version: "1.0.0",+ prompt: "hi",+ })+ ).resolves.toEqual({+ success: false,+ error: "Failed to update prompt",+ });++ (upsertSecret as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.update_vault_secret.execute({ id: "s1", key: "k", value: "v" })+ ).resolves.toEqual({+ success: false,+ error: "Failed to update secret",+ });++ (getRules as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(tools.list_rules.execute({})).resolves.toEqual({+ success: false,+ error: "",+ });++ (getContexts as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(tools.list_contexts.execute({})).resolves.toEqual({+ success: false,+ error: "",+ });++ (getPrompts as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(tools.list_prompts.execute({})).resolves.toEqual({+ success: false,+ error: "",+ });++ (getSecrets as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(tools.list_vault_secrets.execute({})).resolves.toEqual({+ success: false,+ error: "",+ });++ (getCustomModels as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(tools.list_custom_models.execute({})).resolves.toEqual({+ success: false,+ error: "",+ });++ (getTraceImprovement as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.analyze_trace.execute({ span_id: "span-1" })+ ).resolves.toEqual({+ success: false,+ error: "Failed to run trace analysis",+ });++ (getTraceImprovement as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.get_trace_analysis.execute({ span_id: "span-1" })+ ).resolves.toEqual({+ success: false,+ error: "Failed to get trace analysis",+ });++ (getTraceImprovement as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.analyze_trace_batch.execute({ span_ids: ["s1"] })+ ).resolves.toEqual({+ success: false,+ error: "Failed to run batch trace analysis",+ });++ (dataCollector as jest.Mock).mockRejectedValueOnce(emptyError);+ await expect(+ tools.analyze_traces_by_attribute.execute({+ attribute_key: "k",+ attribute_value: "v",+ })+ ).resolves.toEqual({+ success: false,+ error: "Failed to analyze traces by attribute",+ });+ });++ it("updates context with only name and creates prompt version when version_id is omitted", async () => {+ const tools = getChatTools("user-1", "db-1") as any;+ (updateContext as jest.Mock).mockResolvedValueOnce(undefined);+ await expect(+ tools.update_context.execute({ id: "c1", name: "Renamed" })+ ).resolves.toEqual(+ expect.objectContaining({ success: true, message: "Context updated" })+ );+ expect(updateContext).toHaveBeenCalledWith("c1", { name: "Renamed" });++ (deleteRule as jest.Mock).mockResolvedValueOnce([null]);+ await expect(tools.delete_rule.execute({ id: "r1" })).resolves.toEqual({+ success: true,+ message: "Rule deleted",+ details: "ID: r1",+ });++ (upsertPromptVersion as jest.Mock).mockResolvedValueOnce(undefined);+ await expect(+ tools.update_prompt_version.execute({+ prompt_id: "p1",+ version: "2.0.0",+ prompt: "new body",+ })+ ).resolves.toEqual(+ expect.objectContaining({+ success: true,+ message: "New prompt version created",+ })+ );+ });++ it("filters falsy span ids in batch analysis", async () => {+ const tools = getChatTools("user-1", "db-1") as any;+ (getTraceImprovement as jest.Mock).mockResolvedValueOnce({+ data: {+ rootSpanId: "s1",+ runs: [+ {+ id: "run-1",+ rootSpanId: "s1",+ selectedSpanId: "s1",+ runNumber: 1,+ analysisJson: "{}",+ summary: "ok",+ modelProvider: "openai",+ modelName: "gpt-4o",+ promptTokens: 1,+ completionTokens: 1,+ cost: 0,+ worstSeverity: "info",+ createdAt: "2026-01-01T00:00:00.000Z",+ },+ ],+ },+ });++ const batch = await tools.analyze_trace_batch.execute({+ span_ids: ["", "s1", null],+ });+ expect(batch.processed).toBe(1);+ expect(batch.limitApplied).toBe(true);+ expect(batch.results[0].success).toBe(true);+ });});diff --git a/src/client/src/__tests__/lib/platform/coding-agents/cost.test.ts b/src/client/src/__tests__/lib/platform/coding-agents/cost.test.tsnew file mode 100644index 00000000..2a2bd15e--- /dev/null+++ b/src/client/src/__tests__/lib/platform/coding-agents/cost.test.ts@@ -0,0 +1,71 @@+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+ OTEL_TRACES_TABLE_NAME: "otel_traces",+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterPreviousParams: jest.fn((params) => ({+ ...params,+ timeLimit: {+ ...params.timeLimit,+ start: new Date("2023-12-31"),+ end: new Date("2024-01-01"),+ },+ })),+}));++import { dataCollector } from "@/lib/platform/common";+import { getFilterPreviousParams } from "@/helpers/server/platform";+import { getCodingAgentsTotalCost } from "@/lib/platform/coding-agents/cost";++const mockedDataCollector = dataCollector as jest.MockedFunction<+ typeof dataCollector+>;+const mockedGetFilterPreviousParams =+ getFilterPreviousParams as jest.MockedFunction<+ typeof getFilterPreviousParams+ >;++const params = {+ timeLimit: {+ start: new Date("2024-01-01T00:00:00.000Z"),+ end: new Date("2024-01-02T00:00:00.000Z"),+ type: "custom",+ },+};++describe("getCodingAgentsTotalCost", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ mockedGetFilterPreviousParams.mockClear();+ mockedDataCollector.mockResolvedValue({+ data: [{ total_cost: 2.5, previous_total_cost: 1.1 }],+ });+ });++ it("queries current and previous coding-agent costs joined by start date", async () => {+ const res = await getCodingAgentsTotalCost(params);++ expect(mockedGetFilterPreviousParams).toHaveBeenCalledWith(params);+ expect(mockedDataCollector).toHaveBeenCalledTimes(1);++ const query = mockedDataCollector.mock.calls[0][0].query as string;+ expect(query).toContain("otel_traces");+ expect(query).toContain("total_cost");+ expect(query).toContain("previous_total_cost");+ expect(query).toContain("coding_agent.");+ expect(query).toContain(String(params.timeLimit.start));+ expect(res).toEqual({+ data: [{ total_cost: 2.5, previous_total_cost: 1.1 }],+ });+ });++ it("returns dataCollector errors unchanged", async () => {+ const err = new Error("clickhouse down");+ mockedDataCollector.mockResolvedValueOnce({ data: null, err });++ const res = await getCodingAgentsTotalCost(params);++ expect(res).toEqual({ data: null, err });+ });+});diff --git a/src/client/src/__tests__/lib/platform/coding-agents/queries.test.ts b/src/client/src/__tests__/lib/platform/coding-agents/queries.test.tsindex a9b2b9d7..9d288438 100644--- a/src/client/src/__tests__/lib/platform/coding-agents/queries.test.ts+++ b/src/client/src/__tests__/lib/platform/coding-agents/queries.test.ts@@ -1,22 +1,49 @@/**- * Unit coverage for the SQL building blocks in the coding-agents- * queries layer. We test pure helpers that build clauses around the- * `coding_agent.session.is_subagent` filter and the `CHAT_ID_EXPR`- * coalesce contract. These are the regression-prone bits — the rest- * of the file is a series of templated strings around them.+ * Unit coverage for the coding-agents queries layer.*- * Integration coverage (hitting a real ClickHouse) lives in the- * playground / CI smoke layer; this file is only the *shape* of the- * SQL the function builds, which is enough to catch the two- * historical regressions:- *- * 1. is_subagent filter accidentally dropped (returns to "every- * subagent gets its own session row" behavior).- * 2. CHAT_ID_EXPR loses one of the fallback attributes, causing- * pre-Cursor-1.99 transcripts to fold into the wrong session.+ * Pure SQL builders (`buildSessionsHaving`) are tested for clause+ * shape; the exported ClickHouse helpers are exercised with a mocked+ * `dataCollector` so we cover entry points, pagination modes, cohort+ * floor behaviour, and dispute error paths without hitting a real DB.*/+import { dataCollector } from "@/lib/platform/common";+import {+ DisputeError,+ getCodingSessionDigest,+ getCodingUserDigest,+ listCodingUsers,+ listSessions,+ submitClassificationDispute,+ writeAuditLog,+} from "@/lib/platform/coding-agents/queries";import { buildSessionsHaving, escape } from "@/lib/platform/coding-agents/query-builders";+import type { CodingAgentAuth } from "@/lib/platform/coding-agents/auth";++jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+}));++jest.mock("@/clickhouse/migrations/create-coding-agents-audit-migration", () => ({+ CODING_AGENT_AUDIT_LOG_TABLE: "coding_agent_audit_log",+ CODING_AGENT_DISPUTES_TABLE: "coding_agent_disputes",+}));++const mockDataCollector = jest.mocked(dataCollector);++const auth: CodingAgentAuth = {+ userId: "user-1",+ organizationId: "org-1",+ role: "admin",+ rawRole: "admin",+};++const viewerAuth: CodingAgentAuth = {+ userId: "viewer-1",+ organizationId: "org-1",+ role: "viewer",+ rawRole: "member",+};describe("buildSessionsHaving", () => {it("hides subagent rows by default", () => {@@ -25,6 +52,17 @@ describe("buildSessionsHaving", () => {expect(out).toContain("is_subagent = 0");});+ it("documents that parent chats with folded subagents stay visible", () => {+ // Regression lock: is_subagent = 0 must remain the default hide+ // predicate. The SQL that *computes* is_subagent (in queries.ts)+ // ignores Cursor self-parent_id echoes so a parent chat that+ // absorbed Task/subagent spans is NOT classified as is_subagent=1.+ // This HAVING clause is what the Sessions tab applies after that.+ const out = buildSessionsHaving({ vendor: "cursor" });+ expect(out).toMatch(/HAVING[\s\S]*is_subagent = 0/);+ expect(out).toContain("vendor = 'cursor'");+ });+it("includes subagent rows when explicitly opted in", () => {const out = buildSessionsHaving({ includeSubagents: true });// When the operator opts in AND no other filters apply,@@ -71,3 +109,641 @@ describe("buildSessionsHaving", () => {expect(out).toContain("is_subagent = 0");});});++describe("coding agent query service", () => {+ const originalOrgFilter = process.env.OPENLIT_REQUIRE_ORG_FILTER;++ beforeEach(() => {+ jest.clearAllMocks();+ delete process.env.OPENLIT_REQUIRE_ORG_FILTER;+ Object.defineProperty(global, "crypto", {+ value: { randomUUID: jest.fn(() => "dispute-1") },+ configurable: true,+ });+ });++ afterAll(() => {+ if (originalOrgFilter === undefined) {+ delete process.env.OPENLIT_REQUIRE_ORG_FILTER;+ } else {+ process.env.OPENLIT_REQUIRE_ORG_FILTER = originalOrgFilter;+ }+ });++ it("lists sessions with cursor pagination and total count", async () => {+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [+ { session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "u1" },+ { session_id: "s2", started_at: "2026-01-01T00:00:00Z", user: "u2" },+ ],+ })+ .mockResolvedValueOnce({ err: null, data: [{ total: "12" }] });++ await expect(+ listSessions(auth, {+ limit: 1,+ withTotal: true,+ vendor: "cursor",+ user: "u1",+ classification: "work",+ })+ ).resolves.toEqual({+ rows: [{ session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "u1" }],+ nextCursor: "2026-01-02T00:00:00Z",+ total: 12,+ });++ expect(mockDataCollector).toHaveBeenCalledTimes(2);+ expect(mockDataCollector.mock.calls[0][0]).toMatchObject({+ query: expect.stringContaining("LIMIT 2"),+ });+ });++ it("lists sessions with offset pagination, sort, and time bounds", async () => {+ process.env.OPENLIT_REQUIRE_ORG_FILTER = "1";+ const since = new Date("2026-01-01T00:00:00.000Z");+ const until = new Date("2026-01-31T00:00:00.000Z");+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [+ { session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "u1" },+ ],+ });++ const result = await listSessions(auth, {+ offset: 10,+ limit: 25,+ sortBy: "cost",+ sortDir: "asc",+ since,+ until,+ cursor: "ignored-when-offset",+ });++ expect(result).toEqual({+ rows: [{ session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "u1" }],+ nextCursor: null,+ total: null,+ });++ const query = mockDataCollector.mock.calls[0][0].query as string;+ expect(query).toContain("OFFSET 10");+ expect(query).toContain("ORDER BY cost_usd ASC");+ expect(query).toContain("2026-01-01T00:00:00.000Z");+ expect(query).toContain("2026-01-31T00:00:00.000Z");+ expect(query).toContain("openlit.organization.id");+ expect(query).toContain("org-1");+ expect(query).not.toContain("ignored-when-offset");+ });++ it("lists sessions with a cursor clause and no next page", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [{ session_id: "s1", started_at: "2026-01-01T00:00:00Z", user: "u1" }],+ });++ await expect(+ listSessions(auth, { cursor: "2026-01-05T00:00:00Z", limit: 50 })+ ).resolves.toEqual({+ rows: [{ session_id: "s1", started_at: "2026-01-01T00:00:00Z", user: "u1" }],+ nextCursor: null,+ total: null,+ });++ expect(mockDataCollector.mock.calls[0][0].query).toContain(+ "WHERE started_at < '2026-01-05T00:00:00Z'"+ );+ });++ it("falls back to default sort columns for unknown sortBy values", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [{ session_id: "s1", started_at: "2026-01-01T00:00:00Z", user: "u1" }],+ });+ await listSessions(auth, {+ offset: 0,+ limit: 10,+ sortBy: "not-a-real-column" as any,+ });+ expect(mockDataCollector.mock.calls[0][0].query).toContain(+ "ORDER BY started_at DESC"+ );++ mockDataCollector.mockResolvedValueOnce({ err: null, data: [] });+ await listCodingUsers(auth, {+ sortBy: "not-a-real-column" as any,+ });+ expect(mockDataCollector.mock.calls[1][0].query).toMatch(+ /ORDER BY\s+user_last_ts\s+DESC/+ );+ });++ it("skips cohort lookup when all users are unknown/low_cohort", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [+ { session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "unknown" },+ { session_id: "s2", started_at: "2026-01-01T00:00:00Z", user: "low_cohort" },+ ],+ });++ const result = await listSessions(viewerAuth, { limit: 10 });+ expect(result.rows).toHaveLength(2);+ expect(mockDataCollector).toHaveBeenCalledTimes(1);+ });++ it("applies the cohort floor for non-admin session lists", async () => {+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [+ { session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "alice" },+ { session_id: "s2", started_at: "2026-01-01T00:00:00Z", user: "unknown" },+ ],+ })+ .mockResolvedValueOnce({+ err: null,+ data: [{ user: "alice", sessions: 2 }],+ });++ const result = await listSessions(viewerAuth, { limit: 10 });+ expect(result.rows).toEqual([+ { session_id: "s1", started_at: "2026-01-02T00:00:00Z", user: "low_cohort" },+ { session_id: "s2", started_at: "2026-01-01T00:00:00Z", user: "unknown" },+ ]);+ expect(mockDataCollector.mock.calls[1][0].query).toContain("alice");+ });++ it("throws when listSessions dataCollector fails", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: new Error("clickhouse down"),+ data: [],+ });++ await expect(listSessions(auth)).rejects.toThrow("clickhouse down");+ });++ it("lists users and normalizes numeric fields", async () => {+ mockDataCollector+ .mockResolvedValueOnce({ err: null, data: [{ total: "1" }] })+ .mockResolvedValueOnce({+ err: null,+ data: [+ {+ user: "[email protected]",+ last_seen: "2026-01-01T00:00:00Z",+ session_count: "3",+ tool_call_count: "7",+ cost_usd: "1.25",+ total_tokens: "1000",+ top_vendor: "cursor",+ classification_work: "2",+ classification_personal: "1",+ lines_added: "10",+ lines_accepted: "8",+ lines_rejected: "2",+ acceptance_pct: "80",+ commit_count: "1",+ pr_count: "1",+ },+ ],+ });++ await expect(+ listCodingUsers(auth, {+ withTotal: true,+ vendor: "cursor",+ sortBy: "tokens",+ sortDir: "asc",+ })+ ).resolves.toEqual({+ rows: [+ expect.objectContaining({+ user: "[email protected]",+ session_count: 3,+ tool_call_count: 7,+ cost_usd: 1.25,+ total_tokens: 1000,+ classification_work: 2,+ classification_personal: 1,+ lines_added: 10,+ lines_accepted: 8,+ lines_rejected: 2,+ acceptance_pct: 80,+ commit_count: 1,+ pr_count: 1,+ }),+ ],+ total: 1,+ });+ });++ it("lists users for viewers without a total query", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [{ user: "low_cohort", session_count: "1", acceptance_pct: "0" }],+ });++ const result = await listCodingUsers(viewerAuth, { limit: 10 });+ expect(result.total).toBeNull();+ expect(result.rows[0]).toMatchObject({ user: "low_cohort", session_count: 1 });+ expect(mockDataCollector).toHaveBeenCalledTimes(1);+ expect(mockDataCollector.mock.calls[0][0].query).toContain("low_cohort");+ });++ it("throws when listCodingUsers dataCollector fails", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: new Error("users query failed"),+ data: [],+ });++ await expect(listCodingUsers(auth)).rejects.toThrow("users query failed");+ });++ it("returns null for empty or missing session digests", async () => {+ await expect(getCodingSessionDigest(auth, "")).resolves.toBeNull();++ mockDataCollector.mockResolvedValueOnce({ err: null, data: [] });+ await expect(getCodingSessionDigest(auth, "s1")).resolves.toBeNull();+ });++ it("normalizes session digest rows", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [+ {+ session_id: "s1",+ lines_added: "12",+ lines_removed: "2",+ lines_accepted: "8",+ lines_rejected: "2",+ edit_accept_count: "4",+ edit_reject_count: "1",+ commit_count: "2",+ pr_count: "1",+ total_tokens: "100",+ input_tokens: "40",+ output_tokens: "60",+ cost_usd: "0.5",+ duration_ms: "3000",+ model: "gpt-4.1",+ repo_url: "https://github.com/openlit/openlit",+ branch: "main",+ working_dir: "/repo/openlit",+ working_dir_label: "repo/openlit",+ },+ ],+ });++ await expect(getCodingSessionDigest(auth, "s1")).resolves.toEqual({+ session_id: "s1",+ lines_added: 12,+ lines_removed: 2,+ lines_accepted: 8,+ lines_rejected: 2,+ edit_accept_count: 4,+ edit_reject_count: 1,+ commit_count: 2,+ pr_count: 1,+ acceptance_pct: 80,+ total_tokens: 100,+ input_tokens: 40,+ output_tokens: 60,+ cost_usd: 0.5,+ duration_ms: 3000,+ model: "gpt-4.1",+ repo_url: "https://github.com/openlit/openlit",+ branch: "main",+ working_dir: "/repo/openlit",+ working_dir_label: "repo/openlit",+ });+ });++ it("returns acceptance_pct 0 when there are no edit decisions", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: null,+ data: [+ {+ session_id: "s1",+ edit_accept_count: 0,+ edit_reject_count: 0,+ lines_added: 0,+ lines_removed: 0,+ lines_accepted: 0,+ lines_rejected: 0,+ commit_count: 0,+ pr_count: 0,+ total_tokens: 0,+ input_tokens: 0,+ output_tokens: 0,+ cost_usd: 0,+ duration_ms: 0,+ model: "",+ },+ ],+ });++ await expect(getCodingSessionDigest(auth, "s1")).resolves.toMatchObject({+ acceptance_pct: 0,+ edit_accept_count: 0,+ edit_reject_count: 0,+ });+ });++ it("hides session digests below the cohort floor for viewers", async () => {+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [+ {+ session_id: "s1",+ user: "alice",+ edit_accept_count: 0,+ edit_reject_count: 0,+ },+ ],+ })+ .mockResolvedValueOnce({ err: null, data: [{ sessions: 2 }] });++ await expect(getCodingSessionDigest(viewerAuth, "s1")).resolves.toBeNull();+ });++ it("returns session digests for viewers above the cohort floor", async () => {+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [+ {+ session_id: "s1",+ user: "alice",+ edit_accept_count: "1",+ edit_reject_count: "1",+ lines_added: 0,+ },+ ],+ })+ .mockResolvedValueOnce({ err: null, data: [{ sessions: 5 }] });++ await expect(getCodingSessionDigest(viewerAuth, "s1")).resolves.toMatchObject({+ session_id: "s1",+ acceptance_pct: 50,+ });+ });++ it("fails closed when session cohort lookup errors", async () => {+ const errorSpy = jest.spyOn(console, "error").mockImplementation();+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [{ session_id: "s1", user: "alice", edit_accept_count: 1 }],+ })+ .mockResolvedValueOnce({ err: new Error("cohort failed"), data: [] });++ await expect(getCodingSessionDigest(viewerAuth, "s1")).resolves.toBeNull();+ expect(errorSpy).toHaveBeenCalledWith(+ "coding_agent.session.cohort_lookup_failed",+ expect.any(Error)+ );+ errorSpy.mockRestore();+ });++ it("throws when getCodingSessionDigest dataCollector fails", async () => {+ mockDataCollector.mockResolvedValueOnce({+ err: new Error("digest failed"),+ data: [],+ });++ await expect(getCodingSessionDigest(auth, "s1")).rejects.toThrow(+ "digest failed"+ );+ });++ it("returns null for empty user digests", async () => {+ await expect(getCodingUserDigest(auth, "")).resolves.toBeNull();++ mockDataCollector+ .mockResolvedValueOnce({ err: null, data: [] })+ .mockResolvedValueOnce({ err: null, data: [] });+ await expect(getCodingUserDigest(auth, "alice")).resolves.toBeNull();+ });++ it("normalizes user digests and top vendors", async () => {+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [+ {+ user: "alice",+ first_seen: "2026-01-01T00:00:00Z",+ last_seen: "2026-01-02T00:00:00Z",+ session_count: "6",+ tool_call_count: "12",+ cost_usd: "1.5",+ classification_work: "4",+ classification_personal: "1",+ classification_disputed: "0",+ classification_unknown: "1",+ lines_added: "20",+ lines_removed: "5",+ lines_accepted: "15",+ lines_rejected: "5",+ edit_accept_count: "3",+ edit_reject_count: "1",+ commit_count: "2",+ pr_count: "1",+ acceptance_pct: "75",+ },+ ],+ })+ .mockResolvedValueOnce({+ err: null,+ data: [{ vendor: "cursor", sessions: "4" }],+ });++ await expect(getCodingUserDigest(auth, "alice")).resolves.toEqual({+ user: "alice",+ first_seen: "2026-01-01T00:00:00Z",+ last_seen: "2026-01-02T00:00:00Z",+ session_count: 6,+ tool_call_count: 12,+ cost_usd: 1.5,+ classification_work: 4,+ classification_personal: 1,+ classification_disputed: 0,+ classification_unknown: 1,+ lines_added: 20,+ lines_removed: 5,+ lines_accepted: 15,+ lines_rejected: 5,+ edit_accept_count: 3,+ edit_reject_count: 1,+ commit_count: 2,+ pr_count: 1,+ acceptance_pct: 75,+ top_vendors: [{ vendor: "cursor", sessions: 4 }],+ });++ expect(mockDataCollector.mock.calls[0][0].query).toContain("alice");+ expect(mockDataCollector.mock.calls[1][0].query).toContain("GROUP BY vendor");+ });++ it("hides user digests below the cohort floor for viewers", async () => {+ mockDataCollector+ .mockResolvedValueOnce({+ err: null,+ data: [{ user: "alice", session_count: "2", acceptance_pct: 0 }],+ })+ .mockResolvedValueOnce({ err: null, data: [] });++ await expect(getCodingUserDigest(viewerAuth, "alice")).resolves.toBeNull();+ });++ it("throws when getCodingUserDigest dataCollector fails", async () => {+ mockDataCollector+ .mockResolvedValueOnce({ err: new Error("user digest failed"), data: [] })+ .mockResolvedValueOnce({ err: null, data: [] });++ await expect(getCodingUserDigest(auth, "alice")).rejects.toThrow(+ "user digest failed"+ );+ });++ it("submits classification disputes and audit logs", async () => {+ mockDataCollector+ .mockResolvedValueOnce({ err: null, data: [{ hit: 1 }] })+ .mockResolvedValueOnce({ err: null, data: [] })+ .mockResolvedValueOnce({ err: null, data: [{ n: 0 }] })+ .mockResolvedValueOnce({ err: null, data: [] })+ .mockResolvedValueOnce({ err: null, data: [] });++ await expect(+ submitClassificationDispute(auth, {+ sessionId: "s1",+ currentClassification: "work",+ requestedClassification: "personal",+ rationale: "Personal experiment.",+ })+ ).resolves.toEqual({ id: "dispute-1" });++ expect(mockDataCollector).toHaveBeenLastCalledWith(+ expect.objectContaining({+ values: [+ expect.objectContaining({+ action: "coding_agent.classification.dispute",+ subject: "s1",+ }),+ ],+ }),+ "insert"+ );+ });++ it.each([+ [[{ err: null, data: [] }], "not_found", 404],+ [+ [+ { err: null, data: [{ hit: 1 }] },+ { err: null, data: [{ hit: 1 }] },+ ],+ "duplicate",+ 409,+ ],+ [+ [+ { err: null, data: [{ hit: 1 }] },+ { err: null, data: [] },+ { err: null, data: [{ n: 20 }] },+ ],+ "rate_limited",+ 429,+ ],+ // Fail-closed lookup errors map to the same user-facing codes.+ [[{ err: new Error("lookup failed"), data: [] }], "not_found", 404],+ [+ [+ { err: null, data: [{ hit: 1 }] },+ { err: new Error("dedupe failed"), data: [] },+ ],+ "duplicate",+ 409,+ ],+ [+ [+ { err: null, data: [{ hit: 1 }] },+ { err: null, data: [] },+ { err: new Error("rate failed"), data: [] },+ ],+ "rate_limited",+ 429,+ ],+ ])(+ "throws %s dispute errors",+ async (responses, code, status) => {+ const errorSpy = jest.spyOn(console, "error").mockImplementation();+ for (const response of responses) {+ mockDataCollector.mockResolvedValueOnce(response as any);+ }++ await expect(+ submitClassificationDispute(auth, {+ sessionId: "s1",+ currentClassification: "work",+ requestedClassification: "personal",+ rationale: "Personal experiment.",+ })+ ).rejects.toMatchObject({ code, status });+ errorSpy.mockRestore();+ }+ );++ it("throws when dispute insert fails", async () => {+ mockDataCollector+ .mockResolvedValueOnce({ err: null, data: [{ hit: 1 }] })+ .mockResolvedValueOnce({ err: null, data: [] })+ .mockResolvedValueOnce({ err: null, data: [{ n: 0 }] })+ .mockResolvedValueOnce({ err: new Error("insert failed"), data: [] });++ await expect(+ submitClassificationDispute(auth, {+ sessionId: "s1",+ currentClassification: "work",+ requestedClassification: "personal",+ rationale: "Personal experiment.",+ })+ ).rejects.toThrow("insert failed");+ });++ it("swallows audit log insert failures", async () => {+ const errorSpy = jest.spyOn(console, "error").mockImplementation();+ mockDataCollector.mockResolvedValueOnce({+ err: new Error("insert failed"),+ data: [],+ });++ await expect(+ writeAuditLog(auth, {+ action: "coding_agent.test",+ subject: "s1",+ })+ ).resolves.toBeUndefined();++ expect(errorSpy).toHaveBeenCalledWith(+ "coding_agent.audit_log.insert_failed",+ expect.any(Error)+ );+ errorSpy.mockRestore();+ });+});++describe("DisputeError", () => {+ it("carries status and machine-readable codes", () => {+ const error = new DisputeError("duplicate", 409, "Duplicate");++ expect(error).toBeInstanceOf(Error);+ expect(error.code).toBe("duplicate");+ expect(error.status).toBe(409);+ expect(error.message).toBe("Duplicate");+ });+});diff --git a/src/client/src/__tests__/lib/platform/controller/agent-feature.test.ts b/src/client/src/__tests__/lib/platform/controller/agent-feature.test.tsindex 41c68f13..7c912c23 100644--- a/src/client/src/__tests__/lib/platform/controller/agent-feature.test.ts+++ b/src/client/src/__tests__/lib/platform/controller/agent-feature.test.ts@@ -1,3 +1,4 @@+jest.mock("@/lib/platform/controller", () => ({getServiceById: jest.fn(),getControllerInstanceById: jest.fn(),@@ -8,10 +9,79 @@ jest.mock("@/lib/platform/controller", () => ({updateFeatureDesiredState: jest.fn(),}));-import agentHandler from "@/lib/platform/controller/features/agent";+import {+ getServiceById,+ getControllerInstanceById,+ getControllerIdsForWorkload,+ getControllerConfig,+ getFeatureDesiredStates,+ queueAction,+ updateFeatureDesiredState,+} from "@/lib/platform/controller";+import agentHandler, {+ buildAgentStatusResponse,+} from "@/lib/platform/controller/features/agent";import { KNOWN_ACTIONS } from "@/types/controller";+const mockGetServiceById = getServiceById as jest.Mock;+const mockGetInstance = getControllerInstanceById as jest.Mock;+const mockGetControllerIds = getControllerIdsForWorkload as jest.Mock;+const mockGetConfig = getControllerConfig as jest.Mock;+const mockGetDesired = getFeatureDesiredStates as jest.Mock;+const mockQueueAction = queueAction as jest.Mock;+const mockUpdateDesired = updateFeatureDesiredState as jest.Mock;++function capablePythonService(+ overrides: Record<string, unknown> = {},+ instanceOverrides: Record<string, unknown> = {}+) {+ mockGetServiceById.mockResolvedValue({+ data: [+ {+ id: "svc-1",+ workload_key: "linux:python-ai-app",+ cluster_id: "default",+ service_name: "python-ai-app",+ namespace: "default",+ controller_instance_id: "ctrl-1",+ language_runtime: "python",+ resource_attributes: {+ "openlit.agent_observability.status": "disabled",+ },+ ...overrides,+ },+ ],+ });+ mockGetInstance.mockResolvedValue({+ data: [+ {+ mode: "linux",+ resource_attributes: {+ "controller.capabilities": "python_sdk_injection_linux_systemd_v1",+ },+ ...instanceOverrides,+ },+ ],+ });+ mockGetControllerIds.mockResolvedValue({ data: [] });+ mockQueueAction.mockResolvedValue({ data: "ok" });+ mockUpdateDesired.mockResolvedValue({ data: "ok" });+ mockGetConfig.mockResolvedValue({ data: [] });+ mockGetDesired.mockResolvedValue({ data: [] });+}++beforeEach(() => {+ jest.clearAllMocks();+});+describe("agent controller feature", () => {+ it("validates enable/disable/status only", () => {+ expect(agentHandler.validatePayload("enable", {})).toBeNull();+ expect(agentHandler.validatePayload("disable", {})).toBeNull();+ expect(agentHandler.validatePayload("status", {})).toBeNull();+ expect(agentHandler.validatePayload("push", {})).toMatch(/Unknown operation/);+ });+it("queues nodejs runtime payloads for JavaScript/TypeScript services", () => {const actions = agentHandler.reconcile([@@ -80,4 +150,381 @@ describe("agent controller feature", () => {expect(actions[0].actionType).toBe(KNOWN_ACTIONS.DISABLE_AGENT);expect(JSON.parse(actions[0].payload).target_runtime).toBe("python");});++ it("normalizes node runtime alias and uses desired config when non-empty", () => {+ const actions = agentHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "discovered",+ language_runtime: "node",+ resource_attributes: {},+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "agent",+ desired_status: "enabled",+ config: '{"custom":true}',+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions[0].payload).toBe('{"custom":true}');+ });++ it("skips unsupported runtimes and missing desired states", () => {+ const actions = agentHandler.reconcile(+ [+ {+ workload_key: "wk-java",+ instrumentation_status: "discovered",+ language_runtime: "java",+ },+ {+ workload_key: "wk-skip",+ instrumentation_status: "discovered",+ language_runtime: "python",+ },+ ],+ new Map([+ [+ "wk-java",+ {+ workload_key: "wk-java",+ cluster_id: "default",+ feature: "agent",+ desired_status: "enabled",+ config: "{}",+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions).toEqual([]);+ });++ it("returns 404 when service is missing", async () => {+ mockGetServiceById.mockResolvedValue({ data: [] });+ const res = await agentHandler.applyOperation("svc-1", "status", {});+ expect(res.status).toBe(404);+ });++ it("returns 400 for unsupported language runtime", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [+ {+ language_runtime: "java",+ controller_instance_id: "ctrl-1",+ },+ ],+ });+ const res = await agentHandler.applyOperation("svc-1", "enable", {});+ expect(res.status).toBe(400);+ });++ it("returns status payload for status operation", async () => {+ capablePythonService();+ const res = await agentHandler.applyOperation("svc-1", "status", {});+ expect(res.status).toBe(200);+ const body = await res.json();+ expect(body).toMatchObject({+ supported: true,+ mode: "linux",+ status: "disabled",+ service: "python-ai-app",+ });+ });++ it("returns 409 when controller lacks agent capability", async () => {+ capablePythonService({}, {+ mode: "kubernetes",+ resource_attributes: { "controller.capabilities": "lifecycle_kubernetes_v1" },+ });+ const res = await agentHandler.applyOperation("svc-1", "enable", {});+ expect(res.status).toBe(409);+ });++ it("returns 500 when workload_key is missing on enable", async () => {+ capablePythonService({ workload_key: "" });+ const res = await agentHandler.applyOperation("svc-1", "enable", {});+ expect(res.status).toBe(500);+ });++ it("queues ENABLE_AGENT with export overrides from controller config", async () => {+ capablePythonService();+ mockGetControllerIds.mockResolvedValue({+ data: [{ controller_instance_id: "ctrl-2" }],+ });+ mockGetConfig.mockResolvedValue({+ data: [+ {+ config: JSON.stringify({+ export: {+ otlp_endpoint: "http://collector:4318",+ otlp_protocol: "http/protobuf",+ otlp_headers: { "x-api-key": "k" },+ otlp_traces_endpoint: "http://t",+ otlp_metrics_endpoint: "http://m",+ otlp_logs_endpoint: "http://l",+ },+ }),+ },+ ],+ });++ const res = await agentHandler.applyOperation(+ "svc-1",+ "enable",+ { enable_http_instrumentation: false },+ "db-1"+ );+ const body = await res.json();++ expect(body.status).toBe("queued");+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ "linux:python-ai-app",+ "default",+ "agent",+ "enabled",+ "{}",+ "db-1"+ );+ const payload = JSON.parse(mockQueueAction.mock.calls[0][3]);+ expect(payload).toMatchObject({+ target_runtime: "python",+ otlp_endpoint: "http://collector:4318",+ otlp_protocol: "http/protobuf",+ enable_http_instrumentation: false,+ });+ });++ it("allows payload otlp_endpoint to override config", async () => {+ capablePythonService();+ mockGetConfig.mockResolvedValue({+ data: [+ {+ config: JSON.stringify({+ export: { otlp_endpoint: "http://from-config" },+ }),+ },+ ],+ });++ await agentHandler.applyOperation("svc-1", "enable", {+ otlp_endpoint: "http://from-payload",+ });++ const payload = JSON.parse(mockQueueAction.mock.calls[0][3]);+ expect(payload.otlp_endpoint).toBe("http://from-payload");+ });++ it("tolerates invalid controller config JSON on enable", async () => {+ capablePythonService();+ mockGetConfig.mockResolvedValue({ data: [{ config: "{not-json" }] });++ const res = await agentHandler.applyOperation("svc-1", "enable", {});+ expect(res.status).toBe(200);+ });++ it("queues DISABLE_AGENT and falls back to service controller", async () => {+ capablePythonService({ cluster_id: undefined, namespace: undefined });++ const res = await agentHandler.applyOperation("svc-1", "disable", {});+ const body = await res.json();++ expect(body.message).toMatch(/removal queued/);+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ "linux:python-ai-app",+ "default",+ "agent",+ "none",+ "{}",+ undefined+ );+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.DISABLE_AGENT,+ "linux:python-ai-app",+ expect.any(String),+ undefined+ );+ });++ it("supports kubernetes and docker capability exact matches", async () => {+ capablePythonService(+ { language_runtime: "nodejs" },+ {+ mode: "kubernetes",+ resource_attributes: {+ "controller.capabilities": "nodejs_sdk_injection_kubernetes_v1",+ },+ }+ );+ const res = await agentHandler.applyOperation("svc-1", "disable", {});+ expect(res.status).toBe(200);++ capablePythonService(+ {},+ {+ mode: "docker",+ resource_attributes: {+ "controller.capabilities": "python_sdk_injection_docker_v1",+ },+ }+ );+ const res2 = await agentHandler.applyOperation("svc-1", "disable", {});+ expect(res2.status).toBe(200);+ });++ it("returns 500 on unexpected errors", async () => {+ mockGetServiceById.mockRejectedValue(new Error("boom"));+ const res = await agentHandler.applyOperation("svc-1", "status", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({ error: "boom" });+ });++ it("uses fallback error message for non-Error throws", async () => {+ mockGetServiceById.mockRejectedValue({});+ const res = await agentHandler.applyOperation("svc-1", "status", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({+ error: "Agent operation failed",+ });+ });+});++describe("buildAgentStatusResponse", () => {+ beforeEach(() => {+ mockGetDesired.mockResolvedValue({ data: [] });+ });++ it("marks automatable false for user-scoped systemd", async () => {+ const status = await buildAgentStatusResponse(+ {+ service_name: "app",+ workload_key: "wk-1",+ language_runtime: "python",+ resource_attributes: { "systemd.scope": "user" },+ },+ true,+ "linux"+ );+ expect(status.automatable).toBe(false);+ expect(status.reason).toMatch(/User-scoped systemd/);+ });++ it("marks automatable false when docker metadata is missing", async () => {+ const status = await buildAgentStatusResponse(+ {+ service_name: "app",+ language_runtime: "python",+ resource_attributes: {},+ },+ true,+ "docker"+ );+ expect(status.automatable).toBe(false);+ expect(status.reason).toMatch(/Docker container metadata/);+ });++ it("marks automatable false for existing_openlit source", async () => {+ const status = await buildAgentStatusResponse(+ {+ service_name: "app",+ language_runtime: "python",+ resource_attributes: {+ "openlit.agent_observability.status": "enabled",+ "openlit.agent_observability.source": "existing_openlit",+ },+ },+ true,+ "linux"+ );+ expect(status.automatable).toBe(false);+ expect(status.reason).toMatch(/not controller-managed/);+ });++ it("uses preflight reason when unsupported and no attr reason", async () => {+ const docker = await buildAgentStatusResponse(+ { service_name: "app", language_runtime: "python", resource_attributes: {} },+ false,+ "docker"+ );+ expect(docker.reason).toMatch(/Docker Agent Observability/);++ const linux = await buildAgentStatusResponse(+ { service_name: "app", language_runtime: "nodejs", resource_attributes: {} },+ false,+ "linux"+ );+ expect(linux.reason).toMatch(/JavaScript\/TypeScript/);++ const other = await buildAgentStatusResponse(+ { service_name: "app", language_runtime: "python", resource_attributes: {} },+ false,+ "kubernetes"+ );+ expect(other.reason).toMatch(/does not advertise/);+ });++ it("detects naked pods and transitioning desired state", async () => {+ mockGetDesired.mockResolvedValue({+ data: [{ desired_status: "enabled" }],+ });++ const status = await buildAgentStatusResponse(+ {+ service_name: "app",+ namespace: "ns",+ workload_key: "wk-1",+ cluster_id: "c1",+ language_runtime: "python",+ resource_attributes: {+ "openlit.agent_observability.status": "disabled",+ "k8s.workload.kind": "Pod",+ "openlit.is_containerized": "true",+ },+ },+ true,+ "kubernetes",+ "db-1"+ );++ expect(status).toMatchObject({+ is_naked_pod: true,+ transitioning: true,+ desired_status: "enabled",+ is_containerized: true,+ namespace: "ns",+ workload_kind: "Pod",+ });+ });++ it("flags manual status and unsupported runtime reason", async () => {+ const status = await buildAgentStatusResponse(+ {+ service_name: "app",+ language_runtime: "go",+ resource_attributes: {+ "openlit.agent_observability.status": "manual",+ "openlit.observability.reason":+ "Agent observability is only available for Python and JavaScript/TypeScript services",+ },+ },+ false,+ "linux"+ );+ expect(status.enabled).toBe(true);+ expect(status.is_manual).toBe(true);+ expect(status.supported).toBe(false);+ expect(status.reason).toMatch(/only available for Python/);+ });});diff --git a/src/client/src/__tests__/lib/platform/controller/envs-feature.test.ts b/src/client/src/__tests__/lib/platform/controller/envs-feature.test.tsnew file mode 100644index 00000000..adf5090b--- /dev/null+++ b/src/client/src/__tests__/lib/platform/controller/envs-feature.test.ts@@ -0,0 +1,144 @@++jest.mock("@/lib/platform/controller", () => ({+ getServiceById: jest.fn(),+ getControllerInstanceById: jest.fn(),+ getControllerIdsForWorkload: jest.fn(),+ getControllerConfig: jest.fn(),+ getFeatureDesiredStates: jest.fn(),+ queueAction: jest.fn(),+ updateFeatureDesiredState: jest.fn(),+}));++import {+ getServiceById,+ getControllerIdsForWorkload,+ queueAction,+ updateFeatureDesiredState,+} from "@/lib/platform/controller";+import envsHandler from "@/lib/platform/controller/features/envs";+import { KNOWN_ACTIONS } from "@/types/controller";++const mockGetServiceById = getServiceById as jest.Mock;+const mockGetControllerIds = getControllerIdsForWorkload as jest.Mock;+const mockQueueAction = queueAction as jest.Mock;+const mockUpdateDesired = updateFeatureDesiredState as jest.Mock;++const baseService = {+ id: "svc-1",+ workload_key: "k8s:default:deployment/app",+ cluster_id: "cluster-a",+ service_name: "app",+ namespace: "default",+ controller_instance_id: "ctrl-1",+};++beforeEach(() => {+ jest.clearAllMocks();+ mockGetControllerIds.mockResolvedValue({ data: [] });+ mockQueueAction.mockResolvedValue({ data: "ok" });+ mockUpdateDesired.mockResolvedValue({ data: "ok" });+});++describe("envs feature", () => {+ it("validates push/remove only", () => {+ expect(envsHandler.validatePayload("push", {})).toBeNull();+ expect(envsHandler.validatePayload("remove", {})).toBeNull();+ expect(envsHandler.validatePayload("enable", {})).toMatch(/Unknown operation/);+ });++ it("returns empty reconcile actions (stub)", () => {+ expect(envsHandler.reconcile([], new Map())).toEqual([]);+ });++ it("returns 404 when service is missing", async () => {+ mockGetServiceById.mockResolvedValue({ data: [] });+ const res = await envsHandler.applyOperation("svc-1", "push", { A: "1" });+ expect(res.status).toBe(404);+ });++ it("returns 500 when workload_key is missing", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ ...baseService, workload_key: "" }],+ });+ const res = await envsHandler.applyOperation("svc-1", "push", { A: "1" });+ expect(res.status).toBe(500);+ });++ it("queues PUSH_ENVS and updates desired state", async () => {+ mockGetServiceById.mockResolvedValue({ data: [baseService] });+ mockGetControllerIds.mockResolvedValue({+ data: [{ controller_instance_id: "ctrl-2" }],+ });++ const res = await envsHandler.applyOperation(+ "svc-1",+ "push",+ { FOO: "bar" },+ "db-1"+ );+ const body = await res.json();++ expect(body).toMatchObject({+ status: "queued",+ action: KNOWN_ACTIONS.PUSH_ENVS,+ controllers: 1,+ });+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ baseService.workload_key,+ "cluster-a",+ "envs",+ "active",+ JSON.stringify({ FOO: "bar" }),+ "db-1"+ );+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-2",+ KNOWN_ACTIONS.PUSH_ENVS,+ baseService.workload_key,+ JSON.stringify({ FOO: "bar" }),+ "db-1"+ );+ });++ it("falls back to service controller when no workload controllers found", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ ...baseService, cluster_id: undefined, namespace: undefined }],+ });++ const res = await envsHandler.applyOperation("svc-1", "remove", {});+ const body = await res.json();++ expect(body.action).toBe(KNOWN_ACTIONS.REMOVE_ENVS);+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ baseService.workload_key,+ "default",+ "envs",+ "none",+ "{}",+ undefined+ );+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.REMOVE_ENVS,+ baseService.workload_key,+ "{}",+ undefined+ );+ });++ it("returns 500 when an unexpected error is thrown", async () => {+ mockGetServiceById.mockRejectedValue(new Error("db boom"));+ const res = await envsHandler.applyOperation("svc-1", "push", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({ error: "db boom" });+ });++ it("uses fallback error message for non-Error throws", async () => {+ mockGetServiceById.mockRejectedValue({});+ const res = await envsHandler.applyOperation("svc-1", "push", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({+ error: "ENVs operation failed",+ });+ });+});diff --git a/src/client/src/__tests__/lib/platform/controller/features-index.test.ts b/src/client/src/__tests__/lib/platform/controller/features-index.test.tsnew file mode 100644index 00000000..2c8db71f--- /dev/null+++ b/src/client/src/__tests__/lib/platform/controller/features-index.test.ts@@ -0,0 +1,27 @@+jest.mock("@/lib/platform/controller", () => ({+ getServiceById: jest.fn(),+ getControllerInstanceById: jest.fn(),+ getControllerIdsForWorkload: jest.fn(),+ getControllerConfig: jest.fn(),+ getFeatureDesiredStates: jest.fn(),+ queueAction: jest.fn(),+ updateFeatureDesiredState: jest.fn(),+}));++import {+ getAllFeatureHandlers,+ getFeatureHandler,+ registerFeature,+} from "@/lib/platform/controller/features";++describe("controller features index", () => {+ it("re-exports registry helpers and side-loads feature modules", () => {+ expect(typeof registerFeature).toBe("function");+ expect(typeof getFeatureHandler).toBe("function");+ expect(typeof getAllFeatureHandlers).toBe("function");++ const handlers = getAllFeatureHandlers();+ expect(handlers.length).toBeGreaterThan(0);+ expect(getFeatureHandler(handlers[0].feature)).toBe(handlers[0]);+ });+});diff --git a/src/client/src/__tests__/lib/platform/controller/instrumentation-feature.test.ts b/src/client/src/__tests__/lib/platform/controller/instrumentation-feature.test.tsnew file mode 100644index 00000000..3ad4cfd0--- /dev/null+++ b/src/client/src/__tests__/lib/platform/controller/instrumentation-feature.test.ts@@ -0,0 +1,276 @@++jest.mock("@/lib/platform/controller", () => ({+ getServiceById: jest.fn(),+ getControllerInstanceById: jest.fn(),+ getControllerIdsForWorkload: jest.fn(),+ getControllerConfig: jest.fn(),+ getFeatureDesiredStates: jest.fn(),+ queueAction: jest.fn(),+ updateFeatureDesiredState: jest.fn(),+}));++import {+ getServiceById,+ getControllerIdsForWorkload,+ queueAction,+ updateFeatureDesiredState,+} from "@/lib/platform/controller";+import instrumentationHandler from "@/lib/platform/controller/features/instrumentation";+import { KNOWN_ACTIONS } from "@/types/controller";++const mockGetServiceById = getServiceById as jest.Mock;+const mockGetControllerIds = getControllerIdsForWorkload as jest.Mock;+const mockQueueAction = queueAction as jest.Mock;+const mockUpdateDesired = updateFeatureDesiredState as jest.Mock;++const baseService = {+ id: "svc-1",+ workload_key: "k8s:default:deployment/app",+ cluster_id: "cluster-a",+ service_name: "app",+ namespace: "default",+ controller_instance_id: "ctrl-1",+ instrumentation_status: "discovered",+};++beforeEach(() => {+ jest.clearAllMocks();+ mockGetControllerIds.mockResolvedValue({ data: [] });+ mockQueueAction.mockResolvedValue({ data: "ok" });+ mockUpdateDesired.mockResolvedValue({ data: "ok" });+});++describe("instrumentation feature", () => {+ it("validates enable/disable only", () => {+ expect(instrumentationHandler.validatePayload("enable", {})).toBeNull();+ expect(instrumentationHandler.validatePayload("disable", {})).toBeNull();+ expect(instrumentationHandler.validatePayload("push", {})).toMatch(+ /Unknown operation/+ );+ });++ it("returns 404 when service is missing", async () => {+ mockGetServiceById.mockResolvedValue({ data: [] });+ const res = await instrumentationHandler.applyOperation(+ "svc-1",+ "enable",+ {}+ );+ expect(res.status).toBe(404);+ });++ it("returns 500 when workload_key is missing", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ ...baseService, workload_key: undefined }],+ });+ const res = await instrumentationHandler.applyOperation(+ "svc-1",+ "enable",+ {}+ );+ expect(res.status).toBe(500);+ });++ it("queues INSTRUMENT on enable", async () => {+ mockGetServiceById.mockResolvedValue({ data: [baseService] });+ mockGetControllerIds.mockResolvedValue({+ data: [{ controller_instance_id: "ctrl-2" }],+ });++ const res = await instrumentationHandler.applyOperation(+ "svc-1",+ "enable",+ {},+ "db-1"+ );+ const body = await res.json();++ expect(body).toEqual({+ status: "queued",+ action: KNOWN_ACTIONS.INSTRUMENT,+ controllers: 1,+ });+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ baseService.workload_key,+ "cluster-a",+ "instrumentation",+ "instrumented",+ "{}",+ "db-1"+ );+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-2",+ KNOWN_ACTIONS.INSTRUMENT,+ baseService.workload_key,+ "{}",+ "db-1"+ );+ });++ it("queues UNINSTRUMENT and falls back to service controller", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ ...baseService, cluster_id: undefined, namespace: undefined }],+ });++ const res = await instrumentationHandler.applyOperation(+ "svc-1",+ "disable",+ {}+ );+ const body = await res.json();++ expect(body.action).toBe(KNOWN_ACTIONS.UNINSTRUMENT);+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ baseService.workload_key,+ "default",+ "instrumentation",+ "none",+ "{}",+ undefined+ );+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.UNINSTRUMENT,+ baseService.workload_key,+ "{}",+ undefined+ );+ });++ it("returns 500 on unexpected errors", async () => {+ mockGetServiceById.mockRejectedValue(new Error("boom"));+ const res = await instrumentationHandler.applyOperation(+ "svc-1",+ "enable",+ {}+ );+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({ error: "boom" });+ });++ it("uses fallback error message for non-Error throws", async () => {+ mockGetServiceById.mockRejectedValue({});+ const res = await instrumentationHandler.applyOperation(+ "svc-1",+ "enable",+ {}+ );+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({+ error: "Instrumentation operation failed",+ });+ });++ describe("reconcile", () => {+ it("emits INSTRUMENT when desired instrumented and actual is not", () => {+ const actions = instrumentationHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "discovered",+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "instrumentation",+ desired_status: "instrumented",+ config: '{"x":1}',+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions).toEqual([+ {+ actionType: KNOWN_ACTIONS.INSTRUMENT,+ serviceKey: "wk-1",+ payload: '{"x":1}',+ },+ ]);+ });++ it("emits UNINSTRUMENT when desired none and actual instrumented", () => {+ const actions = instrumentationHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "instrumented",+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "instrumentation",+ desired_status: "none",+ config: "{}",+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions[0].actionType).toBe(KNOWN_ACTIONS.UNINSTRUMENT);+ });++ it("skips services without desired state and matching statuses", () => {+ const actions = instrumentationHandler.reconcile(+ [+ { workload_key: "wk-skip", instrumentation_status: "discovered" },+ {+ workload_key: "wk-ok",+ instrumentation_status: "instrumented",+ },+ ],+ new Map([+ [+ "wk-ok",+ {+ workload_key: "wk-ok",+ cluster_id: "default",+ feature: "instrumentation",+ desired_status: "instrumented",+ config: "{}",+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions).toEqual([]);+ });++ it("defaults payload to {} when desired config is missing", () => {+ const actions = instrumentationHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "discovered",+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "instrumentation",+ desired_status: "instrumented",+ config: undefined as any,+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions[0].payload).toBe("{}");+ });+ });+});diff --git a/src/client/src/__tests__/lib/platform/controller/lifecycle-feature.test.ts b/src/client/src/__tests__/lib/platform/controller/lifecycle-feature.test.tsnew file mode 100644index 00000000..aa1a402d--- /dev/null+++ b/src/client/src/__tests__/lib/platform/controller/lifecycle-feature.test.ts@@ -0,0 +1,355 @@++jest.mock("@/lib/platform/controller", () => ({+ getServiceById: jest.fn(),+ getControllerInstanceById: jest.fn(),+ getControllerIdsForWorkload: jest.fn(),+ getControllerConfig: jest.fn(),+ getFeatureDesiredStates: jest.fn(),+ queueAction: jest.fn(),+ updateFeatureDesiredState: jest.fn(),+}));++import {+ getServiceById,+ getControllerInstanceById,+ getControllerIdsForWorkload,+ getFeatureDesiredStates,+ queueAction,+ updateFeatureDesiredState,+} from "@/lib/platform/controller";+import lifecycleHandler from "@/lib/platform/controller/features/lifecycle";+import { KNOWN_ACTIONS } from "@/types/controller";++const mockGetServiceById = getServiceById as jest.Mock;+const mockGetInstance = getControllerInstanceById as jest.Mock;+const mockGetControllerIds = getControllerIdsForWorkload as jest.Mock;+const mockGetDesired = getFeatureDesiredStates as jest.Mock;+const mockQueueAction = queueAction as jest.Mock;+const mockUpdateDesired = updateFeatureDesiredState as jest.Mock;++function mockCapableInstance(+ mode: string,+ capabilities: string,+ overrides: Record<string, unknown> = {}+) {+ mockGetServiceById.mockResolvedValue({+ data: [+ {+ id: "svc-1",+ workload_key: "wk-1",+ cluster_id: "default",+ service_name: "app",+ namespace: "ns",+ controller_instance_id: "ctrl-1",+ resource_attributes: {},+ ...overrides,+ },+ ],+ });+ mockGetInstance.mockResolvedValue({+ data: [+ {+ mode,+ resource_attributes: { "controller.capabilities": capabilities },+ },+ ],+ });+ mockGetControllerIds.mockResolvedValue({ data: [] });+ mockQueueAction.mockResolvedValue({ data: "ok" });+ mockUpdateDesired.mockResolvedValue({ data: "ok" });+}++beforeEach(() => {+ jest.clearAllMocks();+});++describe("lifecycle feature", () => {+ it("validates start/stop/restart only", () => {+ expect(lifecycleHandler.validatePayload("start", {})).toBeNull();+ expect(lifecycleHandler.validatePayload("stop", {})).toBeNull();+ expect(lifecycleHandler.validatePayload("restart", {})).toBeNull();+ expect(lifecycleHandler.validatePayload("enable", {})).toMatch(+ /Unknown operation/+ );+ });++ it("returns 404 when service is missing", async () => {+ mockGetServiceById.mockResolvedValue({ data: [] });+ const res = await lifecycleHandler.applyOperation("svc-1", "stop", {});+ expect(res.status).toBe(404);+ });++ it("returns 500 when workload_key is missing", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ controller_instance_id: "ctrl-1", workload_key: "" }],+ });+ const res = await lifecycleHandler.applyOperation("svc-1", "stop", {});+ expect(res.status).toBe(500);+ });++ it("returns 409 when controller lacks lifecycle capability", async () => {+ mockCapableInstance("kubernetes", "python_sdk_injection_kubernetes_v1");+ const res = await lifecycleHandler.applyOperation("svc-1", "stop", {});+ expect(res.status).toBe(409);+ });++ it("queues STOP_WORKLOAD when capable", async () => {+ mockCapableInstance("kubernetes", "lifecycle_kubernetes_v1");+ mockGetControllerIds.mockResolvedValue({+ data: [{ controller_instance_id: "ctrl-2" }],+ });++ const res = await lifecycleHandler.applyOperation("svc-1", "stop", {}, "db-1");+ const body = await res.json();++ expect(body).toEqual({+ status: "queued",+ action: KNOWN_ACTIONS.STOP_WORKLOAD,+ controllers: 1,+ });+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ "wk-1",+ "default",+ "lifecycle",+ "stopped",+ "{}",+ "db-1"+ );+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-2",+ KNOWN_ACTIONS.STOP_WORKLOAD,+ "wk-1",+ "{}",+ "db-1"+ );+ });++ it("queues RESTART_WORKLOAD without updating desired state", async () => {+ mockCapableInstance("docker", "lifecycle_docker_v1");++ const res = await lifecycleHandler.applyOperation("svc-1", "restart", {});+ const body = await res.json();++ expect(body.action).toBe(KNOWN_ACTIONS.RESTART_WORKLOAD);+ expect(mockUpdateDesired).not.toHaveBeenCalled();+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.RESTART_WORKLOAD,+ "wk-1",+ "{}",+ undefined+ );+ });++ it("supports linux prefix capabilities", async () => {+ mockCapableInstance("linux", "lifecycle_linux_systemd_v1,other");+ const res = await lifecycleHandler.applyOperation("svc-1", "restart", {});+ expect(res.status).toBe(200);+ });++ it("defaults mode to linux when instance is missing", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [+ {+ workload_key: "wk-1",+ service_name: "app",+ controller_instance_id: "ctrl-1",+ resource_attributes: { "systemd.unit": "app.service" },+ },+ ],+ });+ mockGetInstance.mockResolvedValue({ data: [] });+ mockGetControllerIds.mockResolvedValue({ data: null });+ mockGetDesired.mockResolvedValue({ data: [] });++ const res = await lifecycleHandler.applyOperation("svc-1", "start", {});+ // no lifecycle_linux_* capability on empty instance+ expect(res.status).toBe(409);+ });++ it("rejects start for linux bare process without snapshot", async () => {+ mockCapableInstance("linux", "lifecycle_linux_bare_v1", {+ resource_attributes: {},+ });+ mockGetDesired.mockResolvedValue({ data: [{ feature: "lifecycle", config: "{}" }] });++ const res = await lifecycleHandler.applyOperation("svc-1", "start", {});+ expect(res.status).toBe(409);+ await expect(res.json()).resolves.toMatchObject({+ error: expect.stringContaining("No saved snapshot"),+ });+ });++ it("rejects start for k8s naked pod without snapshot", async () => {+ mockCapableInstance("kubernetes", "lifecycle_kubernetes_v1", {+ resource_attributes: { "k8s.workload.kind": "Pod" },+ });+ mockGetDesired.mockResolvedValue({ data: [] });++ const res = await lifecycleHandler.applyOperation("svc-1", "start", {});+ expect(res.status).toBe(409);+ });++ it("starts with snapshot for naked pod / bare process", async () => {+ mockCapableInstance("linux", "lifecycle_linux_bare_v1");+ mockGetDesired.mockResolvedValue({+ data: [{ feature: "lifecycle", config: '{"exe":"/bin/app"}' }],+ });++ const res = await lifecycleHandler.applyOperation("svc-1", "start", {}, "db-1");+ const body = await res.json();++ expect(body.action).toBe(KNOWN_ACTIONS.START_WORKLOAD);+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.START_WORKLOAD,+ "wk-1",+ '{"exe":"/bin/app"}',+ "db-1"+ );+ });++ it("allows start without snapshot for systemd units", async () => {+ mockCapableInstance("linux", "lifecycle_linux_systemd_v1", {+ resource_attributes: { "systemd.unit": "app.service" },+ });+ mockGetDesired.mockResolvedValue({ data: [] });++ const res = await lifecycleHandler.applyOperation("svc-1", "start", {});+ expect(res.status).toBe(200);+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.START_WORKLOAD,+ "wk-1",+ "{}",+ undefined+ );+ });++ it("allows start without snapshot for controlled k8s workloads", async () => {+ mockCapableInstance("kubernetes", "lifecycle_kubernetes_v1", {+ resource_attributes: { "k8s.workload.kind": "Deployment" },+ });+ mockGetDesired.mockResolvedValue({ data: [{ feature: "other", config: "{}" }] });++ const res = await lifecycleHandler.applyOperation("svc-1", "start", {});+ expect(res.status).toBe(200);+ });++ it("returns 500 on unexpected errors", async () => {+ mockGetServiceById.mockRejectedValue(new Error("boom"));+ const res = await lifecycleHandler.applyOperation("svc-1", "stop", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({ error: "boom" });+ });++ it("uses fallback error message for non-Error throws", async () => {+ mockGetServiceById.mockRejectedValue({});+ const res = await lifecycleHandler.applyOperation("svc-1", "stop", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({+ error: "Lifecycle operation failed",+ });+ });++ describe("reconcile", () => {+ it("emits START when desired running and actual stopped", () => {+ const actions = lifecycleHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "discovered",+ resource_attributes: {+ "openlit.lifecycle.status": "stopped",+ },+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "lifecycle",+ desired_status: "running",+ config: '{"snap":1}',+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions).toEqual([+ {+ actionType: KNOWN_ACTIONS.START_WORKLOAD,+ serviceKey: "wk-1",+ payload: '{"snap":1}',+ },+ ]);+ });++ it("emits STOP when desired stopped and actual is not", () => {+ const actions = lifecycleHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "discovered",+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "lifecycle",+ desired_status: "stopped",+ config: "{}",+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions[0].actionType).toBe(KNOWN_ACTIONS.STOP_WORKLOAD);+ });++ it("skips services without desired state", () => {+ expect(+ lifecycleHandler.reconcile(+ [{ workload_key: "wk-x", instrumentation_status: "discovered" }],+ new Map()+ )+ ).toEqual([]);+ });++ it("defaults missing desired config to {}", () => {+ const actions = lifecycleHandler.reconcile(+ [+ {+ workload_key: "wk-1",+ instrumentation_status: "discovered",+ resource_attributes: {+ "openlit.lifecycle.status": "stopped",+ },+ },+ ],+ new Map([+ [+ "wk-1",+ {+ workload_key: "wk-1",+ cluster_id: "default",+ feature: "lifecycle",+ desired_status: "running",+ config: undefined as any,+ updated_at: "t",+ },+ ],+ ])+ );++ expect(actions[0].payload).toBe("{}");+ });+ });+});diff --git a/src/client/src/__tests__/lib/platform/controller/prompts-feature.test.ts b/src/client/src/__tests__/lib/platform/controller/prompts-feature.test.tsnew file mode 100644index 00000000..e91bde22--- /dev/null+++ b/src/client/src/__tests__/lib/platform/controller/prompts-feature.test.ts@@ -0,0 +1,137 @@++jest.mock("@/lib/platform/controller", () => ({+ getServiceById: jest.fn(),+ getControllerInstanceById: jest.fn(),+ getControllerIdsForWorkload: jest.fn(),+ getControllerConfig: jest.fn(),+ getFeatureDesiredStates: jest.fn(),+ queueAction: jest.fn(),+ updateFeatureDesiredState: jest.fn(),+}));++import {+ getServiceById,+ getControllerIdsForWorkload,+ queueAction,+ updateFeatureDesiredState,+} from "@/lib/platform/controller";+import promptsHandler from "@/lib/platform/controller/features/prompts";+import { KNOWN_ACTIONS } from "@/types/controller";++const mockGetServiceById = getServiceById as jest.Mock;+const mockGetControllerIds = getControllerIdsForWorkload as jest.Mock;+const mockQueueAction = queueAction as jest.Mock;+const mockUpdateDesired = updateFeatureDesiredState as jest.Mock;++const baseService = {+ id: "svc-1",+ workload_key: "linux:prompt-app",+ cluster_id: "default",+ service_name: "prompt-app",+ namespace: "",+ controller_instance_id: "ctrl-1",+};++beforeEach(() => {+ jest.clearAllMocks();+ mockGetControllerIds.mockResolvedValue({ data: [] });+ mockQueueAction.mockResolvedValue({ data: "ok" });+ mockUpdateDesired.mockResolvedValue({ data: "ok" });+});++describe("prompts feature", () => {+ it("validates push/remove only", () => {+ expect(promptsHandler.validatePayload("push", {})).toBeNull();+ expect(promptsHandler.validatePayload("remove", {})).toBeNull();+ expect(promptsHandler.validatePayload("status", {})).toMatch(+ /Unknown operation/+ );+ });++ it("returns empty reconcile actions (stub)", () => {+ expect(promptsHandler.reconcile([], new Map())).toEqual([]);+ });++ it("returns 404 when service is missing", async () => {+ mockGetServiceById.mockResolvedValue({ data: [] });+ const res = await promptsHandler.applyOperation("svc-1", "push", {+ id: "p1",+ });+ expect(res.status).toBe(404);+ });++ it("returns 500 when workload_key is missing", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ ...baseService, workload_key: null }],+ });+ const res = await promptsHandler.applyOperation("svc-1", "push", {});+ expect(res.status).toBe(500);+ });++ it("queues PUSH_PROMPTS", async () => {+ mockGetServiceById.mockResolvedValue({ data: [baseService] });+ mockGetControllerIds.mockResolvedValue({+ data: [+ { controller_instance_id: "ctrl-a" },+ { controller_instance_id: "ctrl-b" },+ ],+ });++ const res = await promptsHandler.applyOperation(+ "svc-1",+ "push",+ { prompt: "hi" },+ "db-1"+ );+ const body = await res.json();++ expect(body).toEqual({+ status: "queued",+ action: KNOWN_ACTIONS.PUSH_PROMPTS,+ controllers: 2,+ });+ expect(mockQueueAction).toHaveBeenCalledTimes(2);+ expect(mockUpdateDesired).toHaveBeenCalledWith(+ baseService.workload_key,+ "default",+ "prompts",+ "active",+ JSON.stringify({ prompt: "hi" }),+ "db-1"+ );+ });++ it("queues REMOVE_PROMPTS and falls back to service controller", async () => {+ mockGetServiceById.mockResolvedValue({+ data: [{ ...baseService, cluster_id: undefined }],+ });++ const res = await promptsHandler.applyOperation("svc-1", "remove", {});+ const body = await res.json();++ expect(body.action).toBe(KNOWN_ACTIONS.REMOVE_PROMPTS);+ expect(mockQueueAction).toHaveBeenCalledWith(+ "ctrl-1",+ KNOWN_ACTIONS.REMOVE_PROMPTS,+ baseService.workload_key,+ "{}",+ undefined+ );+ });++ it("returns 500 on unexpected errors", async () => {+ mockGetServiceById.mockRejectedValue(new Error("boom"));+ const res = await promptsHandler.applyOperation("svc-1", "push", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({ error: "boom" });+ });++ it("uses fallback error message for non-Error throws", async () => {+ mockGetServiceById.mockRejectedValue("x");+ const res = await promptsHandler.applyOperation("svc-1", "push", {});+ expect(res.status).toBe(500);+ await expect(res.json()).resolves.toEqual({+ error: "Prompts operation failed",+ });+ });+});diff --git a/src/client/src/__tests__/lib/platform/controller/validate-config.test.ts b/src/client/src/__tests__/lib/platform/controller/validate-config.test.tsindex 78c7885e..9f288ef3 100644--- a/src/client/src/__tests__/lib/platform/controller/validate-config.test.ts+++ b/src/client/src/__tests__/lib/platform/controller/validate-config.test.ts@@ -1,4 +1,7 @@-import { validateControllerConfig } from "@/lib/platform/controller/validate-config";+import {+ ConfigValidationError,+ validateControllerConfig,+} from "@/lib/platform/controller/validate-config";describe("validateControllerConfig", () => {it("preserves the retrieval payload-extraction toggle", () => {@@ -17,4 +20,157 @@ describe("validateControllerConfig", () => {expect(config.payload_extraction.ollama).toBe(false);expect(config.payload_extraction.bedrock).toBe(false);});++ it("rejects non-object configs", () => {+ expect(() => validateControllerConfig(null)).toThrow(ConfigValidationError);+ expect(() => validateControllerConfig("bad")).toThrow(/must be an object/);+ });++ it("sanitizes export headers, optional endpoints, and control chars", () => {+ const config = validateControllerConfig({+ export: {+ otlp_endpoint: "http://collector\x00",+ otlp_protocol: "http/protobuf",+ otlp_headers: {+ Authorization: "Bearer x\x1f",+ "x-extra": "1",+ },+ otlp_traces_endpoint: null,+ otlp_metrics_endpoint: "http://metrics",+ otlp_logs_endpoint: undefined,+ },+ discovery: {},+ payload_extraction: {},+ });++ expect(config.export.otlp_endpoint).toBe("http://collector");+ expect(config.export.otlp_headers.Authorization).toBe("Bearer x");+ expect(config.export.otlp_traces_endpoint).toBeUndefined();+ expect(config.export.otlp_metrics_endpoint).toBe("http://metrics");+ expect(config.export.otlp_logs_endpoint).toBeUndefined();+ });++ it("rejects oversized strings and non-string header values", () => {+ expect(() =>+ validateControllerConfig({+ export: { otlp_endpoint: "x".repeat(2049) },+ discovery: {},+ payload_extraction: {},+ })+ ).toThrow(/exceeds 2048/);++ expect(() =>+ validateControllerConfig({+ export: { otlp_headers: { ok: 1 as any } },+ discovery: {},+ payload_extraction: {},+ })+ ).toThrow(/must be a string/);+ });++ it("handles null discovery targets and non-array k8s selectors", () => {+ const config = validateControllerConfig({+ export: {},+ discovery: {+ auto_discover: true,+ instrument: [null, undefined, { service_name: "svc", open_ports: "80" }],+ exclude: "not-an-array",+ kubernetes: {+ // enabled omitted → ""+ instrument: "not-array",+ exclude: [null, { namespace: "ns", deployment_name: "dep" }],+ },+ },+ payload_extraction: {},+ custom_llm_hosts: [" host.a ", "", "host.b"],+ environment: "prod",+ poll_interval_seconds: 30.9,+ });++ expect(config.discovery.auto_discover).toBe(true);+ expect(config.discovery.instrument[0]).toEqual({+ service_name: undefined,+ open_ports: undefined,+ exe_path: undefined,+ cmd_args: undefined,+ });+ expect(config.discovery.instrument[2].service_name).toBe("svc");+ expect(config.discovery.exclude).toEqual([]);+ expect(config.discovery.kubernetes).toEqual({+ enabled: "",+ instrument: [],+ exclude: [+ {+ namespace: undefined,+ deployment_name: undefined,+ },+ { namespace: "ns", deployment_name: "dep" },+ ],+ });+ expect(config.custom_llm_hosts).toEqual(["host.a", "host.b"]);+ expect(config.environment).toBe("prod");+ expect(config.poll_interval_seconds).toBe(30);+ });++ it("rejects invalid poll intervals and sdk_version grammar", () => {+ expect(() =>+ validateControllerConfig({+ export: {},+ discovery: {},+ payload_extraction: {},+ poll_interval_seconds: 4,+ })+ ).toThrow(/poll_interval_seconds/);++ expect(() =>+ validateControllerConfig({+ export: {},+ discovery: {},+ payload_extraction: {},+ poll_interval_seconds: Number.NaN,+ })+ ).toThrow(/poll_interval_seconds/);++ expect(() =>+ validateControllerConfig({+ export: {},+ discovery: {},+ payload_extraction: {},+ sdk_version: "1.2.3; rm -rf /",+ })+ ).toThrow(/sdk_version/);++ const ok = validateControllerConfig({+ export: {},+ discovery: {},+ payload_extraction: {},+ sdk_version: "",+ poll_interval_seconds: null,+ });+ expect(ok.poll_interval_seconds).toBeUndefined();+ });++ it("defaults discovery/export when missing and treats false-y bools as false", () => {+ const config = validateControllerConfig({+ payload_extraction: {+ openai: false,+ anthropic: "yes",+ gemini: 1,+ qwen: true,+ },+ });++ expect(config.export.otlp_endpoint).toBe("");+ expect(config.discovery.auto_discover).toBe(false);+ expect(config.payload_extraction).toEqual({+ openai: false,+ anthropic: false,+ gemini: false,+ qwen: true,+ bedrock: false,+ retrieval: false,+ custom: false,+ ollama: false,+ });+ });});diff --git a/src/client/src/__tests__/lib/platform/cost/summary.test.ts b/src/client/src/__tests__/lib/platform/cost/summary.test.tsnew file mode 100644index 00000000..b7a25a77--- /dev/null+++ b/src/client/src/__tests__/lib/platform/cost/summary.test.ts@@ -0,0 +1,186 @@+jest.mock("@/lib/platform/llm/cost", () => ({+ getTotalCost: jest.fn(),+}));++jest.mock("@/lib/platform/evaluation/analytics", () => ({+ getEvaluationAnalytics: jest.fn(),+}));++jest.mock("@/lib/platform/openground/cost-analytics", () => ({+ getOpengroundTotalCost: jest.fn(),+}));++jest.mock("@/lib/platform/coding-agents/cost", () => ({+ getCodingAgentsTotalCost: jest.fn(),+}));++jest.mock("@/lib/platform/chat/cost", () => ({+ getOtterTotalCost: jest.fn(),+}));++import { getTotalCost } from "@/lib/platform/llm/cost";+import { getEvaluationAnalytics } from "@/lib/platform/evaluation/analytics";+import { getOpengroundTotalCost } from "@/lib/platform/openground/cost-analytics";+import { getCodingAgentsTotalCost } from "@/lib/platform/coding-agents/cost";+import { getOtterTotalCost } from "@/lib/platform/chat/cost";+import { getCostSummary } from "@/lib/platform/cost/summary";++const mockedGetTotalCost = getTotalCost as jest.MockedFunction<typeof getTotalCost>;+const mockedGetEvaluationAnalytics =+ getEvaluationAnalytics as jest.MockedFunction<typeof getEvaluationAnalytics>;+const mockedGetOpengroundTotalCost =+ getOpengroundTotalCost as jest.MockedFunction<typeof getOpengroundTotalCost>;+const mockedGetCodingAgentsTotalCost =+ getCodingAgentsTotalCost as jest.MockedFunction<+ typeof getCodingAgentsTotalCost+ >;+const mockedGetOtterTotalCost = getOtterTotalCost as jest.MockedFunction<+ typeof getOtterTotalCost+>;++const params = {+ timeLimit: {+ start: new Date("2024-01-01"),+ end: new Date("2024-01-02"),+ type: "custom",+ },+};++describe("getCostSummary", () => {+ beforeEach(() => {+ mockedGetTotalCost.mockReset();+ mockedGetEvaluationAnalytics.mockReset();+ mockedGetOpengroundTotalCost.mockReset();+ mockedGetCodingAgentsTotalCost.mockReset();+ mockedGetOtterTotalCost.mockReset();+ });++ it("sums llm, coding agents, otter, evaluations, and openground costs", async () => {+ mockedGetTotalCost.mockResolvedValue({+ data: [{ total_usage_cost: 1.5, previous_total_usage_cost: 1 }],+ });+ mockedGetCodingAgentsTotalCost.mockResolvedValue({+ data: [{ total_cost: 0.4, previous_total_cost: 0.2 }],+ });+ mockedGetOtterTotalCost.mockResolvedValue({+ data: [{ total_cost: 0.1, previous_total_cost: 0.05 }],+ });+ mockedGetEvaluationAnalytics.mockResolvedValue({+ configured: true,+ data: [{ total_cost: 0.25, previous_total_cost: 0.1 }],+ } as Awaited<ReturnType<typeof getEvaluationAnalytics>>);+ mockedGetOpengroundTotalCost.mockResolvedValue({+ data: [{ total_cost: 0.75, previous_total_cost: 0.5 }],+ });++ const res = await getCostSummary(params);++ expect(res.data[0]).toEqual({+ total_platform_cost: 3.0,+ previous_total_platform_cost: 1.85,+ llm_cost: 1.5,+ previous_llm_cost: 1,+ coding_agents_cost: 0.4,+ previous_coding_agents_cost: 0.2,+ otter_cost: 0.1,+ previous_otter_cost: 0.05,+ evaluations_cost: 0.25,+ previous_evaluations_cost: 0.1,+ openground_cost: 0.75,+ previous_openground_cost: 0.5,+ });+ });++ it("zeros every bucket when data rows are empty", async () => {+ mockedGetTotalCost.mockResolvedValue({ data: [] });+ mockedGetCodingAgentsTotalCost.mockResolvedValue({ data: [] });+ mockedGetOtterTotalCost.mockResolvedValue({ data: [] });+ mockedGetEvaluationAnalytics.mockResolvedValue({+ configured: true,+ data: [],+ } as Awaited<ReturnType<typeof getEvaluationAnalytics>>);+ mockedGetOpengroundTotalCost.mockResolvedValue({ data: [] });++ const res = await getCostSummary(params);++ expect(res.data[0]).toEqual({+ total_platform_cost: 0,+ previous_total_platform_cost: 0,+ llm_cost: 0,+ previous_llm_cost: 0,+ coding_agents_cost: 0,+ previous_coding_agents_cost: 0,+ otter_cost: 0,+ previous_otter_cost: 0,+ evaluations_cost: 0,+ previous_evaluations_cost: 0,+ openground_cost: 0,+ previous_openground_cost: 0,+ });+ });++ it("treats missing and NaN cost fields as zero", async () => {+ mockedGetTotalCost.mockResolvedValue({+ data: [{ total_usage_cost: "bad", previous_total_usage_cost: undefined }],+ });+ mockedGetCodingAgentsTotalCost.mockResolvedValue({+ data: [{ total_cost: null, previous_total_cost: Number.NaN }],+ });+ mockedGetOtterTotalCost.mockResolvedValue({+ data: [{} as { total_cost: number; previous_total_cost: number }],+ });+ mockedGetEvaluationAnalytics.mockResolvedValue({+ configured: true,+ data: [{ total_cost: "" as unknown as number, previous_total_cost: "x" as unknown as number }],+ } as unknown as Awaited<ReturnType<typeof getEvaluationAnalytics>>);+ mockedGetOpengroundTotalCost.mockResolvedValue({+ data: [{ total_cost: 0, previous_total_cost: false as unknown as number }],+ });++ const res = await getCostSummary(params);++ expect(res.data[0].total_platform_cost).toBe(0);+ expect(res.data[0].previous_total_platform_cost).toBe(0);+ expect(res.data[0].llm_cost).toBe(0);+ expect(res.data[0].coding_agents_cost).toBe(0);+ expect(res.data[0].otter_cost).toBe(0);+ expect(res.data[0].evaluations_cost).toBe(0);+ expect(res.data[0].openground_cost).toBe(0);+ });++ it("falls back when firstRow data is non-array or non-object", async () => {+ mockedGetTotalCost.mockResolvedValue({+ data: null as unknown as never[],+ });+ mockedGetCodingAgentsTotalCost.mockResolvedValue({+ data: [null] as unknown as never[],+ });+ mockedGetOtterTotalCost.mockResolvedValue({+ data: ["not-an-object"] as unknown as never[],+ });+ mockedGetEvaluationAnalytics.mockResolvedValue({+ configured: false,+ data: undefined as unknown as never[],+ } as Awaited<ReturnType<typeof getEvaluationAnalytics>>);+ mockedGetOpengroundTotalCost.mockResolvedValue({+ data: { total_cost: 99 } as unknown as never[],+ });++ const res = await getCostSummary(params);++ expect(res.data[0]).toEqual({+ total_platform_cost: 0,+ previous_total_platform_cost: 0,+ llm_cost: 0,+ previous_llm_cost: 0,+ coding_agents_cost: 0,+ previous_coding_agents_cost: 0,+ otter_cost: 0,+ previous_otter_cost: 0,+ evaluations_cost: 0,+ previous_evaluations_cost: 0,+ openground_cost: 0,+ previous_openground_cost: 0,+ });+ });+});diff --git a/src/client/src/__tests__/lib/platform/evaluation/analytics.test.ts b/src/client/src/__tests__/lib/platform/evaluation/analytics.test.tsnew file mode 100644index 00000000..26d1be08--- /dev/null+++ b/src/client/src/__tests__/lib/platform/evaluation/analytics.test.ts@@ -0,0 +1,469 @@+jest.mock("@/lib/platform/evaluation/config", () => ({+ getEvaluationConfig: jest.fn(),+}));++jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterPreviousParams: jest.fn((params) => ({+ ...params,+ timeLimit: {+ ...params.timeLimit,+ start: "2024-01-01T00:00:00.000Z",+ end: "2024-01-01T12:00:00.000Z",+ },+ })),+ dateTruncGroupingLogic: jest.fn(() => "hour"),+}));++jest.mock("@/utils/asaw", () => jest.fn());++import asaw from "@/utils/asaw";+import { dataCollector } from "@/lib/platform/common";+import { getEvaluationConfig } from "@/lib/platform/evaluation/config";+import {+ getEvaluationAnalytics,+ getEvaluationAnalyticsSummary,+ getEvaluationAnalyticsByType,+ getEvaluationAnalyticsTimeseries,+ getEvaluationEvaluatorAnalytics,+} from "@/lib/platform/evaluation/analytics";++const params = {+ timeLimit: {+ start: "2024-01-01T12:00:00.000Z",+ end: "2024-01-02T12:00:00.000Z",+ type: "24H",+ },+};++describe("evaluation analytics", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("returns configured:false when evaluation config is missing", async () => {+ (asaw as jest.Mock).mockResolvedValue(["not found", null]);++ const result = await getEvaluationAnalytics(params as any);+ expect(result).toEqual({ configured: false });+ expect(dataCollector).not.toHaveBeenCalled();+ });++ it("aggregates summary metrics from clickhouse rows", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ {+ id: "cfg-1",+ evaluationTypes: [+ { id: "hallucination", enabled: true, label: "Hallucination" },+ { id: "bias", enabled: false, label: "Bias" },+ ],+ },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [+ {+ tracesEvaluated: 5,+ executions: 8,+ totalCost: 0.12,+ avgPassRate: 75,+ failedScores: 3,+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ tracesEvaluated: 2,+ executions: 4,+ totalCost: 0.05,+ avgPassRate: 50,+ failedScores: 1,+ },+ ],+ });++ const result = await getEvaluationAnalyticsSummary(params as any);+ expect(result.err).toBeUndefined();+ expect(result.data).toMatchObject({+ tracesEvaluated: 5,+ executions: 8,+ totalCost: 0.12,+ avgPassRate: 75,+ failedScores: 3,+ previous_tracesEvaluated: 2,+ evaluations: 2,+ active: 1,+ });++ const firstQuery = (dataCollector as jest.Mock).mock.calls[0][0].query as string;+ expect(firstQuery).toContain("evaluationData.evaluation IN");+ expect(firstQuery).toContain("'hallucination'");+ expect(firstQuery).toContain("'Hallucination'");+ });++ it("maps by-type rows with previous pass rates", async () => {+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [+ {+ evaluation: "hallucination",+ executions: 10,+ passRate: 80,+ failedScores: 2,+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ evaluation: "hallucination",+ executions: 5,+ passRate: 70,+ failedScores: 1,+ },+ ],+ });++ const result = await getEvaluationAnalyticsByType(params as any);+ expect(result.data?.[0]).toMatchObject({+ evaluation: "hallucination",+ executions: 10,+ passRate: 80,+ previousPassRate: 70,+ });+ });++ it("returns full analytics when configured", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [{ id: "hallucination", enabled: true }] },+ ]);+ (getEvaluationConfig as jest.Mock).mockResolvedValue({+ id: "cfg-1",+ evaluationTypes: [{ id: "hallucination", enabled: true }],+ });+ (dataCollector as jest.Mock).mockResolvedValue({ data: [] });++ const result = await getEvaluationAnalytics(params as any);+ expect(result.configured).toBe(true);+ expect(result.data?.[0]).toMatchObject({+ traces_evaluated: expect.any(Number),+ executions: expect.any(Number),+ });+ expect(result.timeseries).toEqual([]);+ expect(result.byType).toEqual([]);+ });++ it("maps timeseries rows and surfaces collector errors", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ {+ id: "cfg-1",+ evaluationTypes: [{ id: "hallucination", enabled: true, label: "Hallucination" }],+ },+ ]);+ (dataCollector as jest.Mock).mockResolvedValueOnce({+ data: [{ timestamp: "2024/01/01 12:00", executions: "4", passRate: "90" }],+ });++ const ok = await getEvaluationAnalyticsTimeseries(params as any);+ expect(ok.err).toBeUndefined();+ expect(ok.data).toEqual([+ { timestamp: "2024/01/01 12:00", executions: 4, passRate: 90 },+ ]);+ expect((dataCollector as jest.Mock).mock.calls[0][0].query).toContain(+ "DATE_TRUNC('hour'"+ );++ (dataCollector as jest.Mock).mockResolvedValueOnce({+ err: new Error("timeseries failed"),+ data: [],+ });+ const failed = await getEvaluationAnalyticsTimeseries(params as any);+ expect(failed.err).toEqual(expect.any(Error));+ });++ it("returns evaluator analytics for a configured type", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ {+ id: "cfg-1",+ evaluationTypes: [+ {+ id: "hallucination",+ label: "Hallucination",+ enabled: true,+ description: "Detects hallucinations",+ },+ ],+ },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [+ {+ executions: 8,+ totalCost: 0.2,+ avgPassRate: 75,+ failedScores: 2,+ tracesEvaluated: 5,+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ executions: 3,+ totalCost: 0.1,+ avgPassRate: 50,+ failedScores: 1,+ tracesEvaluated: 2,+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [{ timestamp: "2024/01/01 12:00", executions: 2, passRate: 80 }],+ })+ .mockResolvedValueOnce({+ data: [+ {+ id: "run-1",+ spanId: "span-1",+ createdAt: "2024-01-01T12:00:00.000Z",+ verdict: "no",+ score: 0.9,+ classification: "pass",+ explanation: "ok",+ source: "auto",+ cost: 0.01,+ },+ ],+ });++ const result = await getEvaluationEvaluatorAnalytics(+ params as any,+ "hallucination"+ );+ expect(result).toMatchObject({+ configured: true,+ found: true,+ evaluator: {+ id: "hallucination",+ label: "Hallucination",+ enabled: true,+ },+ data: [+ {+ executions: 8,+ previous_executions: 3,+ avg_pass_rate: 75,+ previous_avg_pass_rate: 50,+ failed_scores: 2,+ previous_failed_scores: 1,+ total_cost: 0.2,+ previous_total_cost: 0.1,+ traces_evaluated: 5,+ previous_traces_evaluated: 2,+ },+ ],+ timeseries: [{ timestamp: "2024/01/01 12:00", executions: 2, passRate: 80 }],+ recentResults: [+ expect.objectContaining({+ id: "run-1",+ spanId: "span-1",+ verdict: "no",+ score: 0.9,+ source: "auto",+ }),+ ],+ });+ });++ it("handles missing evaluator config and collector errors", async () => {+ (asaw as jest.Mock).mockResolvedValueOnce(["missing", null]);+ await expect(+ getEvaluationEvaluatorAnalytics(params as any, "hallucination")+ ).resolves.toEqual({ configured: false, found: false });++ (asaw as jest.Mock).mockResolvedValueOnce([+ null,+ { id: "cfg-1", evaluationTypes: [{ id: "hallucination", enabled: true }] },+ ]);+ await expect(+ getEvaluationEvaluatorAnalytics(params as any, "not-a-real-evaluator")+ ).resolves.toEqual({ configured: true, found: false });++ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [{ id: "hallucination", enabled: true }] },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ err: new Error("summary failed"), data: [] })+ .mockResolvedValueOnce({ data: [] })+ .mockResolvedValueOnce({ data: [] })+ .mockResolvedValueOnce({ data: [] });++ const errored = await getEvaluationEvaluatorAnalytics(+ params as any,+ "hallucination"+ );+ expect(errored).toMatchObject({+ configured: true,+ found: true,+ data: [],+ timeseries: [],+ recentResults: [],+ });+ });++ it("falls back to built-in evaluation types when config types are empty", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [] },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{}] })+ .mockResolvedValueOnce({ data: [{}] });++ const result = await getEvaluationAnalyticsSummary(params as any);+ expect(result.err).toBeUndefined();+ expect(result.data?.evaluations).toBeGreaterThan(0);+ expect(result.data?.active).toBe(0);+ const query = (dataCollector as jest.Mock).mock.calls[0][0].query as string;+ expect(query).toContain("evaluationData.evaluation IN");+ });++ it("uses 1 = 0 when all configured evaluator ids are empty", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [{ id: "", label: "Broken" }] },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{}] })+ .mockResolvedValueOnce({ data: [{}] });++ await getEvaluationAnalyticsSummary(params as any);+ const query = (dataCollector as jest.Mock).mock.calls[0][0].query as string;+ expect(query).toContain("1 = 0");+ });++ it("surfaces summary and by-type collector errors", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [{ id: "hallucination", enabled: true }] },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ err: new Error("current failed"), data: [] })+ .mockResolvedValueOnce({ data: [] });+ await expect(getEvaluationAnalyticsSummary(params as any)).resolves.toEqual({+ err: expect.any(Error),+ });++ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [] })+ .mockResolvedValueOnce({ err: new Error("previous failed"), data: [] });+ await expect(getEvaluationAnalyticsSummary(params as any)).resolves.toEqual({+ err: expect.any(Error),+ });++ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ err: new Error("by-type failed"), data: [] })+ .mockResolvedValueOnce({ data: [] });+ await expect(getEvaluationAnalyticsByType(params as any)).resolves.toEqual({+ err: expect.any(Error),+ });++ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{ evaluation: "x", executions: 1, passRate: 50 }] })+ .mockResolvedValueOnce({ err: new Error("prev by-type"), data: [] });+ await expect(getEvaluationAnalyticsByType(params as any)).resolves.toEqual({+ err: expect.any(Error),+ });+ });++ it("fills analytics defaults when summary data is missing", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [{ id: "hallucination", enabled: true }] },+ ]);+ (dataCollector as jest.Mock).mockResolvedValue({ data: [] });++ const result = await getEvaluationAnalytics(params as any);+ expect(result.configured).toBe(true);+ expect(result.data?.[0]).toMatchObject({+ evaluations: expect.any(Number),+ active: expect.any(Number),+ traces_evaluated: expect.any(Number),+ executions: expect.any(Number),+ });+ expect(Array.isArray(result.timeseries)).toBe(true);+ expect(Array.isArray(result.byType)).toBe(true);+ });++ it("resolves built-in-only evaluators and empty recent source", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ { id: "cfg-1", evaluationTypes: [] },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [+ {+ executions: 1,+ totalCost: 0,+ avgPassRate: 100,+ failedScores: 0,+ tracesEvaluated: 1,+ },+ ],+ })+ .mockResolvedValueOnce({ data: [{}] })+ .mockResolvedValueOnce({ data: [] })+ .mockResolvedValueOnce({+ data: [+ {+ id: "run-2",+ spanId: "span-2",+ createdAt: "2024-01-01T12:00:00.000Z",+ verdict: "yes",+ score: 0.1,+ classification: "fail",+ explanation: "bad",+ source: "",+ cost: 0,+ },+ ],+ });++ const result = await getEvaluationEvaluatorAnalytics(+ params as any,+ "hallucination"+ );+ expect(result.found).toBe(true);+ expect(result.evaluator?.id).toBe("hallucination");+ expect(result.evaluator?.label).toBeTruthy();+ expect(result.recentResults?.[0].source).toBeUndefined();+ });++ it("escapes quotes in evaluator name variants", async () => {+ (asaw as jest.Mock).mockResolvedValue([+ null,+ {+ id: "cfg-1",+ evaluationTypes: [{ id: "custom", label: "O'Brien Judge", enabled: true }],+ },+ ]);+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{}] })+ .mockResolvedValueOnce({ data: [{}] });++ await getEvaluationAnalyticsSummary(params as any);+ const query = (dataCollector as jest.Mock).mock.calls[0][0].query as string;+ expect(query).toContain("O\\'Brien Judge");+ });+});diff --git a/src/client/src/__tests__/lib/platform/evaluation/run-evaluation.test.ts b/src/client/src/__tests__/lib/platform/evaluation/run-evaluation.test.tsindex 11c4ecac..d7f8c0b5 100644--- a/src/client/src/__tests__/lib/platform/evaluation/run-evaluation.test.ts+++ b/src/client/src/__tests__/lib/platform/evaluation/run-evaluation.test.ts@@ -224,16 +224,15 @@ describe('runEvaluation — generateText and response parsing', () => {expect(callArgs.temperature).toBe(0);});- it('system prompt uses generic TypeA/TypeB examples, not hardcoded eval type names', async () => {+ it('system prompt uses placeholder evaluation names, not TypeA/TypeB or hardcoded types', async () => {await runEvaluation(BASE_PARAMS);const [{ prompt }] = (generateText as jest.Mock).mock.calls[0];- // The JSON example in the prompt should use generic placeholders- expect(prompt).toContain('TypeA');- expect(prompt).toContain('TypeB');+ // Placeholders must not look like real evaluator ids — LLMs copy examples into CH.+ expect(prompt).toContain('<TypeNameFromContextHeader>');+ expect(prompt).not.toContain('"TypeA"');+ expect(prompt).not.toContain('"TypeB"');- // The prompt template itself should NOT hardcode specific evaluation type names- // (the actual type names come from contexts, not from the template)const templateWithoutContexts = prompt.split('Contexts:')[0];expect(templateWithoutContexts).not.toContain('"Hallucination"');expect(templateWithoutContexts).not.toContain('"Bias"');diff --git a/src/client/src/__tests__/lib/platform/llm/cost.test.ts b/src/client/src/__tests__/lib/platform/llm/cost.test.tsindex 15e9a63b..ea356c26 100644--- a/src/client/src/__tests__/lib/platform/llm/cost.test.ts+++ b/src/client/src/__tests__/lib/platform/llm/cost.test.ts@@ -1,63 +1,87 @@-jest.mock('@/lib/platform/common', () => ({- dataCollector: jest.fn().mockResolvedValue({ data: [] }),- OTEL_TRACES_TABLE_NAME: 'otel_traces',+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+ OTEL_TRACES_TABLE_NAME: "otel_traces",}));-import { getTotalCost, getAverageCost, getCostByApplication, getCostByEnvironment } from '@/lib/platform/llm/cost';-import { dataCollector } from '@/lib/platform/common';+jest.mock("@/helpers/server/platform", () => ({+ dateTruncGroupingLogic: jest.fn(() => "hour"),+ getFilterPreviousParams: jest.fn((params) => params),+ getFilterWhereCondition: jest.fn(() => "1 = 1"),+}));++jest.mock("@/helpers/server/trace", () => ({+ getTraceMappingKeyFullPath: jest.fn((key: string) => {+ if (key === "cost") return "gen_ai.usage.cost";+ if (key === "model") return "gen_ai.request.model";+ if (key === "applicationName") return "service.name";+ return key;+ }),+ getTraceMappingKeyFullPaths: jest.fn(() => ["gen_ai.provider.name", "gen_ai.system"]),+}));++import { dataCollector } from "@/lib/platform/common";+import {+ getCostByApplication,+ getCostByModel,+ getCostByProvider,+ getCostPerTime,+} from "@/lib/platform/llm/cost";++const mockedDataCollector = dataCollector as jest.MockedFunction<typeof dataCollector>;-const baseParams = {- timeLimit: { start: '2024-01-01', end: '2024-01-31', type: '1M' },+const params = {+ timeLimit: {+ start: new Date("2024-01-01"),+ end: new Date("2024-01-02"),+ type: "custom",+ },};-beforeEach(() => {- jest.clearAllMocks();-});+describe("llm cost analytics queries", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ mockedDataCollector.mockResolvedValue({ data: [] });+ });-describe('getTotalCost', () => {- it('calls dataCollector and returns result', async () => {- const result = await getTotalCost(baseParams);- expect(dataCollector).toHaveBeenCalledTimes(1);- const { query } = (dataCollector as jest.Mock).mock.calls[0][0];- expect(query).toContain('total_usage_cost');- expect(query).toContain('otel_traces');- expect(result).toEqual({ data: [] });- });-- it('query includes JOIN for current/previous comparison', async () => {- await getTotalCost(baseParams);- const { query } = (dataCollector as jest.Mock).mock.calls[0][0];- expect(query).toContain('JOIN');- expect(query).toContain('previous_total_usage_cost');- });-});+ it("queries cost by application via ServiceName fallbacks", async () => {+ await getCostByApplication(params);+ const query = mockedDataCollector.mock.calls[0][0].query as string;+ expect(query).toContain("AS applicationName");+ expect(query).toContain("ServiceName");+ expect(query).toContain("ResourceAttributes['service.name']");+ expect(query).toContain("SpanAttributes['gen_ai.application_name']");+ expect(query).not.toContain(+ "ResourceAttributes['SpanAttributes.gen_ai.application_name']"+ );+ });-describe('getAverageCost', () => {- it('calls dataCollector with average query', async () => {- await getAverageCost(baseParams);- expect(dataCollector).toHaveBeenCalledTimes(1);- const { query } = (dataCollector as jest.Mock).mock.calls[0][0];- expect(query).toContain('average_usage_cost');- expect(query).toContain('JOIN');- });-});+ it("queries cost by provider", async () => {+ await getCostByProvider(params);+ expect(mockedDataCollector).toHaveBeenCalledWith(+ expect.objectContaining({+ query: expect.stringContaining("AS provider"),+ })+ );+ expect(mockedDataCollector.mock.calls[0][0].query).toContain(+ "gen_ai.usage.cost"+ );+ });-describe('getCostByApplication', () => {- it('calls dataCollector with GROUP BY applicationName', async () => {- await getCostByApplication(baseParams);- expect(dataCollector).toHaveBeenCalledTimes(1);- const { query } = (dataCollector as jest.Mock).mock.calls[0][0];- expect(query).toContain('applicationName');- expect(query).toContain('GROUP BY');- });-});+ it("queries cost by model", async () => {+ await getCostByModel(params);+ expect(mockedDataCollector).toHaveBeenCalledWith(+ expect.objectContaining({+ query: expect.stringContaining("AS model"),+ })+ );+ });-describe('getCostByEnvironment', () => {- it('calls dataCollector with GROUP BY environment', async () => {- await getCostByEnvironment(baseParams);- expect(dataCollector).toHaveBeenCalledTimes(1);- const { query } = (dataCollector as jest.Mock).mock.calls[0][0];- expect(query).toContain('environment');- expect(query).toContain('GROUP BY');- });+ it("queries cost per time", async () => {+ await getCostPerTime(params);+ expect(mockedDataCollector).toHaveBeenCalledWith(+ expect.objectContaining({+ query: expect.stringContaining("request_time"),+ })+ );+ });});diff --git a/src/client/src/__tests__/lib/platform/llm/endpoint.test.ts b/src/client/src/__tests__/lib/platform/llm/endpoint.test.tsindex b2f3631e..282b5b7a 100644--- a/src/client/src/__tests__/lib/platform/llm/endpoint.test.ts+++ b/src/client/src/__tests__/lib/platform/llm/endpoint.test.ts@@ -27,5 +27,10 @@ describe('getResultGenerationByEndpoint', () => {expect(query).toContain('provider');expect(query).toContain('GROUP BY');expect(query).toContain('otel_traces');+ expect(query).toContain('gen_ai.provider.name');+ // Coding-agent telemetry belongs on the Coding Agents dashboard.+ expect(query).toContain("coding_agent.client");+ expect(query).toContain('openlit-cli');+ expect(query).not.toContain('multiIf');});});diff --git a/src/client/src/__tests__/lib/platform/manage-dashboard/widget.test.ts b/src/client/src/__tests__/lib/platform/manage-dashboard/widget.test.tsindex 7b27edea..71cf3502 100644--- a/src/client/src/__tests__/lib/platform/manage-dashboard/widget.test.ts+++ b/src/client/src/__tests__/lib/platform/manage-dashboard/widget.test.ts@@ -141,6 +141,141 @@ describe("runWidgetQuery", () => {);});+ it("allows attribute keys whose names embed blocklisted keywords", async () => {+ // `gen_ai.system` is a data key, not the SYSTEM command. The+ // `system` substring is word-bounded by `.` and `'`, so a naive+ // keyword scan would wrongly reject this safe SELECT.+ const query =+ "SELECT if(notEmpty(SpanAttributes['gen_ai.provider.name']), SpanAttributes['gen_ai.provider.name'], SpanAttributes['gen_ai.system']) AS provider, COUNT(*) AS count FROM otel_traces GROUP BY provider";+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query }) }],+ err: null,+ })+ .mockResolvedValueOnce({+ data: [{ provider: "openai", count: 5 }],+ err: null,+ });++ const result = await runWidgetQuery("w1", { filter: {} as any });++ expect(result).toEqual({ data: [{ provider: "openai", count: 5 }] });+ expect(dataCollector).toHaveBeenLastCalledWith({+ query,+ enable_readonly: true,+ });+ });++ it("does not flag blocklisted keywords that appear only inside string literals", async () => {+ const query =+ "SELECT ServiceName FROM otel_traces WHERE ServiceName = 'drop-table-service'";+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query }) }],+ err: null,+ })+ .mockResolvedValueOnce({ data: [], err: null });++ const result = await runWidgetQuery("w1", { filter: {} as any });++ expect(result).toEqual({ data: [] });+ });++ it("strips literals with backslash and doubled-quote escapes without ReDoS", async () => {+ // Pathological closed literal that previously triggered CodeQL+ // js/redos on /'(?:\\.|''|[^'])*'/. Must complete in linear time.+ const evilLiteral = "'" + "\\&".repeat(40) + "'";+ const query = `SELECT 1 WHERE x = ${evilLiteral} OR y = 'it''s fine' OR z = 'a\\'b'`;+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query: "SELECT 1" }) }],+ err: null,+ })+ .mockResolvedValueOnce({ data: [{ ok: 1 }], err: null });++ const started = Date.now();+ const result = await runWidgetQuery("w1", {+ userQuery: query,+ filter: {} as any,+ });+ expect(Date.now() - started).toBeLessThan(500);+ expect(result).toEqual({ data: [{ ok: 1 }] });+ });++ it("still sees keywords when a string literal is left unclosed", async () => {+ (dataCollector as jest.Mock).mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query: "SELECT 1" }) }],+ err: null,+ });++ const result = await runWidgetQuery("w1", {+ userQuery: "SELECT 1 WHERE x = 'unclosed DROP TABLE otel_traces",+ filter: {} as any,+ });++ expect(result).toEqual({ err: "Query contains disallowed operations" });+ expect(dataCollector).toHaveBeenCalledTimes(1);+ });++ it("still blocks real SYSTEM commands outside string literals", async () => {+ (dataCollector as jest.Mock).mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query: "SELECT 1" }) }],+ err: null,+ });++ const result = await runWidgetQuery("w1", {+ userQuery: "SELECT 1 FROM otel_traces WHERE x = 'ok'; SYSTEM RELOAD DICTIONARIES",+ filter: {} as any,+ });++ expect(result).toEqual({ err: "Query contains disallowed operations" });+ expect(dataCollector).toHaveBeenCalledTimes(1);+ });++ it("interpolates {{filter.*}} placeholders without a template engine", async () => {+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query: "SELECT 1" }) }],+ err: null,+ })+ .mockResolvedValueOnce({ data: [{ n: 1 }], err: null });++ const result = await runWidgetQuery("w1", {+ userQuery:+ "SELECT 1 WHERE ts >= '{{filter.timeLimit.start}}' AND env = '{{filter.env}}'",+ filter: {+ timeLimit: { start: "2024-01-01", end: "2024-01-02" },+ env: "prod",+ } as any,+ });++ expect(result).toEqual({ data: [{ n: 1 }] });+ expect(dataCollector).toHaveBeenLastCalledWith({+ query: "SELECT 1 WHERE ts >= '2024-01-01' AND env = 'prod'",+ enable_readonly: true,+ });+ });++ it("ignores non-filter mustache-like tags so they cannot run as code", async () => {+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({+ data: [{ id: "w1", config: JSON.stringify({ query: "SELECT 1" }) }],+ err: null,+ })+ .mockResolvedValueOnce({ data: [], err: null });++ const result = await runWidgetQuery("w1", {+ userQuery: "SELECT '{{#evil}}{{/evil}}' AS x, '{{filter.ok}}' AS y",+ filter: { ok: "safe" } as any,+ });++ expect(result).toEqual({ data: [] });+ expect(dataCollector).toHaveBeenLastCalledWith({+ query: "SELECT '{{#evil}}{{/evil}}' AS x, 'safe' AS y",+ enable_readonly: true,+ });+ });+it("mock escapeSingleQuotes escapes backslashes before quotes", () => {expect(escapeSingleQuotes("a\\b'c")).toBe("a\\\\b\\'c");});diff --git a/src/client/src/__tests__/lib/platform/observability.test.ts b/src/client/src/__tests__/lib/platform/observability.test.tsindex fe39debc..cb66e63f 100644--- a/src/client/src/__tests__/lib/platform/observability.test.ts+++ b/src/client/src/__tests__/lib/platform/observability.test.ts@@ -203,4 +203,134 @@ describe("observability platform queries", () => {"MetricName = 'cpu.usage'");});++ it("returns empty totals and default sort when log/metric payloads are sparse", async () => {+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [{}], err: null })+ .mockResolvedValueOnce({ data: [], err: "records failed" });++ await expect(getLogs(params as any)).resolves.toEqual({+ err: "records failed",+ records: [],+ total: 0,+ });+ expect((dataCollector as jest.Mock).mock.calls[1][0].query).toContain(+ "ORDER BY Timestamp desc"+ );++ (dataCollector as jest.Mock).mockClear();+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: null, err: null })+ .mockResolvedValueOnce({ data: [], err: null });++ await expect(getMetrics(params as any)).resolves.toEqual({+ err: null,+ records: [],+ total: 0,+ });+ });++ it("covers exceptions summaries, longer buckets, and empty summary data", async () => {+ (dataCollector as jest.Mock).mockResolvedValue({ data: null, err: null });++ const weekParams = {+ ...params,+ timeLimit: {+ type: "CUSTOM",+ start: "2025-01-01T00:00:00.000Z",+ end: "2025-03-01T00:00:00.000Z",+ },+ };+ const week = await getSignalSummary(weekParams as any, "exceptions");+ expect(week.bucket).toBe("week");+ expect(week.total).toBe(0);+ expect(week.peak).toBe(0);+ expect((dataCollector as jest.Mock).mock.calls[0][0].query).toContain(+ "FROM otel_traces"+ );++ const monthParams = {+ ...params,+ timeLimit: {+ type: "CUSTOM",+ start: "2024-01-01T00:00:00.000Z",+ end: "2025-06-01T00:00:00.000Z",+ },+ };+ const month = await getSignalSummary(monthParams as any, "logs");+ expect(month.bucket).toBe("month");+ expect((dataCollector as jest.Mock).mock.calls[1][0].query).toContain("%Y/%m");++ const dayParams = {+ ...params,+ timeLimit: {+ type: "CUSTOM",+ start: "2026-01-01T00:00:00.000Z",+ end: "2026-01-20T00:00:00.000Z",+ },+ };+ const day = await getSignalSummary(dayParams as any, "metrics");+ expect(day.bucket).toBe("day");+ });++ it("applies Field/Resource/Scope custom filters and metric name filters", async () => {+ await getLogsConfig({+ ...params,+ selectedConfig: {+ traceIds: ["t1"],+ spanIds: ["s1"],+ customFilters: [+ { attributeType: "ResourceAttributes", key: "env", value: "prod" },+ { attributeType: "ScopeAttributes", key: "lib", value: "otel" },+ { attributeType: "Field", key: "SeverityText", value: "ERROR" },+ { attributeType: "Field", key: "Drop;Me", value: "nope" },+ { attributeType: "LogAttributes", key: "", value: "skip" },+ ],+ },+ } as any);++ const logsQuery = (dataCollector as jest.Mock).mock.calls[0][0].query;+ expect(logsQuery).toContain("ResourceAttributes['env'] = 'prod'");+ expect(logsQuery).toContain("ScopeAttributes['lib'] = 'otel'");+ expect(logsQuery).toContain("SeverityText = 'ERROR'");+ expect(logsQuery).toContain("TraceId IN ('t1')");+ expect(logsQuery).toContain("SpanId IN ('s1')");++ (dataCollector as jest.Mock).mockClear();+ await getMetricsConfig({+ ...params,+ selectedConfig: {+ metricNames: ["cpu"],+ customFilters: [+ { attributeType: "Attributes", key: "host", value: "a" },+ { attributeType: "Field", key: "MetricName", value: "cpu" },+ ],+ },+ } as any);+ const metricsQuery = (dataCollector as jest.Mock).mock.calls[0][0].query;+ expect(metricsQuery).toContain("MetricName IN ('cpu')");+ expect(metricsQuery).toContain("Attributes['host'] = 'a'");+ expect(metricsQuery).toContain("MetricName = 'cpu'");+ });++ it("propagates attribute-key query errors and sanitizes empty row ids", async () => {+ (dataCollector as jest.Mock)+ .mockResolvedValueOnce({ data: [], err: "log keys failed" })+ .mockResolvedValueOnce({ data: [], err: null })+ .mockResolvedValueOnce({ data: [], err: null });++ const keys = await getLogAttributeKeys(params as any);+ expect(keys.err).toBe("log keys failed");+ expect(keys.logAttributeKeys).toEqual([]);++ (dataCollector as jest.Mock).mockClear();+ await getLogByRowId("abc");+ expect((dataCollector as jest.Mock).mock.calls[0][0].query).toContain("= 0");++ (dataCollector as jest.Mock).mockClear();+ await getMetricDetail("cpu.usage");+ expect((dataCollector as jest.Mock).mock.calls[0][0].query).toContain(+ "MetricName = 'cpu.usage'"+ );+ });});diff --git a/src/client/src/__tests__/lib/platform/openground/cost-analytics.test.ts b/src/client/src/__tests__/lib/platform/openground/cost-analytics.test.tsnew file mode 100644index 00000000..565feb16--- /dev/null+++ b/src/client/src/__tests__/lib/platform/openground/cost-analytics.test.ts@@ -0,0 +1,79 @@+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+}));++jest.mock("@/lib/db-config", () => ({+ getDBConfigByUser: jest.fn(),+}));++jest.mock("@/utils/asaw", () => ({+ __esModule: true,+ default: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterPreviousParams: jest.fn((params) => params),+}));++import { dataCollector } from "@/lib/platform/common";+import asaw from "@/utils/asaw";+import {+ getOpengroundCostByProvider,+ getOpengroundTotalCost,+} from "@/lib/platform/openground/cost-analytics";++const mockedDataCollector = dataCollector as jest.MockedFunction<typeof dataCollector>;+const mockedAsaw = asaw as jest.MockedFunction<typeof asaw>;++const params = {+ timeLimit: {+ start: new Date("2024-01-01"),+ end: new Date("2024-01-02"),+ type: "custom",+ },+};++describe("openground cost analytics", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ mockedAsaw.mockReset();+ mockedAsaw.mockResolvedValue([null, { id: "db-1" }]);+ mockedDataCollector.mockResolvedValue({ data: [] });+ });++ it("returns zeros when no active database config", async () => {+ mockedAsaw.mockResolvedValueOnce([null, null]);+ const res = await getOpengroundTotalCost(params);+ expect(res).toEqual({+ data: [{ total_cost: 0, previous_total_cost: 0 }],+ });+ expect(mockedDataCollector).not.toHaveBeenCalled();+ });++ it("queries total openground cost for the active database", async () => {+ await getOpengroundTotalCost(params);+ const query = mockedDataCollector.mock.calls[0][0].query as string;+ expect(query).toContain("openlit_openground_providers");+ // Float64 cost columns must not use toFloat64OrZero (ClickHouse rejects it).+ expect(query).not.toContain("toFloat64OrZero");+ expect(query).toContain("sum(ifNull(p.cost, 0))");+ expect(mockedDataCollector).toHaveBeenCalledWith(+ expect.objectContaining({ query }),+ "query",+ "db-1"+ );+ });++ it("queries openground cost by provider", async () => {+ await getOpengroundCostByProvider(params);+ const query = mockedDataCollector.mock.calls[0][0].query as string;+ expect(query).toContain("AS provider");+ expect(query).not.toContain("toFloat64OrZero");+ expect(query).toContain("SUM(ifNull(p.cost, 0))");+ expect(mockedDataCollector).toHaveBeenCalledWith(+ expect.objectContaining({ query }),+ "query",+ "db-1"+ );+ });+});diff --git a/src/client/src/__tests__/lib/platform/pricing/cron-analytics.test.ts b/src/client/src/__tests__/lib/platform/pricing/cron-analytics.test.tsnew file mode 100644index 00000000..6aa198e5--- /dev/null+++ b/src/client/src/__tests__/lib/platform/pricing/cron-analytics.test.ts@@ -0,0 +1,91 @@+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterPreviousParams: jest.fn((params) => ({+ ...params,+ timeLimit: {+ ...params.timeLimit,+ start: "2024-01-01T00:00:00.000Z",+ end: "2024-01-01T12:00:00.000Z",+ },+ })),+}));++import { dataCollector } from "@/lib/platform/common";+import { getPricingCronAnalytics } from "@/lib/platform/pricing/cron-analytics";++const mockedDataCollector = dataCollector as jest.MockedFunction<+ typeof dataCollector+>;++const params = {+ timeLimit: {+ start: "2024-01-01T12:00:00.000Z",+ end: "2024-01-02T12:00:00.000Z",+ type: "24H",+ },+};++describe("getPricingCronAnalytics", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ });++ it("returns summary stats and recent runs with costs applied", async () => {+ mockedDataCollector+ .mockResolvedValueOnce({+ data: [+ {+ totalRuns: 4,+ successfulRuns: 3,+ totalUpdated: 12,+ totalSpans: 40,+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ totalRuns: 2,+ successfulRuns: 2,+ totalUpdated: 5,+ totalSpans: 20,+ },+ ],+ })+ .mockResolvedValueOnce({+ data: [+ {+ startedAt: "2024-01-02 10:00:00",+ finishedAt: "2024-01-02 10:00:05",+ duration: 5,+ runStatus: "SUCCESS",+ totalSpans: 10,+ totalUpdated: 4,+ totalFailed: 0,+ totalSkipped: 6,+ },+ ],+ });++ const res = await getPricingCronAnalytics(params);++ expect(res.data[0]).toEqual({+ total_runs: 4,+ previous_total_runs: 2,+ successful_runs: 3,+ previous_successful_runs: 2,+ total_updated: 12,+ previous_total_updated: 5,+ total_spans: 40,+ previous_total_spans: 20,+ });+ expect(res.runs).toHaveLength(1);+ expect(res.runs[0].totalUpdated).toBe(4);+ expect(mockedDataCollector.mock.calls[2][0].query).toContain(+ "totalUpdated"+ );+ });+});diff --git a/src/client/src/__tests__/lib/platform/pricing/guidance.test.ts b/src/client/src/__tests__/lib/platform/pricing/guidance.test.tsnew file mode 100644index 00000000..b4248b8e--- /dev/null+++ b/src/client/src/__tests__/lib/platform/pricing/guidance.test.ts@@ -0,0 +1,80 @@+jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+ OTEL_TRACES_TABLE_NAME: "otel_traces",+}));++jest.mock("@/helpers/server/platform", () => ({+ getFilterWhereCondition: jest.fn(() => "1 = 1"),+}));++jest.mock("@/helpers/server/trace", () => ({+ getTraceMappingKeyFullPath: jest.fn(() => "gen_ai.usage.cost"),+}));++jest.mock("@/lib/platform/pricing/config", () => ({+ getPricingConfig: jest.fn(),+}));++import { dataCollector } from "@/lib/platform/common";+import { getPricingConfig } from "@/lib/platform/pricing/config";+import { getCostPricingGuidance } from "@/lib/platform/pricing/guidance";++const mockedDataCollector = dataCollector as jest.MockedFunction<+ typeof dataCollector+>;+const mockedGetPricingConfig = getPricingConfig as jest.MockedFunction<+ typeof getPricingConfig+>;++const params = {+ timeLimit: {+ start: "2024-01-01T00:00:00.000Z",+ end: "2024-01-02T00:00:00.000Z",+ type: "24H",+ },+};++describe("getCostPricingGuidance", () => {+ beforeEach(() => {+ mockedDataCollector.mockReset();+ mockedGetPricingConfig.mockReset();+ });++ it("shows backfill banner when auto pricing is off and missing costs exist", async () => {+ mockedGetPricingConfig.mockResolvedValue({ auto: false } as never);+ mockedDataCollector.mockResolvedValue({+ data: [{ missing_cost_spans: 12 }],+ });++ const res = await getCostPricingGuidance(params);++ expect(res).toEqual({+ autoEnabled: false,+ missingCostSpans: 12,+ showBackfillBanner: true,+ });+ });++ it("hides banner when auto pricing is already enabled", async () => {+ mockedGetPricingConfig.mockResolvedValue({ auto: true } as never);+ mockedDataCollector.mockResolvedValue({+ data: [{ missing_cost_spans: 12 }],+ });++ const res = await getCostPricingGuidance(params);++ expect(res.showBackfillBanner).toBe(false);+ expect(res.autoEnabled).toBe(true);+ });++ it("hides banner when there are no missing costs", async () => {+ mockedGetPricingConfig.mockResolvedValue({ auto: false } as never);+ mockedDataCollector.mockResolvedValue({+ data: [{ missing_cost_spans: 0 }],+ });++ const res = await getCostPricingGuidance(params);++ expect(res.showBackfillBanner).toBe(false);+ });+});diff --git a/src/client/src/__tests__/lib/platform/pricing/pricing.test.ts b/src/client/src/__tests__/lib/platform/pricing/pricing.test.tsindex 0fd3fd68..5105883d 100644--- a/src/client/src/__tests__/lib/platform/pricing/pricing.test.ts+++ b/src/client/src/__tests__/lib/platform/pricing/pricing.test.ts@@ -27,7 +27,7 @@ jest.mock('@/helpers/server/trace', () => ({const map: Record<string, string> = {cost: 'gen_ai.usage.cost',model: 'gen_ai.request.model',- provider: 'gen_ai.system',+ provider: 'gen_ai.provider.name',promptTokens: 'gen_ai.usage.input_tokens',completionTokens: 'gen_ai.usage.output_tokens',type: 'gen_ai.operation.name',@@ -36,6 +36,7 @@ jest.mock('@/helpers/server/trace', () => ({}),getTraceMappingKeyFullPaths: jest.fn((key: string) => {const map: Record<string, string[]> = {+ provider: ['gen_ai.provider.name', 'gen_ai.system'],promptTokens: ['gen_ai.usage.input_tokens', 'input_tokens', 'prompt_tokens'],completionTokens: ['gen_ai.usage.output_tokens', 'output_tokens', 'completion_tokens'],};diff --git a/src/client/src/__tests__/lib/platform/request/request.test.ts b/src/client/src/__tests__/lib/platform/request/request.test.tsindex 465f03e1..6d2d2d9b 100644--- a/src/client/src/__tests__/lib/platform/request/request.test.ts+++ b/src/client/src/__tests__/lib/platform/request/request.test.ts@@ -46,7 +46,7 @@ describe('getRequestPerTime', () => {describe('getGroupByExpression', () => {it('returns predefined group-by expressions', () => {expect(getGroupByExpression('model')).toBe("SpanAttributes['gen_ai.request.model']");- expect(getGroupByExpression('provider')).toBe("SpanAttributes['gen_ai.system']");+ expect(getGroupByExpression('provider')).toBe("SpanAttributes['gen_ai.provider.name']");});it('builds safe span/resource/field group-by expressions', () => {@@ -93,6 +93,10 @@ describe('getRequestsConfig', () => {expect(query).toContain('providers');expect(query).toContain('models');expect(query).toContain('totalRows');+ expect(query).toContain('services');+ // Provider dropdown folds the current OTel key and the legacy fallback.+ expect(query).toContain("gen_ai.provider.name");+ expect(query).toContain("gen_ai.system");});});@@ -265,6 +269,11 @@ describe('getHeirarchyViaSpanId', () => {const allSpansQuery = secondCall[0].query as string;expect(allSpansQuery).toContain("SpanAttributes['coding_agent.session.id'] = 'parent-1'");expect(allSpansQuery).toContain("ResourceAttributes['coding_agent.agent.parent_id'] = 'parent-1'");+ // Coding sessions take the newest window (DESC) and UNION prompt/response+ // turns so Chat view cannot lose user inputs outside a 5k ASC slice.+ expect(allSpansQuery).toContain('ORDER BY Timestamp DESC');+ expect(allSpansQuery).toContain("gen_ai.input.messages");+ expect(allSpansQuery).toContain('UNION DISTINCT');expect(result.err).toBeNull();});@@ -351,7 +360,7 @@ describe('getAttributeKeys', () => {describe('getGroupedRequests', () => {it('returns predefined and attribute group expressions', () => {expect(getGroupByExpression('model')).toBe("SpanAttributes['gen_ai.request.model']");- expect(getGroupByExpression('provider')).toBe("SpanAttributes['gen_ai.system']");+ expect(getGroupByExpression('provider')).toBe("SpanAttributes['gen_ai.provider.name']");expect(getGroupByExpression('applicationName')).toBe("ResourceAttributes['service.name']");expect(getGroupByExpression('custom.attribute')).toBe("SpanAttributes['custom.attribute']");expect(getGroupByExpression('ResourceAttributes:service.namespace')).toBe(diff --git a/src/client/src/__tests__/lib/platform/telemetry-snapshot/config.test.ts b/src/client/src/__tests__/lib/platform/telemetry-snapshot/config.test.tsnew file mode 100644index 00000000..d5fdc741--- /dev/null+++ b/src/client/src/__tests__/lib/platform/telemetry-snapshot/config.test.ts@@ -0,0 +1,110 @@+const mockUpdateCrontab = jest.fn();+const mockDeleteCronJob = jest.fn();++jest.mock("@/helpers/server/cron", () =>+ jest.fn().mockImplementation(() => ({+ updateCrontab: mockUpdateCrontab,+ deleteCronJob: mockDeleteCronJob,+ }))+);++import { restoreTelemetrySnapshotCron } from "@/lib/platform/telemetry-snapshot/config";++describe("restoreTelemetrySnapshotCron", () => {+ const originalEnabled = process.env.TELEMETRY_ENABLED;+ const originalSchedule = process.env.TELEMETRY_SNAPSHOT_SCHEDULE;+ let cwdSpy: jest.SpyInstance<string, []>;++ beforeEach(() => {+ jest.clearAllMocks();+ cwdSpy = jest.spyOn(process, "cwd").mockReturnValue("/app");+ delete process.env.TELEMETRY_ENABLED;+ delete process.env.TELEMETRY_SNAPSHOT_SCHEDULE;+ });++ afterEach(() => {+ cwdSpy.mockRestore();+ if (originalEnabled === undefined) {+ delete process.env.TELEMETRY_ENABLED;+ } else {+ process.env.TELEMETRY_ENABLED = originalEnabled;+ }+ if (originalSchedule === undefined) {+ delete process.env.TELEMETRY_SNAPSHOT_SCHEDULE;+ } else {+ process.env.TELEMETRY_SNAPSHOT_SCHEDULE = originalSchedule;+ }+ });++ it("deletes the cron and skips install when telemetry is disabled", async () => {+ process.env.TELEMETRY_ENABLED = "false";+ const logSpy = jest.spyOn(console, "log").mockImplementation();++ await restoreTelemetrySnapshotCron("https://api.example");++ expect(mockDeleteCronJob).toHaveBeenCalledWith("openlit-telemetry-snapshot");+ expect(mockUpdateCrontab).not.toHaveBeenCalled();+ expect(logSpy).toHaveBeenCalledWith(+ expect.stringContaining("Telemetry disabled")+ );+ logSpy.mockRestore();+ });++ it("swallows delete errors when telemetry is disabled", async () => {+ process.env.TELEMETRY_ENABLED = "false";+ mockDeleteCronJob.mockImplementation(() => {+ throw new Error("missing");+ });+ jest.spyOn(console, "log").mockImplementation();++ await expect(+ restoreTelemetrySnapshotCron("https://api.example")+ ).resolves.toBeUndefined();+ });++ it("installs the cron with default schedule", async () => {+ const logSpy = jest.spyOn(console, "log").mockImplementation();++ await restoreTelemetrySnapshotCron("https://api.openlit.local");++ expect(mockUpdateCrontab).toHaveBeenCalledWith({+ cronId: "openlit-telemetry-snapshot",+ cronSchedule: "17 3 * * *",+ cronEnvVars: {+ API_URL: "https://api.openlit.local",+ CRON_ID: "openlit-telemetry-snapshot",+ },+ cronScriptPath: "/app/scripts/telemetry-snapshot/snapshot.js",+ cronLogPath: "/app/logs/telemetry-snapshot/snapshot.log",+ });+ expect(logSpy).toHaveBeenCalledWith("Installed telemetry snapshot cron");+ logSpy.mockRestore();+ });++ it("honors TELEMETRY_SNAPSHOT_SCHEDULE override", async () => {+ process.env.TELEMETRY_SNAPSHOT_SCHEDULE = "0 12 * * *";+ jest.spyOn(console, "log").mockImplementation();++ await restoreTelemetrySnapshotCron("https://api.example");++ expect(mockUpdateCrontab).toHaveBeenCalledWith(+ expect.objectContaining({ cronSchedule: "0 12 * * *" })+ );+ });++ it("logs and swallows install failures", async () => {+ mockUpdateCrontab.mockImplementation(() => {+ throw new Error("crontab write failed");+ });+ const errSpy = jest.spyOn(console, "error").mockImplementation();++ await expect(+ restoreTelemetrySnapshotCron("https://api.example")+ ).resolves.toBeUndefined();+ expect(errSpy).toHaveBeenCalledWith(+ "Failed to install telemetry snapshot cron:",+ expect.any(Error)+ );+ errSpy.mockRestore();+ });+});diff --git a/src/client/src/__tests__/lib/platform/telemetry-snapshot/index.test.ts b/src/client/src/__tests__/lib/platform/telemetry-snapshot/index.test.tsnew file mode 100644index 00000000..bab7f944--- /dev/null+++ b/src/client/src/__tests__/lib/platform/telemetry-snapshot/index.test.ts@@ -0,0 +1,179 @@+jest.mock("@/lib/prisma", () => ({+ __esModule: true,+ default: {+ user: { count: jest.fn() },+ organisation: { count: jest.fn() },+ project: { count: jest.fn() },+ databaseConfig: {+ count: jest.fn(),+ findMany: jest.fn(),+ },+ evaluationConfigs: { count: jest.fn() },+ pricingConfigs: { count: jest.fn() },+ },+}));++jest.mock("@/lib/posthog", () => ({+ __esModule: true,+ default: {+ capture: jest.fn().mockResolvedValue(undefined),+ },+}));++jest.mock("@/lib/platform/common", () => ({+ dataCollector: jest.fn(),+ OTEL_TRACES_TABLE_NAME: "otel_traces",+}));++jest.mock("@/lib/platform/telemetry-snapshot/install-id", () => ({+ getInstallId: jest.fn().mockResolvedValue("install-abc"),+}));++jest.mock("fs", () => ({+ ...jest.requireActual("fs"),+ readFileSync: jest.fn(),+}));++import prisma from "@/lib/prisma";+import PostHogServer from "@/lib/posthog";+import { dataCollector } from "@/lib/platform/common";+import { getInstallId } from "@/lib/platform/telemetry-snapshot/install-id";+import { captureInstanceSnapshot } from "@/lib/platform/telemetry-snapshot";+import { readFileSync } from "fs";+import { SERVER_EVENTS } from "@/constants/events";++describe("captureInstanceSnapshot", () => {+ const originalEnabled = process.env.TELEMETRY_ENABLED;+ const originalVersion = process.env.npm_package_version;++ beforeEach(() => {+ jest.clearAllMocks();+ delete process.env.TELEMETRY_ENABLED;+ delete process.env.npm_package_version;++ (prisma.user.count as jest.Mock).mockResolvedValue(2);+ (prisma.organisation.count as jest.Mock).mockResolvedValue(1);+ (prisma.project.count as jest.Mock).mockResolvedValue(3);+ (prisma.databaseConfig.count as jest.Mock).mockResolvedValue(1);+ (prisma.evaluationConfigs.count as jest.Mock).mockResolvedValue(0);+ (prisma.pricingConfigs.count as jest.Mock).mockResolvedValue(1);+ (prisma.databaseConfig.findMany as jest.Mock).mockResolvedValue([+ { id: "db-1" },+ ]);+ (dataCollector as jest.Mock).mockResolvedValue({ data: [{ c: 5 }] });+ (getInstallId as jest.Mock).mockResolvedValue("install-abc");+ });++ afterEach(() => {+ if (originalEnabled === undefined) {+ delete process.env.TELEMETRY_ENABLED;+ } else {+ process.env.TELEMETRY_ENABLED = originalEnabled;+ }+ if (originalVersion === undefined) {+ delete process.env.npm_package_version;+ } else {+ process.env.npm_package_version = originalVersion;+ }+ });++ it("returns early when telemetry is disabled", async () => {+ process.env.TELEMETRY_ENABLED = "false";+ await expect(captureInstanceSnapshot()).resolves.toEqual({+ success: false,+ reason: "telemetry_disabled",+ });+ expect(getInstallId).not.toHaveBeenCalled();+ });++ it("captures aggregate counts and sends PostHog event", async () => {+ process.env.npm_package_version = "9.9.9";++ const result = await captureInstanceSnapshot();++ expect(result.success).toBe(true);+ expect(result.properties).toMatchObject({+ install_id: "install-abc",+ openlit_version: "9.9.9",+ users_total: 2,+ organisations_total: 1,+ projects_total: 3,+ db_configs_total: 1,+ eval_auto_enabled: false,+ pricing_auto_enabled: true,+ spans_total: 5,+ });+ expect(PostHogServer.capture).toHaveBeenCalledWith({+ event: SERVER_EVENTS.INSTANCE_TELEMETRY_SNAPSHOT,+ distinctId: "install-abc",+ properties: expect.objectContaining({ install_id: "install-abc" }),+ });+ });++ it("reads version from package.json when npm_package_version is unset", async () => {+ (readFileSync as jest.Mock).mockReturnValue(+ JSON.stringify({ version: "1.2.3" })+ );++ const result = await captureInstanceSnapshot();++ expect(result.properties?.openlit_version).toBe("1.2.3");+ });++ it("falls back to unknown version when package.json cannot be read", async () => {+ (readFileSync as jest.Mock).mockImplementation(() => {+ throw new Error("missing");+ });++ const result = await captureInstanceSnapshot();++ expect(result.properties?.openlit_version).toBe("unknown");+ });++ it("treats ClickHouse query failures as zero counts", async () => {+ (dataCollector as jest.Mock).mockRejectedValue(new Error("ch down"));+ process.env.npm_package_version = "1.0.0";++ const result = await captureInstanceSnapshot();++ expect(result.success).toBe(true);+ expect(result.properties).toMatchObject({+ spans_total: 0,+ traces_total: 0,+ app_agents_total: 0,+ });+ });++ it("sums totals across multiple database configs", async () => {+ (prisma.databaseConfig.findMany as jest.Mock).mockResolvedValue([+ { id: "db-1" },+ { id: "db-2" },+ ]);+ (dataCollector as jest.Mock).mockResolvedValue({ data: [{ c: 3 }] });+ process.env.npm_package_version = "1.0.0";++ const result = await captureInstanceSnapshot();++ expect(result.properties?.spans_total).toBe(6);+ });++ it("ignores non-numeric ClickHouse values", async () => {+ (dataCollector as jest.Mock).mockResolvedValue({+ data: [{ c: "not-a-number" }],+ });+ process.env.npm_package_version = "1.0.0";++ const result = await captureInstanceSnapshot();++ expect(result.properties?.spans_total).toBe(0);+ });++ it("handles empty ClickHouse rows", async () => {+ (dataCollector as jest.Mock).mockResolvedValue({ data: [] });+ process.env.npm_package_version = "1.0.0";++ const result = await captureInstanceSnapshot();++ expect(result.properties?.spans_total).toBe(0);+ });+});diff --git a/src/client/src/__tests__/lib/platform/telemetry-snapshot/install-id.test.ts b/src/client/src/__tests__/lib/platform/telemetry-snapshot/install-id.test.tsnew file mode 100644index 00000000..9e111bb5--- /dev/null+++ b/src/client/src/__tests__/lib/platform/telemetry-snapshot/install-id.test.ts@@ -0,0 +1,67 @@+jest.mock("@/lib/prisma", () => ({+ __esModule: true,+ default: {+ clickhouseMigrations: {+ findFirst: jest.fn(),+ create: jest.fn(),+ },+ },+}));++import prisma from "@/lib/prisma";+import { getInstallId } from "@/lib/platform/telemetry-snapshot/install-id";++const findFirst = prisma.clickhouseMigrations.findFirst as jest.Mock;+const create = prisma.clickhouseMigrations.create as jest.Mock;++describe("getInstallId", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("returns the existing install id when present", async () => {+ findFirst.mockResolvedValue({+ clickhouseMigrationId: "existing-id",+ databaseConfigId: "telemetry:install-id",+ });++ await expect(getInstallId()).resolves.toBe("existing-id");+ expect(create).not.toHaveBeenCalled();+ });++ it("creates and returns a new id when none exists", async () => {+ findFirst.mockResolvedValue(null);+ create.mockResolvedValue({});++ const id = await getInstallId();+ expect(typeof id).toBe("string");+ expect(id.length).toBeGreaterThan(0);+ expect(create).toHaveBeenCalledWith({+ data: {+ databaseConfigId: "telemetry:install-id",+ clickhouseMigrationId: id,+ },+ });+ });++ it("re-reads after a create race and returns the winner", async () => {+ findFirst+ .mockResolvedValueOnce(null)+ .mockResolvedValueOnce({+ clickhouseMigrationId: "winner-id",+ databaseConfigId: "telemetry:install-id",+ });+ create.mockRejectedValue(new Error("unique constraint"));++ await expect(getInstallId()).resolves.toBe("winner-id");+ });++ it("returns the locally generated id when create fails and re-read is empty", async () => {+ findFirst.mockResolvedValue(null);+ create.mockRejectedValue(new Error("db down"));++ const id = await getInstallId();+ expect(typeof id).toBe("string");+ expect(id.length).toBeGreaterThan(0);+ });+});diff --git a/src/client/src/__tests__/lib/rbac/current.test.ts b/src/client/src/__tests__/lib/rbac/current.test.tsnew file mode 100644index 00000000..8321a38f--- /dev/null+++ b/src/client/src/__tests__/lib/rbac/current.test.ts@@ -0,0 +1,46 @@+beforeAll(() => {+ Object.defineProperty(global, "Response", {+ value: {+ json: (body: unknown, init?: ResponseInit) => ({+ status: init?.status ?? 200,+ json: jest.fn().mockResolvedValue(body),+ }),+ },+ configurable: true,+ });+});++import {+ requireCurrentOrganisationPermission,+ requireCurrentOrganisationEntitledPermission,+ withCurrentOrganisationPermission,+} from "@/lib/rbac/current";++describe("CE rbac current organisation fallbacks", () => {+ it("requireCurrentOrganisationPermission resolves to null", async () => {+ await expect(+ requireCurrentOrganisationPermission("dashboard.read")+ ).resolves.toBeNull();+ });++ it("requireCurrentOrganisationEntitledPermission resolves to null", async () => {+ await expect(+ requireCurrentOrganisationEntitledPermission(+ "feature.alerts",+ "alerts.read"+ )+ ).resolves.toBeNull();+ });++ it("withCurrentOrganisationPermission returns the handler unchanged", async () => {+ const handler = jest.fn().mockResolvedValue(Response.json({ ok: true }));+ const wrapped = withCurrentOrganisationPermission(+ "dashboard.read",+ handler+ );++ expect(wrapped).toBe(handler);+ await wrapped({} as Request, {});+ expect(handler).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/client/src/__tests__/lib/rbac/route.test.ts b/src/client/src/__tests__/lib/rbac/route.test.tsnew file mode 100644index 00000000..5a0dddad--- /dev/null+++ b/src/client/src/__tests__/lib/rbac/route.test.ts@@ -0,0 +1,261 @@+jest.mock("@/lib/session", () => ({+ getCurrentUser: jest.fn(),+}));++jest.mock("@/lib/db-config", () => ({+ getDBConfigByIdForUser: jest.fn(),+}));++jest.mock("@/lib/organisation", () => ({+ getCurrentOrganisation: jest.fn(),+ getCurrentProjectForOrganisation: jest.fn(),+}));++jest.mock("@/constants/messages", () => ({+ __esModule: true,+ default: jest.fn(() => ({+ UNAUTHORIZED_USER: "Unauthorized",+ PROJECT_ACCESS_REQUIRED: "Project access required",+ })),+}));++import { getCurrentUser } from "@/lib/session";+import { getDBConfigByIdForUser } from "@/lib/db-config";+import {+ getCurrentOrganisation,+ getCurrentProjectForOrganisation,+} from "@/lib/organisation";+import {+ withPermission,+ withEntitledPermission,+ withDbConfigAccess,+} from "@/lib/rbac/route";+import { OPENLIT_CONTEXT_HEADERS } from "@/constants/openlit-context";++beforeAll(() => {+ Object.defineProperty(global, "Response", {+ value: {+ json: (body: unknown, init?: ResponseInit) => ({+ status: init?.status ?? 200,+ json: jest.fn().mockResolvedValue(body),+ }),+ },+ configurable: true,+ });+});++function jsonRequest(+ body: unknown,+ headers: Record<string, string> = {}+): Request {+ return {+ clone: () => ({+ json: jest.fn().mockResolvedValue(body),+ }),+ headers: {+ get: (key: string) => headers[key] ?? null,+ },+ } as unknown as Request;+}++describe("CE rbac route wrappers", () => {+ beforeEach(() => {+ jest.clearAllMocks();+ });++ it("withPermission returns the handler unchanged", async () => {+ const handler = jest.fn().mockResolvedValue(Response.json({ ok: true }));+ const wrapped = withPermission("dashboard.read", handler);++ expect(wrapped).toBe(handler);+ await wrapped({} as Request, {});+ expect(handler).toHaveBeenCalledTimes(1);+ });++ it("withEntitledPermission returns the handler unchanged", async () => {+ const handler = jest.fn().mockResolvedValue(Response.json({ ok: true }));+ const wrapped = withEntitledPermission(+ "feature.alerts",+ "alerts.read",+ handler+ );++ expect(wrapped).toBe(handler);+ await wrapped({} as Request, {});+ expect(handler).toHaveBeenCalledTimes(1);+ });+});++describe("withDbConfigAccess", () => {+ const handler = jest.fn();++ beforeEach(() => {+ jest.clearAllMocks();+ handler.mockReset();+ handler.mockResolvedValue(Response.json({ ok: true }));+ });++ it("returns 401 when there is no current user", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue(null);+ const wrapped = withDbConfigAccess(handler);++ const res = await wrapped(jsonRequest({}), {});++ expect(res.status).toBe(401);+ await expect(res.json()).resolves.toEqual({ error: "Unauthorized" });+ expect(handler).not.toHaveBeenCalled();+ });++ it("returns 403 when no db config id and org has no current project", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getCurrentOrganisation as jest.Mock).mockResolvedValue({ id: "org1" });+ (getCurrentProjectForOrganisation as jest.Mock).mockResolvedValue(null);+ const wrapped = withDbConfigAccess(handler);++ const res = await wrapped(jsonRequest({}), {});++ expect(res.status).toBe(403);+ await expect(res.json()).resolves.toEqual({+ error: "Project access required",+ });+ expect(handler).not.toHaveBeenCalled();+ });++ it("calls handler when no db config id and no current organisation", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getCurrentOrganisation as jest.Mock).mockResolvedValue(null);+ const wrapped = withDbConfigAccess(handler);++ await wrapped(jsonRequest({}), {});++ expect(handler).toHaveBeenCalledTimes(1);+ expect(getCurrentProjectForOrganisation).not.toHaveBeenCalled();+ });++ it("calls handler when no db config id and org has a current project", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getCurrentOrganisation as jest.Mock).mockResolvedValue({ id: "org1" });+ (getCurrentProjectForOrganisation as jest.Mock).mockResolvedValue({+ id: "p1",+ });+ const wrapped = withDbConfigAccess(handler);++ await wrapped(jsonRequest({}), {});++ expect(handler).toHaveBeenCalledTimes(1);+ });++ it("returns 403 when db config is not accessible to the user", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getDBConfigByIdForUser as jest.Mock).mockResolvedValue(null);+ const wrapped = withDbConfigAccess(handler);++ const res = await wrapped(jsonRequest({ databaseConfigId: "db-1" }), {});++ expect(getDBConfigByIdForUser).toHaveBeenCalledWith({+ id: "db-1",+ userId: "u1",+ });+ expect(res.status).toBe(403);+ await expect(res.json()).resolves.toEqual({+ error: "Project access required",+ });+ expect(handler).not.toHaveBeenCalled();+ });++ it("calls handler when db config is accessible", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getDBConfigByIdForUser as jest.Mock).mockResolvedValue({ id: "db-1" });+ const wrapped = withDbConfigAccess(handler);++ await wrapped(jsonRequest({ databaseConfigId: "db-1" }), {});++ expect(handler).toHaveBeenCalledTimes(1);+ });++ it.each([+ [{ dbConfigId: "from-dbConfigId" }, "from-dbConfigId"],+ [{ selectedConfig: "from-selected-string" }, "from-selected-string"],+ [+ { selectedConfig: { databaseConfigId: "from-selected-databaseConfigId" } },+ "from-selected-databaseConfigId",+ ],+ [+ { selectedConfig: { dbConfigId: "from-selected-dbConfigId" } },+ "from-selected-dbConfigId",+ ],+ ])("extracts database config id from body %#", async (body, expectedId) => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getDBConfigByIdForUser as jest.Mock).mockResolvedValue({ id: expectedId });+ const wrapped = withDbConfigAccess(handler);++ await wrapped(jsonRequest(body), {});++ expect(getDBConfigByIdForUser).toHaveBeenCalledWith({+ id: expectedId,+ userId: "u1",+ });+ expect(handler).toHaveBeenCalledTimes(1);+ });++ it("extracts database config id from request header", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getDBConfigByIdForUser as jest.Mock).mockResolvedValue({ id: "hdr-db" });+ const wrapped = withDbConfigAccess(handler);++ await wrapped(+ jsonRequest(+ {},+ { [OPENLIT_CONTEXT_HEADERS.databaseConfigId]: "hdr-db" }+ ),+ {}+ );++ expect(getDBConfigByIdForUser).toHaveBeenCalledWith({+ id: "hdr-db",+ userId: "u1",+ });+ expect(handler).toHaveBeenCalledTimes(1);+ });++ it("treats invalid JSON body as empty and continues without db config id", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getCurrentOrganisation as jest.Mock).mockResolvedValue(null);+ const request = {+ clone: () => ({+ json: jest.fn().mockRejectedValue(new Error("bad json")),+ }),+ headers: { get: () => null },+ } as unknown as Request;+ const wrapped = withDbConfigAccess(handler);++ await wrapped(request, {});++ expect(handler).toHaveBeenCalledTimes(1);+ expect(getDBConfigByIdForUser).not.toHaveBeenCalled();+ });++ it("uses empty body when request has no clone", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getCurrentOrganisation as jest.Mock).mockResolvedValue(null);+ const request = {+ headers: { get: () => null },+ } as unknown as Request;+ const wrapped = withDbConfigAccess(handler);++ await wrapped(request, {});++ expect(handler).toHaveBeenCalledTimes(1);+ });++ it("ignores blank string database config ids", async () => {+ (getCurrentUser as jest.Mock).mockResolvedValue({ id: "u1" });+ (getCurrentOrganisation as jest.Mock).mockResolvedValue(null);+ const wrapped = withDbConfigAccess(handler);++ await wrapped(jsonRequest({ databaseConfigId: " " }), {});++ expect(getDBConfigByIdForUser).not.toHaveBeenCalled();+ expect(handler).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/client/src/__tests__/lib/rbac/routes/api/routes.test.ts b/src/client/src/__tests__/lib/rbac/routes/api/routes.test.tsindex 8979d49a..b9e5b100 100644--- a/src/client/src/__tests__/lib/rbac/routes/api/routes.test.ts+++ b/src/client/src/__tests__/lib/rbac/routes/api/routes.test.ts@@ -4,6 +4,7 @@ import * as alertByIdRoute from "@/lib/rbac/routes/api/alerts/[id]/route";import * as alertTestRoute from "@/lib/rbac/routes/api/alerts/[id]/test/route";import * as alertEventsRoute from "@/lib/rbac/routes/api/alerts/events/route";import * as alertSpecRoute from "@/lib/rbac/routes/api/alerts/spec/route";+import * as alertConditionValuesRoute from "@/lib/rbac/routes/api/alerts/condition-values/route";import * as alertDestinationsRoute from "@/lib/rbac/routes/api/alert-destinations/route";import * as alertDestinationByIdRoute from "@/lib/rbac/routes/api/alert-destinations/[id]/route";import * as alertDestinationTestRoute from "@/lib/rbac/routes/api/alert-destinations/[id]/test/route";@@ -19,6 +20,7 @@ describe("CE alert route handlers (no entitlement/RBAC available in this edition["alerts/[id]/test/route.ts POST", alertTestRoute.POST],["alerts/events/route.ts GET", alertEventsRoute.GET],["alerts/spec/route.ts GET", alertSpecRoute.GET],+ ["alerts/condition-values/route.ts GET", alertConditionValuesRoute.GET],["alert-destinations/route.ts GET", alertDestinationsRoute.GET],["alert-destinations/route.ts POST", alertDestinationsRoute.POST],["alert-destinations/[id]/route.ts GET", alertDestinationByIdRoute.GET],diff --git a/src/client/src/__tests__/middleware/enterprise-matchers.test.ts b/src/client/src/__tests__/middleware/enterprise-matchers.test.tsnew file mode 100644index 00000000..4b0ce29e--- /dev/null+++ b/src/client/src/__tests__/middleware/enterprise-matchers.test.ts@@ -0,0 +1,8 @@+import { ENTERPRISE_MIDDLEWARE_MATCHERS } from "@/middleware/enterprise-matchers";++describe("ENTERPRISE_MIDDLEWARE_MATCHERS", () => {+ it("is an empty CE placeholder array", () => {+ expect(ENTERPRISE_MIDDLEWARE_MATCHERS).toEqual([]);+ expect(Array.isArray(ENTERPRISE_MIDDLEWARE_MATCHERS)).toBe(true);+ });+});diff --git a/src/client/src/__tests__/utils/breadcrumbs.test.ts b/src/client/src/__tests__/utils/breadcrumbs.test.tsindex 42348fab..6ff0128f 100644--- a/src/client/src/__tests__/utils/breadcrumbs.test.ts+++ b/src/client/src/__tests__/utils/breadcrumbs.test.ts@@ -171,6 +171,30 @@ describe('generatePageHeader', () => {expect(header.breadcrumbs).toContainEqual({ title: 'Settings', href: '/evaluations/settings' });});+ it('generates breadcrumbs for /evaluations/evaluators/:id', () => {+ const header = generatePageHeader('/evaluations/evaluators/bias');+ expect(header.title).toBe('Evaluator');+ expect(header.breadcrumbs).toContainEqual({ title: 'Evaluations', href: '/evaluations' });+ expect(header.breadcrumbs).toContainEqual({ title: 'Evaluator', href: '/evaluations' });+ });++ it('generates breadcrumbs for /evaluations/types/:id', () => {+ const header = generatePageHeader('/evaluations/types/bias');+ expect(header.title).toBe('Evaluation Type');+ expect(header.breadcrumbs).toContainEqual({ title: 'Evaluations', href: '/evaluations' });+ expect(header.breadcrumbs).toContainEqual({+ title: 'Evaluators',+ href: '/evaluations?tab=evaluators',+ });+ });++ it('generates breadcrumbs for /organisation under Settings', () => {+ const header = generatePageHeader('/organisation');+ expect(header.title).toBe('Organisation');+ expect(header.breadcrumbs).toContainEqual({ title: 'Settings', href: '/settings' });+ expect(header.breadcrumbs).toContainEqual({ title: 'Organisation', href: '/organisation' });+ });+it('generates breadcrumbs for /settings/database-config', () => {const header = generatePageHeader('/settings/database-config');expect(header.title).toBe('Database Config');@@ -246,3 +270,35 @@ describe('updatePageHeaderWithData', () => {expect(updated.breadcrumbs).toEqual(baseHeader.breadcrumbs);});});++describe('ROUTE_CONFIGS handlers', () => {+ it('invokes every route title/breadcrumb/description handler', () => {+ const { ROUTE_CONFIGS } = require('@/utils/breadcrumbs') as typeof import('@/utils/breadcrumbs');+ const params = { id: 'abc', userId: encodeURIComponent('[email protected]') };++ expect(ROUTE_CONFIGS.length).toBeGreaterThan(10);++ for (const config of ROUTE_CONFIGS) {+ expect(typeof config.getTitle('/sample', params)).toBe('string');+ const crumbs = config.getBreadcrumbs('/sample', params);+ expect(Array.isArray(crumbs)).toBe(true);+ if (config.getDescription) {+ expect(typeof config.getDescription('/sample', params)).toBe('string');+ }+ }+ });++ it('extracts coding-agents user id params', () => {+ const result = extractRouteParams(+ '/coding-agents/users/alice%40example.com',+ /^\/coding-agents\/users\/[^/]+$/+ );+ expect(result.userId).toBe('alice%40example.com');+ });++ it('falls back for unknown routes', () => {+ const header = generatePageHeader('/totally-unknown-page');+ expect(header.title).toBe('Totally unknown page');+ expect(header.breadcrumbs).toEqual([]);+ });+});diff --git a/src/client/src/app/(playground)/agents/page.tsx b/src/client/src/app/(playground)/agents/page.tsxindex b5471b78..93996970 100644--- a/src/client/src/app/(playground)/agents/page.tsx+++ b/src/client/src/app/(playground)/agents/page.tsx@@ -641,7 +641,7 @@ export default function AgentsPage() {return (<div className="flex h-full w-full flex-col overflow-hidden"><FeaturePageHeader- eyebrow={getMessage().FEATURE_AGENTS}+ eyebrow={getMessage().SIDEBAR_MONITOR}title={activeTab === "coding"? getMessage().AGENTS_TAB_CODING@@ -649,7 +649,7 @@ export default function AgentsPage() {? getMessage().AGENTS_TAB_CONTROLLERS: getMessage().AGENTS_TAB_SERVICES}- icon={<Bot className="h-5 w-5" />}+ icon={<Bot className="h-4 w-4" />}tone={AGENTS_HEADER_TONE}actions={(<div className="flex min-w-0 flex-wrap items-center justify-end gap-2">@@ -673,7 +673,7 @@ export default function AgentsPage() {<Buttonvariant="secondary"size="sm"- className="h-8 bg-primary text-stone-100 hover:bg-primary dark:bg-primary dark:text-stone-100 dark:hover:bg-primary"+ className="h-7 bg-primary text-stone-100 hover:bg-primary dark:bg-primary dark:text-stone-100 dark:hover:bg-primary"onClick={() => setSetupModal("controller")}><Plus className="w-3 h-3 mr-1.5" />@@ -684,7 +684,7 @@ export default function AgentsPage() {<Buttonvariant="secondary"size="sm"- className="h-8 bg-primary text-stone-100 hover:bg-primary dark:bg-primary dark:text-stone-100 dark:hover:bg-primary"+ className="h-7 bg-primary text-stone-100 hover:bg-primary dark:bg-primary dark:text-stone-100 dark:hover:bg-primary"onClick={() => setSetupModal("coding")}><Plus className="w-3 h-3 mr-1.5" />diff --git a/src/client/src/app/(playground)/agents/service-table.tsx b/src/client/src/app/(playground)/agents/service-table.tsxindex aebdd791..09d259d4 100644--- a/src/client/src/app/(playground)/agents/service-table.tsx+++ b/src/client/src/app/(playground)/agents/service-table.tsx@@ -420,11 +420,13 @@ function ActionsCell({const columns: Columns<ServiceColumnKey, EnrichedAgent> = {service: {header: () => getMessage().AGENTS_COLUMN_SERVICE,+ width: "minmax(10rem, 2fr)",cell: ({ row }) => (- <div className="flex items-center gap-2 overflow-hidden">+ <div className="flex items-center gap-2 min-w-0"><Linkhref={`/agents/${row.agent_key}?from=services`}className="font-medium text-stone-900 dark:text-stone-100 hover:underline truncate"+ title={row.service_name}onClick={(e) => e.stopPropagation()}>{row.service_name}@@ -434,17 +436,13 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {{row.cluster_id}</span>)}- {row.environment && row.environment !== "default" && (- <span className="text-[10px] px-1.5 py-0.5 rounded bg-stone-100 text-stone-600 dark:bg-stone-800 dark:text-stone-400 border border-stone-200 dark:border-stone-700 flex-shrink-0">- {row.environment}- </span>- )}</div>),enableHiding: false,},system: {header: () => getMessage().AGENTS_COLUMN_SYSTEM,+ width: "minmax(4.5rem, 0.6fr)",cell: ({ row }) => {if (row.isSdkOnly) {return (@@ -459,7 +457,7 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {: getMessage().AGENTS_SYSTEM_LINUX;return (<div- className="flex items-center text-stone-600 dark:text-stone-400"+ className="flex items-center justify-center text-stone-600 dark:text-stone-400"title={title}>{row.mode === "kubernetes" ? (@@ -474,7 +472,8 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {},},runtime: {- header: () => "Runtime",+ header: () => getMessage().AGENTS_COLUMN_RUNTIME,+ width: "minmax(5rem, 0.6fr)",cell: ({ row }) => {const title = runtimeTitle(row.sdk_language);if (!title) return <StaticDash />;@@ -490,8 +489,9 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {},providers: {header: () => getMessage().AGENTS_COLUMN_PROVIDERS,+ width: "minmax(5.5rem, 0.7fr)",cell: ({ row }) => (- <div className="flex items-center gap-2">+ <div className="flex items-center justify-center gap-1.5">{row.providers && row.providers.length > 0 ? (row.providers.map((p) => (<span key={p} title={p}>@@ -506,8 +506,9 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {},lastSeen: {header: () => getMessage().AGENTS_COLUMN_LAST_SEEN,+ width: "minmax(9rem, 1fr)",cell: ({ row }) => (- <span className="text-xs truncate">+ <span className="text-xs whitespace-nowrap" title={formatBrowserDateTime(row.last_seen)}>{formatBrowserDateTime(row.last_seen)}</span>),@@ -518,6 +519,7 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {{getMessage().AGENTS_COLUMN_ACTIONS}</div>),+ width: "minmax(5rem, 0.7fr)",cell: ({ row, extraFunctions }) => (<ActionsCell service={row} onRefresh={extraFunctions.onRefresh} />),@@ -525,12 +527,14 @@ const columns: Columns<ServiceColumnKey, EnrichedAgent> = {},aiObservability: {header: () => getMessage().AGENTS_COLUMN_LLM_OBSERVABILITY,+ width: "minmax(9rem, 1fr)",cell: ({ row, extraFunctions }) => (<AIObservabilityCell service={row} onRefresh={extraFunctions.onRefresh} />),},agentObservability: {header: () => getMessage().AGENTS_COLUMN_AGENT_OBSERVABILITY,+ width: "minmax(10rem, 1fr)",cell: ({ row, extraFunctions }) => (<AgentObservabilityCell service={row} onRefresh={extraFunctions.onRefresh} />),diff --git a/src/client/src/app/(playground)/costs/page.tsx b/src/client/src/app/(playground)/costs/page.tsxnew file mode 100644index 00000000..f37fef56--- /dev/null+++ b/src/client/src/app/(playground)/costs/page.tsx@@ -0,0 +1,127 @@+"use client";++import { useCallback, useEffect, useState } from "react";+import { useRouter, useSearchParams } from "next/navigation";+import { CircleDollarSign } from "lucide-react";+import getMessage from "@/constants/messages";+import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import Filter from "@/components/(playground)/filter";+import CostsAnalytics from "@/components/(playground)/costs/costs-analytics";+import CostsConfiguration from "@/components/(playground)/costs/costs-configuration";+import ManageModelsSection from "@/components/(playground)/costs/manage-models-section";++type Tab = "analytics" | "models" | "configuration";++const COSTS_HEADER_TONE =+ "border-emerald-200 bg-emerald-50 text-emerald-700 dark:border-emerald-900/70 dark:bg-emerald-950/40 dark:text-emerald-300";++const COSTS_PILL_TAB_ACTIVE =+ "border-emerald-300 bg-emerald-50 text-emerald-800 dark:border-emerald-800 dark:bg-emerald-950/40 dark:text-emerald-200";++function coerceTab(value: string | null): Tab {+ if (value === "configuration") return "configuration";+ if (value === "models") return "models";+ return "analytics";+}++function CostsPillTab({+ active,+ label,+ onClick,+}: {+ active: boolean;+ label: string;+ onClick: () => void;+}) {+ return (+ <button+ type="button"+ onClick={onClick}+ aria-pressed={active}+ className={`inline-flex items-center gap-2 rounded-md border px-2.5 py-1.5 text-xs font-medium transition ${+ active+ ? COSTS_PILL_TAB_ACTIVE+ : "border-stone-200 bg-stone-50 text-stone-600 hover:bg-stone-100 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300 dark:hover:bg-stone-800"+ }`}+ >+ <span>{label}</span>+ </button>+ );+}++export default function CostsPage() {+ const m = getMessage();+ const router = useRouter();+ const searchParams = useSearchParams();+ const [activeTab, setActiveTabState] = useState<Tab>(() =>+ coerceTab(searchParams.get("tab"))+ );++ useEffect(() => {+ setActiveTabState(coerceTab(searchParams.get("tab")));+ }, [searchParams]);++ const setActiveTab = useCallback(+ (tab: Tab) => {+ setActiveTabState(tab);+ const params = new URLSearchParams(window.location.search);+ if (tab === "analytics") {+ params.delete("tab");+ } else {+ params.set("tab", tab);+ }+ const qs = params.toString();+ router.replace(`/costs${qs ? `?${qs}` : ""}`, { scroll: false });+ },+ [router]+ );++ return (+ <div className="flex h-full w-full flex-col overflow-hidden">+ <FeaturePageHeader+ eyebrow={m.SIDEBAR_MONITOR}+ title={m.COSTS_TITLE}+ icon={<CircleDollarSign className="h-4 w-4" />}+ tone={COSTS_HEADER_TONE}+ actions={+ <div className="flex flex-wrap items-center gap-2">+ <CostsPillTab+ active={activeTab === "analytics"}+ label={m.COSTS_TAB_ANALYTICS}+ onClick={() => setActiveTab("analytics")}+ />+ <CostsPillTab+ active={activeTab === "models"}+ label={m.COSTS_TAB_MODELS}+ onClick={() => setActiveTab("models")}+ />+ <CostsPillTab+ active={activeTab === "configuration"}+ label={m.COSTS_TAB_CONFIGURATION}+ onClick={() => setActiveTab("configuration")}+ />+ </div>+ }+ />++ {activeTab === "analytics" ? (+ <section className="flex min-h-0 flex-1 flex-col overflow-hidden">+ <div className="flex items-center justify-end border-b border-stone-200 px-4 py-2 dark:border-stone-800">+ <Filter />+ </div>+ <div className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-4">+ <CostsAnalytics onConfigure={() => setActiveTab("configuration")} />+ </div>+ </section>+ ) : activeTab === "models" ? (+ <section className="flex min-h-0 flex-1 flex-col overflow-hidden p-4">+ <ManageModelsSection />+ </section>+ ) : (+ <section className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-4">+ <CostsConfiguration />+ </section>+ )}+ </div>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/costs/auto-pricing-runs.tsx b/src/client/src/app/(playground)/dashboard/costs/auto-pricing-runs.tsxnew file mode 100644index 00000000..5c502e5a--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/costs/auto-pricing-runs.tsx@@ -0,0 +1,212 @@+"use client";++import { useCallback, useEffect } from "react";+import { format, parseISO, isValid } from "date-fns";+import {+ Activity,+ BadgeCheck,+ CircleDollarSign,+ ScanSearch,+} from "lucide-react";+import StatCard from "@/components/(playground)/stat-card";+import { Button } from "@/components/ui/button";+import { Card, CardContent } from "@/components/ui/card";+import { Skeleton } from "@/components/ui/skeleton";+import getMessage from "@/constants/messages";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import { getFilterDetails } from "@/selectors/filter";+import { getPingStatus } from "@/selectors/database-config";+import { useRootStore } from "@/store";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import type { PricingCronRunRow } from "@/lib/platform/pricing/cron-analytics";++type PricingCronAnalyticsResponse = {+ data?: Array<Record<string, number>>;+ runs?: PricingCronRunRow[];+ err?: unknown;+};++function formatStartedAt(value: string) {+ try {+ const date = parseISO(value.includes("T") ? value : value.replace(" ", "T"));+ if (!isValid(date)) return value;+ return format(date, "yyyy-MM-dd HH:mm:ss");+ } catch {+ return value;+ }+}++function statusClass(status: string) {+ if (status === "SUCCESS") {+ return "text-emerald-700 dark:text-emerald-300";+ }+ if (status === "PARTIAL_SUCCESS") {+ return "text-amber-700 dark:text-amber-300";+ }+ if (status === "FAILURE") {+ return "text-red-700 dark:text-red-300";+ }+ return "text-stone-600 dark:text-stone-300";+}++export default function AutoPricingRuns({+ onConfigure,+}: {+ onConfigure: () => void;+}) {+ const m = getMessage();+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const { data, fireRequest, isFetched, isLoading } =+ useFetchWrapper<PricingCronAnalyticsResponse>();++ const fetchData = useCallback(() => {+ fireRequest({+ body: JSON.stringify({+ ...getFilterParamsForDashboard(filter),+ limit: 25,+ }),+ requestType: "POST",+ url: "/api/pricing/cron-analytics",+ });+ }, [filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchData();+ }+ }, [filter, fetchData, pingStatus]);++ const loading = isLoading || !isFetched || pingStatus === "pending";+ const runs = data?.runs || [];+ const summary = data?.data?.[0];+ const totalRuns = Number(summary?.total_runs) || 0;+ const summaryUrl = "/api/pricing/cron-analytics";++ return (+ <div className="flex flex-col gap-3">+ <div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4">+ <StatCard+ dataKey="total_runs"+ heading={m.COSTS_STAT_CRON_RUNS}+ icon={Activity}+ url={summaryUrl}+ roundTo={0}+ />+ <StatCard+ dataKey="successful_runs"+ heading={m.COSTS_STAT_CRON_SUCCESS}+ icon={BadgeCheck}+ url={summaryUrl}+ roundTo={0}+ />+ <StatCard+ dataKey="total_updated"+ heading={m.COSTS_STAT_PRICING_APPLIED}+ icon={CircleDollarSign}+ url={summaryUrl}+ roundTo={0}+ />+ <StatCard+ dataKey="total_spans"+ heading={m.COSTS_STAT_SPANS_SCANNED}+ icon={ScanSearch}+ url={summaryUrl}+ roundTo={0}+ />+ </div>++ <Card className="overflow-hidden border-stone-200 shadow-sm dark:border-stone-800">+ <CardContent className="p-0">+ {loading ? (+ <div className="p-4">+ <Skeleton className="h-24 w-full" />+ </div>+ ) : runs.length === 0 ? (+ <div className="flex flex-col items-center justify-center gap-3 px-4 py-8 text-center">+ <p className="text-sm text-stone-500 dark:text-stone-400">+ {m.COSTS_CRON_TABLE_EMPTY}+ </p>+ {totalRuns === 0 ? (+ <>+ <p className="max-w-md text-xs text-stone-400 dark:text-stone-500">+ {m.COSTS_CRON_TABLE_EMPTY_HINT}+ </p>+ <Button size="sm" onClick={onConfigure}>+ {m.COSTS_CRON_TABLE_EMPTY_CTA}+ </Button>+ </>+ ) : null}+ </div>+ ) : (+ <div className="overflow-x-auto">+ <table className="w-full text-sm">+ <thead>+ <tr className="border-y border-stone-200 text-left text-[11px] uppercase tracking-wide text-stone-500 dark:border-stone-800 dark:text-stone-400">+ <th className="px-4 py-2 font-medium">+ {m.COSTS_CRON_TABLE_STARTED}+ </th>+ <th className="px-4 py-2 font-medium">+ {m.COSTS_CRON_TABLE_STATUS}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.COSTS_CRON_TABLE_DURATION}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.COSTS_CRON_TABLE_UPDATED}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.COSTS_CRON_TABLE_SCANNED}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.COSTS_CRON_TABLE_SKIPPED}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.COSTS_CRON_TABLE_FAILED}+ </th>+ </tr>+ </thead>+ <tbody>+ {runs.map((row, index) => (+ <tr+ key={`${row.startedAt}-${index}`}+ className="border-b border-stone-100 hover:bg-stone-50 dark:border-stone-900 dark:hover:bg-stone-900/50"+ >+ <td className="px-4 py-2.5 tabular-nums text-stone-800 dark:text-stone-200">+ {formatStartedAt(row.startedAt)}+ </td>+ <td+ className={`px-4 py-2.5 text-xs font-medium ${statusClass(row.runStatus)}`}+ >+ {row.runStatus}+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums text-stone-800 dark:text-stone-200">+ {row.duration.toLocaleString()}s+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums font-medium text-stone-900 dark:text-stone-100">+ {row.totalUpdated.toLocaleString()}+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums text-stone-800 dark:text-stone-200">+ {row.totalSpans.toLocaleString()}+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums text-stone-800 dark:text-stone-200">+ {row.totalSkipped.toLocaleString()}+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums text-stone-800 dark:text-stone-200">+ {row.totalFailed.toLocaleString()}+ </td>+ </tr>+ ))}+ </tbody>+ </table>+ </div>+ )}+ </CardContent>+ </Card>+ </div>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/costs/cost-per-time.tsx b/src/client/src/app/(playground)/dashboard/costs/cost-per-time.tsxnew file mode 100644index 00000000..8d4bfac3--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/costs/cost-per-time.tsx@@ -0,0 +1,99 @@+"use client";++import { useCallback, useEffect } from "react";+import {+ LineChart,+ Line,+ XAxis,+ YAxis,+ CartesianGrid,+ Tooltip,+ ResponsiveContainer,+} from "recharts";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { useRootStore } from "@/store";+import { getFilterDetails } from "@/selectors/filter";+import { getPingStatus } from "@/selectors/database-config";+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";+import { COLORS } from "../../../../../styles/colors";+import IntermediateState from "@/components/(playground)/intermediate-state";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import getMessage from "@/constants/messages";++export default function CostsPerTime() {+ const m = getMessage();+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const { data, fireRequest, isFetched, isLoading } = useFetchWrapper();++ const fetchData = useCallback(async () => {+ fireRequest({+ body: JSON.stringify(getFilterParamsForDashboard(filter)),+ requestType: "POST",+ url: "/api/metrics/llm/cost/time",+ responseDataKey: "data",+ });+ }, [filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchData();+ }+ }, [filter, fetchData, pingStatus]);++ const chartData = ((data as { request_time?: string; cost?: number }[]) || []).map(+ (row) => ({+ request_time: row.request_time,+ cost: Number(row.cost) || 0,+ })+ );++ const isLoadingData = isLoading || !isFetched || pingStatus === "pending";++ return (+ <Card>+ <CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">+ <CardTitle className="text-sm font-medium text-stone-950 dark:text-stone-100">+ {m.COSTS_CHART_OVER_TIME}+ </CardTitle>+ </CardHeader>+ <CardContent>+ <ResponsiveContainer className="h-40" width="100%" height={160}>+ {isLoadingData ? (+ <IntermediateState type="loading" classNames="h-40" />+ ) : chartData.length === 0 ? (+ <IntermediateState type="nodata" classNames="h-40" />+ ) : (+ <LineChart+ data={chartData}+ margin={{ top: 5, right: 30, left: 20, bottom: 5 }}+ >+ <CartesianGrid strokeDasharray="3 3" />+ <XAxis+ dataKey="request_time"+ className="text-xs stroke-stone-300"+ stroke="currentColor"+ />+ <YAxis+ className="text-xs stroke-stone-300"+ stroke="currentColor"+ domain={[0, "dataMax + 0.01"]}+ />+ <Tooltip labelClassName="dark:text-stone-700" />+ <Line+ type="monotone"+ dataKey="cost"+ stroke={`${COLORS.primary}`}+ activeDot={{ r: 4 }}+ />+ </LineChart>+ )}+ </ResponsiveContainer>+ </CardContent>+ </Card>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/costs/index.tsx b/src/client/src/app/(playground)/dashboard/costs/index.tsxnew file mode 100644index 00000000..771f2f6d--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/costs/index.tsx@@ -0,0 +1,43 @@+"use client";++import CostsNumberStats from "@/app/(playground)/dashboard/costs/number-stats";+import CostsUsageCharts from "@/app/(playground)/dashboard/costs/usage-charts";+import CostsOptimizationCharts from "@/app/(playground)/dashboard/costs/optimization-charts";+import CostsPerTime from "@/app/(playground)/dashboard/costs/cost-per-time";+import AutoPricingRuns from "@/app/(playground)/dashboard/costs/auto-pricing-runs";+import CostsAutoPricingBanner from "@/components/(playground)/costs/costs-auto-pricing-banner";+import getMessage from "@/constants/messages";++export default function CostsDashboard({+ onConfigure,+}: {+ onConfigure: () => void;+}) {+ const m = getMessage();++ return (+ <div className="flex flex-col gap-6 w-full">+ <CostsAutoPricingBanner onConfigure={onConfigure} />+ <section className="flex flex-col gap-3">+ <h2 className="text-sm font-semibold text-stone-900 dark:text-stone-100">+ {m.COSTS_USAGE_SECTION}+ </h2>+ <CostsNumberStats />+ <CostsUsageCharts />+ </section>+ <section className="flex flex-col gap-3">+ <h2 className="text-sm font-semibold text-stone-900 dark:text-stone-100">+ {m.COSTS_AUTO_PRICING_SECTION}+ </h2>+ <AutoPricingRuns onConfigure={onConfigure} />+ </section>+ <section className="flex flex-col gap-3">+ <h2 className="text-sm font-semibold text-stone-900 dark:text-stone-100">+ {m.COSTS_OPTIMIZATION_SECTION}+ </h2>+ <CostsPerTime />+ <CostsOptimizationCharts />+ </section>+ </div>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/costs/number-stats.tsx b/src/client/src/app/(playground)/dashboard/costs/number-stats.tsxnew file mode 100644index 00000000..3a53d754--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/costs/number-stats.tsx@@ -0,0 +1,90 @@+"use client";++import StatCard from "@/components/(playground)/stat-card";+import getMessage from "@/constants/messages";+import {+ Banknote,+ Bot,+ CircleDollarSign,+ FlaskConical,+ Layers,+ MessageSquare,+ MonitorPlay,+ RadioTower,+} from "lucide-react";++export default function CostsNumberStats() {+ const m = getMessage();+ const summaryUrl = "/api/metrics/cost/summary";++ return (+ <div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4 xl:grid-cols-4">+ <StatCard+ dataKey="total_platform_cost"+ heading={m.COSTS_STAT_TOTAL_PLATFORM}+ icon={CircleDollarSign}+ url={summaryUrl}+ textPrefix="$"+ roundTo={7}+ />+ <StatCard+ dataKey="llm_cost"+ heading={m.COSTS_STAT_LLM}+ icon={Layers}+ url={summaryUrl}+ textPrefix="$"+ roundTo={7}+ />+ <StatCard+ dataKey="coding_agents_cost"+ heading={m.COSTS_STAT_CODING_AGENTS}+ icon={Bot}+ url={summaryUrl}+ textPrefix="$"+ roundTo={7}+ />+ <StatCard+ dataKey="otter_cost"+ heading={m.COSTS_STAT_OTTER}+ icon={MessageSquare}+ url={summaryUrl}+ textPrefix="$"+ roundTo={7}+ />+ <StatCard+ dataKey="evaluations_cost"+ heading={m.COSTS_STAT_EVALUATIONS}+ icon={FlaskConical}+ url={summaryUrl}+ textPrefix="$"+ roundTo={7}+ />+ <StatCard+ dataKey="openground_cost"+ heading={m.COSTS_STAT_OPENGROUND}+ icon={MonitorPlay}+ url={summaryUrl}+ textPrefix="$"+ roundTo={7}+ />+ <StatCard+ dataKey="total_requests"+ extraParams={{+ operationType: "llm",+ }}+ heading={m.COSTS_STAT_TOTAL_REQUESTS}+ icon={RadioTower}+ url="/api/metrics/request/total"+ roundTo={0}+ />+ <StatCard+ dataKey="average_usage_cost"+ heading={m.COSTS_STAT_AVG_REQUEST}+ icon={Banknote}+ url="/api/metrics/llm/cost/request/average"+ textPrefix="$"+ roundTo={7}+ />+ </div>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/costs/optimization-charts.tsx b/src/client/src/app/(playground)/dashboard/costs/optimization-charts.tsxnew file mode 100644index 00000000..6deb0b4a--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/costs/optimization-charts.tsx@@ -0,0 +1,31 @@+"use client";++import PieChartCard from "@/components/(playground)/pie-chart-card";+import getMessage from "@/constants/messages";++export default function CostsOptimizationCharts() {+ const m = getMessage();++ return (+ <div className="grid gap-4 md:grid-cols-2 lg:grid-cols-3">+ <PieChartCard+ categoryKey="cost"+ heading={m.COSTS_CHART_BY_PROVIDER}+ indexKey="provider"+ url="/api/metrics/llm/cost/provider"+ />+ <PieChartCard+ categoryKey="cost"+ heading={m.COSTS_CHART_BY_MODEL}+ indexKey="model"+ url="/api/metrics/llm/cost/model"+ />+ <PieChartCard+ categoryKey="cost"+ heading={m.COSTS_CHART_OPENGROUND_BY_PROVIDER}+ indexKey="provider"+ url="/api/metrics/openground/cost/provider"+ />+ </div>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/costs/usage-charts.tsx b/src/client/src/app/(playground)/dashboard/costs/usage-charts.tsxnew file mode 100644index 00000000..1752302b--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/costs/usage-charts.tsx@@ -0,0 +1,25 @@+"use client";++import PieChartCard from "@/components/(playground)/pie-chart-card";+import getMessage from "@/constants/messages";++export default function CostsUsageCharts() {+ const m = getMessage();++ return (+ <div className="grid gap-4 md:grid-cols-2 lg:grid-cols-2">+ <PieChartCard+ categoryKey="cost"+ heading={m.COSTS_CHART_BY_ENVIRONMENT}+ indexKey="environment"+ url="/api/metrics/llm/cost/environment"+ />+ <PieChartCard+ categoryKey="cost"+ heading={m.COSTS_CHART_BY_APPLICATION}+ indexKey="applicationName"+ url="/api/metrics/llm/cost/application"+ />+ </div>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/evaluations/index.tsx b/src/client/src/app/(playground)/dashboard/evaluations/index.tsxnew file mode 100644index 00000000..c2bdb569--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/evaluations/index.tsx@@ -0,0 +1,19 @@+"use client";++import EvaluationNumberStats from "./number-stats";+import EvaluationMetricsPerTime from "./metrics-per-time";++/**+ * Evaluations analytics dashboard — composed like LLMDashboard on the+ * agents detail Dashboard tab (StatCards + time-series chart).+ */+export default function EvaluationsDashboard() {+ return (+ <>+ <EvaluationNumberStats />+ <div className="flex flex-col gap-4">+ <EvaluationMetricsPerTime />+ </div>+ </>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/evaluations/metrics-per-time.tsx b/src/client/src/app/(playground)/dashboard/evaluations/metrics-per-time.tsxnew file mode 100644index 00000000..8908dad3--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/evaluations/metrics-per-time.tsx@@ -0,0 +1,134 @@+"use client";++import { useCallback, useEffect, useState } from "react";+import {+ LineChart,+ Line,+ XAxis,+ YAxis,+ CartesianGrid,+ Tooltip,+ ResponsiveContainer,+} from "recharts";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { useRootStore } from "@/store";+import { getFilterDetails } from "@/selectors/filter";+import { getPingStatus } from "@/selectors/database-config";+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";+import { COLORS } from "../../../../../styles/colors";+import IntermediateState from "@/components/(playground)/intermediate-state";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import getMessage from "@/constants/messages";+import type { EvaluationAnalyticsResponse } from "@/types/evaluation";++type ChartMode = "passRate" | "executions";++/**+ * Pass rate / executions over time — same Card + Recharts pattern as+ * dashboard/llm/requests-per-time.tsx used on the agents Dashboard tab.+ */+export default function EvaluationMetricsPerTime() {+ const m = getMessage();+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const [chartMode, setChartMode] = useState<ChartMode>("passRate");+ const { data, fireRequest, isFetched, isLoading } =+ useFetchWrapper<EvaluationAnalyticsResponse>();++ const fetchData = useCallback(async () => {+ fireRequest({+ body: JSON.stringify(getFilterParamsForDashboard(filter)),+ requestType: "POST",+ url: "/api/evaluation/analytics",+ });+ }, [filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchData();+ }+ }, [filter, fetchData, pingStatus]);++ const chartData = ((data?.timeseries as any[]) || []).map((row) => ({+ request_time: row.timestamp,+ passRate: Number(row.passRate) || 0,+ executions: Number(row.executions) || 0,+ }));++ const isLoadingData = isLoading || !isFetched || pingStatus === "pending";++ return (+ <Card>+ <CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">+ <div className="flex items-center gap-2">+ <button+ type="button"+ onClick={() => setChartMode("passRate")}+ className={`rounded-md border px-2.5 py-1.5 text-xs font-medium transition ${+ chartMode === "passRate"+ ? "border-stone-300 bg-stone-100 text-stone-900 dark:border-stone-700 dark:bg-stone-800 dark:text-stone-100"+ : "border-stone-200 bg-stone-50 text-stone-600 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300"+ }`}+ >+ {m.EVALUATION_CHART_PASS_RATE}+ </button>+ <button+ type="button"+ onClick={() => setChartMode("executions")}+ className={`rounded-md border px-2.5 py-1.5 text-xs font-medium transition ${+ chartMode === "executions"+ ? "border-stone-300 bg-stone-100 text-stone-900 dark:border-stone-700 dark:bg-stone-800 dark:text-stone-100"+ : "border-stone-200 bg-stone-50 text-stone-600 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300"+ }`}+ >+ {m.EVALUATION_CHART_EXECUTIONS}+ </button>+ </div>+ <CardTitle className="text-sm font-medium text-stone-950 dark:text-stone-100">+ {chartMode === "passRate"+ ? m.EVALUATION_CHART_PASS_RATE+ : m.EVALUATION_CHART_EXECUTIONS}+ </CardTitle>+ </CardHeader>+ <CardContent>+ <ResponsiveContainer className="h-40" width="100%" height={160}>+ {isLoadingData ? (+ <IntermediateState type="loading" classNames="h-40" />+ ) : chartData.length === 0 ? (+ <IntermediateState type="nodata" classNames="h-40" />+ ) : (+ <LineChart+ data={chartData}+ margin={{ top: 5, right: 30, left: 20, bottom: 5 }}+ >+ <CartesianGrid strokeDasharray="3 3" />+ <XAxis+ dataKey="request_time"+ className="text-xs stroke-stone-300"+ stroke="currentColor"+ />+ <YAxis+ className="text-xs stroke-stone-300"+ stroke="currentColor"+ domain={+ chartMode === "passRate" ? [0, 100] : [0, "dataMax + 15"]+ }+ />+ <Tooltip labelClassName="dark:text-stone-700" />+ <Line+ type="monotone"+ dataKey={chartMode === "passRate" ? "passRate" : "executions"}+ stroke={`${COLORS.primary}`}+ activeDot={{ r: 4 }}+ />+ </LineChart>+ )}+ </ResponsiveContainer>+ </CardContent>+ </Card>+ );+}diff --git a/src/client/src/app/(playground)/dashboard/evaluations/number-stats.tsx b/src/client/src/app/(playground)/dashboard/evaluations/number-stats.tsxnew file mode 100644index 00000000..03118bce--- /dev/null+++ b/src/client/src/app/(playground)/dashboard/evaluations/number-stats.tsx@@ -0,0 +1,86 @@+"use client";++import StatCard from "@/components/(playground)/stat-card";+import getMessage from "@/constants/messages";+import {+ Activity,+ BadgeCheck,+ Banknote,+ CheckCircle2,+ Layers,+ ListChecks,+ XCircle,+ Zap,+} from "lucide-react";++/**+ * KPI row for Evaluations analytics — same StatCard components used by+ * the LLM dashboard on the agents detail page. Eight cards for a full grid.+ */+export default function EvaluationNumberStats() {+ const m = getMessage();+ const url = "/api/evaluation/analytics";++ return (+ <div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4">+ <StatCard+ dataKey="evaluations"+ heading={m.EVALUATION_STAT_EVALUATIONS}+ icon={Layers}+ url={url}+ roundTo={0}+ />+ <StatCard+ dataKey="active"+ heading={m.EVALUATION_STAT_ACTIVE}+ icon={BadgeCheck}+ url={url}+ roundTo={0}+ />+ <StatCard+ dataKey="traces_evaluated"+ heading={m.EVALUATION_STAT_TRACES_EVALUATED}+ icon={ListChecks}+ url={url}+ roundTo={0}+ />+ <StatCard+ dataKey="executions"+ heading={m.EVALUATION_STAT_EXECUTIONS}+ icon={Activity}+ url={url}+ roundTo={0}+ />+ <StatCard+ dataKey="auto_executions"+ heading={m.EVALUATION_STAT_AUTO_EXECUTIONS}+ icon={Zap}+ url={url}+ roundTo={0}+ />+ <StatCard+ dataKey="total_cost"+ heading={m.EVALUATION_STAT_TOTAL_COST}+ icon={Banknote}+ url={url}+ textPrefix="$"+ roundTo={4}+ />+ <StatCard+ dataKey="avg_pass_rate"+ heading={m.EVALUATION_STAT_AVG_PASS_RATE}+ icon={CheckCircle2}+ url={url}+ textSuffix="%"+ roundTo={0}+ />+ <StatCard+ dataKey="failed_scores"+ heading={m.EVALUATION_STAT_FAILED_SCORES}+ icon={XCircle}+ url={url}+ roundTo={0}+ />+ </div>+ );+}diff --git a/src/client/src/app/(playground)/evaluations/evaluators/[id]/page.tsx b/src/client/src/app/(playground)/evaluations/evaluators/[id]/page.tsxnew file mode 100644index 00000000..a5f52598--- /dev/null+++ b/src/client/src/app/(playground)/evaluations/evaluators/[id]/page.tsx@@ -0,0 +1,423 @@+"use client";++import { Fragment, useCallback, useEffect, useMemo, useRef, useState } from "react";+import Link from "next/link";+import { useParams } from "next/navigation";+import { round } from "lodash";+import {+ Activity,+ ArrowLeft,+ Banknote,+ CheckCircle2,+ ChevronDown,+ ChevronRight,+ Settings2,+ XCircle,+} from "lucide-react";+import {+ CartesianGrid,+ Line,+ LineChart,+ ResponsiveContainer,+ Tooltip,+ XAxis,+ YAxis,+} from "recharts";+import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import Filter from "@/components/(playground)/filter";+import IntermediateState from "@/components/(playground)/intermediate-state";+import StatCard from "@/components/(playground)/stat-card";+import { Badge } from "@/components/ui/badge";+import { Button } from "@/components/ui/button";+import { Card, CardContent, CardHeader } from "@/components/ui/card";+import getMessage from "@/constants/messages";+import { COLORS } from "../../../../../../styles/colors";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import { getPingStatus } from "@/selectors/database-config";+import { getFilterDetails } from "@/selectors/filter";+import { useRootStore } from "@/store";+import type { EvaluationEvaluatorAnalyticsResponse } from "@/types/evaluation";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";++type ChartMode = "passRate" | "executions";++const HEADER_TONE =+ "border-orange-200 bg-orange-50 text-orange-700 dark:border-orange-900/70 dark:bg-orange-950/40 dark:text-orange-300";++function severityScoreClass(score: number) {+ // Score is severity (0 = clean / pass, higher = worse) — inverse of pass-rate coloring.+ if (score <= 0.25) return "text-emerald-600 dark:text-emerald-400";+ if (score <= 0.5) return "text-lime-600 dark:text-lime-400";+ return "text-amber-600 dark:text-amber-400";+}++export default function EvaluationEvaluatorDetailPage() {+ const m = getMessage();+ const params = useParams();+ const evaluatorId = decodeURIComponent(String(params.id || ""));+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const [chartMode, setChartMode] = useState<ChartMode>("passRate");+ const [expandedKey, setExpandedKey] = useState<string | null>(null);+ const expandedRowRef = useRef<HTMLTableRowElement | null>(null);++ const { data, isLoading, isFetched, fireRequest, error } =+ useFetchWrapper<EvaluationEvaluatorAnalyticsResponse>();++ const fetchData = useCallback(() => {+ if (!evaluatorId) return;+ fireRequest({+ body: JSON.stringify(getFilterParamsForDashboard(filter)),+ requestType: "POST",+ url: `/api/evaluation/analytics/${encodeURIComponent(evaluatorId)}`,+ });+ }, [evaluatorId, filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchData();+ }+ }, [filter, fetchData, pingStatus]);++ useEffect(() => {+ if (!expandedKey) return;+ expandedRowRef.current?.scrollIntoView({+ behavior: "smooth",+ block: "nearest",+ });+ }, [expandedKey]);++ const metricsUrl = `/api/evaluation/analytics/${encodeURIComponent(evaluatorId)}`;+ const chartData = useMemo(+ () =>+ ((data?.timeseries as any[]) || []).map((row) => ({+ request_time: row.timestamp,+ passRate: Number(row.passRate) || 0,+ executions: Number(row.executions) || 0,+ })),+ [data?.timeseries]+ );++ const recentResults = data?.recentResults || [];+ const loading = isLoading || !isFetched || pingStatus === "pending";+ const notFound = isFetched && (error || (data && !data.found));+ const backLabel = m.EVALUATION_BACK_TO_ANALYTICS;++ const header = (+ <FeaturePageHeader+ eyebrow={m.FEATURE_EVALS}+ title={data?.evaluator?.label || evaluatorId}+ icon={<Activity className="h-4 w-4" />}+ tone={HEADER_TONE}+ description={data?.evaluator?.description}+ leading={+ <Button+ asChild+ variant="outline"+ size="sm"+ className="h-7 w-7 shrink-0 p-0"+ >+ <Link href="/evaluations" title={backLabel} aria-label={backLabel}>+ <ArrowLeft className="h-3.5 w-3.5" />+ </Link>+ </Button>+ }+ actions={+ <div className="flex flex-wrap items-center gap-2">+ {data?.evaluator ? (+ <Badge+ variant={data.evaluator.enabled ? "default" : "secondary"}+ className="h-7"+ >+ {data.evaluator.enabled+ ? m.EVALUATION_DETAIL_ENABLED+ : m.EVALUATION_DETAIL_DISABLED}+ </Badge>+ ) : null}+ <Button asChild variant="outline" size="sm" className="h-8">+ <Link+ href={`/evaluations/types/${encodeURIComponent(evaluatorId)}`}+ >+ <Settings2 className="mr-1.5 h-3.5 w-3.5" />+ {m.EVALUATION_DETAIL_CONFIGURE}+ </Link>+ </Button>+ </div>+ }+ />+ );++ if (notFound) {+ return (+ <div className="flex h-full w-full flex-col overflow-hidden">+ {header}+ <div className="flex flex-1 flex-col items-center justify-center gap-3 p-4 text-center">+ <p className="text-base font-medium text-stone-800 dark:text-stone-200">+ {m.EVALUATION_TYPE_NOT_FOUND}+ </p>+ <p className="max-w-md text-sm text-stone-500 dark:text-stone-400">+ {m.EVALUATION_TYPE_NOT_FOUND_DESCRIPTION}+ </p>+ <Button asChild variant="outline" size="sm" className="h-8">+ <Link href="/evaluations">{backLabel}</Link>+ </Button>+ </div>+ </div>+ );+ }++ return (+ <div className="flex h-full w-full flex-col overflow-hidden">+ {header}+ <div className="flex items-center justify-end border-b border-stone-200 px-4 py-2 dark:border-stone-800">+ <Filter />+ </div>+ <div className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-4 pb-24">+ <div className="grid gap-4 md:grid-cols-2 lg:grid-cols-4">+ <StatCard+ dataKey="executions"+ heading={m.EVALUATION_STAT_EXECUTIONS}+ icon={Activity}+ url={metricsUrl}+ roundTo={0}+ />+ <StatCard+ dataKey="avg_pass_rate"+ heading={m.EVALUATION_STAT_AVG_PASS_RATE}+ icon={CheckCircle2}+ url={metricsUrl}+ textSuffix="%"+ roundTo={0}+ />+ <StatCard+ dataKey="failed_scores"+ heading={m.EVALUATION_STAT_FAILED_SCORES}+ icon={XCircle}+ url={metricsUrl}+ roundTo={0}+ />+ <StatCard+ dataKey="total_cost"+ heading={m.EVALUATION_STAT_TOTAL_COST}+ icon={Banknote}+ url={metricsUrl}+ textPrefix="$"+ roundTo={4}+ />+ </div>++ <Card className="border-stone-200 shadow-sm dark:border-stone-800">+ <CardHeader className="flex flex-row items-center justify-between space-y-0 pb-2">+ <div className="flex items-center gap-2">+ <button+ type="button"+ onClick={() => setChartMode("passRate")}+ className={`rounded-md border px-2.5 py-1.5 text-xs font-medium transition ${+ chartMode === "passRate"+ ? "border-stone-300 bg-stone-100 text-stone-900 dark:border-stone-700 dark:bg-stone-800 dark:text-stone-100"+ : "border-stone-200 bg-stone-50 text-stone-600 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300"+ }`}+ >+ {m.EVALUATION_CHART_PASS_RATE}+ </button>+ <button+ type="button"+ onClick={() => setChartMode("executions")}+ className={`rounded-md border px-2.5 py-1.5 text-xs font-medium transition ${+ chartMode === "executions"+ ? "border-stone-300 bg-stone-100 text-stone-900 dark:border-stone-700 dark:bg-stone-800 dark:text-stone-100"+ : "border-stone-200 bg-stone-50 text-stone-600 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300"+ }`}+ >+ {m.EVALUATION_CHART_EXECUTIONS}+ </button>+ </div>+ </CardHeader>+ <CardContent>+ <ResponsiveContainer className="h-48" width="100%" height={192}>+ {loading ? (+ <IntermediateState type="loading" classNames="h-48" />+ ) : chartData.length === 0 ? (+ <IntermediateState type="nodata" classNames="h-48" />+ ) : (+ <LineChart+ data={chartData}+ margin={{ top: 5, right: 30, left: 20, bottom: 5 }}+ >+ <CartesianGrid strokeDasharray="3 3" />+ <XAxis+ dataKey="request_time"+ className="text-xs stroke-stone-300"+ stroke="currentColor"+ />+ <YAxis+ className="text-xs stroke-stone-300"+ stroke="currentColor"+ domain={+ chartMode === "passRate" ? [0, 100] : [0, "dataMax + 15"]+ }+ />+ <Tooltip labelClassName="dark:text-stone-700" />+ <Line+ type="monotone"+ dataKey={+ chartMode === "passRate" ? "passRate" : "executions"+ }+ stroke={`${COLORS.primary}`}+ activeDot={{ r: 4 }}+ />+ </LineChart>+ )}+ </ResponsiveContainer>+ </CardContent>+ </Card>++ <Card className="overflow-hidden border-stone-200 shadow-sm dark:border-stone-800">+ <CardContent className="p-0">+ {loading ? (+ <div className="p-4">+ <IntermediateState type="loading" classNames="h-24" />+ </div>+ ) : recentResults.length === 0 ? (+ <p className="px-4 py-8 text-center text-sm text-stone-500 dark:text-stone-400">+ {m.EVALUATION_DETAIL_NO_RESULTS}+ </p>+ ) : (+ <div className="overflow-x-auto">+ <table className="w-full text-sm">+ <thead>+ <tr className="border-y border-stone-200 text-left text-[11px] uppercase tracking-wide text-stone-500 dark:border-stone-800 dark:text-stone-400">+ <th className="w-8 px-3 py-2 font-medium" />+ <th className="px-4 py-2 font-medium">+ {m.OBSERVABILITY_TIME}+ </th>+ <th className="px-4 py-2 font-medium">+ {m.EVALUATION_CLASSIFICATION}+ </th>+ <th className="px-4 py-2 font-medium">+ {m.EVALUATION_VERDICT}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.EVALUATION_SCORE}+ </th>+ <th className="px-4 py-2 font-medium">+ {m.EVALUATION_SOURCE}+ </th>+ <th className="px-4 py-2 font-medium">+ {m.OBSERVABILITY_SPAN_ID}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.COST}+ </th>+ </tr>+ </thead>+ <tbody>+ {recentResults.map((result, index) => {+ const rowKey = `${result.id}-${result.spanId}-${result.createdAt}-${index}`;+ const isOpen = expandedKey === rowKey;+ const passed = result.verdict !== "yes";+ return (+ <Fragment key={rowKey}>+ <tr className="border-b border-stone-100 hover:bg-stone-50 dark:border-stone-900 dark:hover:bg-stone-900/50">+ <td className="px-3 py-2.5">+ <button+ type="button"+ className="inline-flex h-6 w-6 items-center justify-center rounded text-stone-400 hover:bg-stone-100 hover:text-stone-600 dark:hover:bg-stone-800 dark:hover:text-stone-300"+ aria-expanded={isOpen}+ aria-label={m.EVALUATION_EXPLANATION}+ onClick={() =>+ setExpandedKey(isOpen ? null : rowKey)+ }+ >+ {isOpen ? (+ <ChevronDown className="h-3.5 w-3.5" />+ ) : (+ <ChevronRight className="h-3.5 w-3.5" />+ )}+ </button>+ </td>+ <td className="whitespace-nowrap px-4 py-2.5 tabular-nums text-stone-600 dark:text-stone-300">+ {result.createdAt}+ </td>+ <td className="max-w-[14rem] truncate px-4 py-2.5 font-medium text-stone-900 dark:text-stone-100">+ {result.classification || "—"}+ </td>+ <td className="px-4 py-2.5">+ <span+ className={`font-medium ${+ passed+ ? "text-emerald-600 dark:text-emerald-400"+ : "text-amber-600 dark:text-amber-400"+ }`}+ >+ {passed+ ? m.EVALUATION_DETAIL_PASS+ : m.EVALUATION_DETAIL_FAIL}+ </span>+ </td>+ <td+ className={`px-4 py-2.5 text-right tabular-nums font-medium ${severityScoreClass(+ Number(result.score) || 0+ )}`}+ >+ {round(result.score, 2)}+ </td>+ <td className="px-4 py-2.5 text-stone-600 dark:text-stone-300">+ {result.source || "—"}+ </td>+ <td className="px-4 py-2.5 font-mono text-xs">+ {result.spanId ? (+ <Link+ href={`/telemetry?traceId=${encodeURIComponent(result.spanId)}`}+ className="text-primary hover:underline"+ >+ {result.spanId.slice(0, 12)}+ {result.spanId.length > 12 ? "…" : ""}+ </Link>+ ) : (+ "—"+ )}+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums text-stone-600 dark:text-stone-300">+ {result.cost+ ? `$${round(result.cost, 6)}`+ : "—"}+ </td>+ </tr>+ {isOpen ? (+ <tr+ ref={expandedRowRef}+ className="border-b border-stone-100 bg-stone-50 dark:border-stone-900 dark:bg-stone-900/50"+ >+ <td+ colSpan={8}+ className="px-4 py-3 text-sm text-stone-600 dark:text-stone-300"+ >+ <p className="mb-1 text-[11px] uppercase tracking-wide text-stone-500 dark:text-stone-400">+ {m.EVALUATION_EXPLANATION}+ </p>+ <p className="whitespace-pre-wrap break-words">+ {result.explanation?.trim()+ ? result.explanation+ : "—"}+ </p>+ </td>+ </tr>+ ) : null}+ </Fragment>+ );+ })}+ </tbody>+ </table>+ </div>+ )}+ </CardContent>+ </Card>+ </div>+ </div>+ );+}diff --git a/src/client/src/app/(playground)/evaluations/page.tsx b/src/client/src/app/(playground)/evaluations/page.tsxindex d89dc4a4..71465fc0 100644--- a/src/client/src/app/(playground)/evaluations/page.tsx+++ b/src/client/src/app/(playground)/evaluations/page.tsx@@ -1,18 +1,136 @@"use client";-import { useEffect } from "react";-import { useRouter } from "next/navigation";+import { useCallback, useEffect, useState } from "react";+import { useRouter, useSearchParams } from "next/navigation";import { usePostHog } from "posthog-js/react";+import { MonitorCog } from "lucide-react";import { CLIENT_EVENTS } from "@/constants/events";+import getMessage from "@/constants/messages";+import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import Filter from "@/components/(playground)/filter";+import EvaluationAnalytics from "@/components/(playground)/evaluations/evaluation-analytics";+import EvaluationConfiguration from "@/components/(playground)/evaluations/evaluation-configuration";+import EvaluationTypesSection from "@/components/(playground)/evaluations/evaluation-types-section";++type Tab = "analytics" | "evaluators" | "configuration";++const EVALUATION_HEADER_TONE =+ "border-orange-200 bg-orange-50 text-orange-700 dark:border-orange-900/70 dark:bg-orange-950/40 dark:text-orange-300";++const EVALUATION_PILL_TAB_ACTIVE =+ "border-orange-300 bg-orange-50 text-orange-800 dark:border-orange-800 dark:bg-orange-950/40 dark:text-orange-200";++function coerceTab(value: string | null): Tab {+ if (value === "configuration") return "configuration";+ if (value === "evaluators") return "evaluators";+ return "analytics";+}++function EvaluationPillTab({+ active,+ label,+ onClick,+}: {+ active: boolean;+ label: string;+ onClick: () => void;+}) {+ return (+ <button+ type="button"+ onClick={onClick}+ aria-pressed={active}+ className={`inline-flex items-center gap-2 rounded-md border px-2.5 py-1.5 text-xs font-medium transition ${+ active+ ? EVALUATION_PILL_TAB_ACTIVE+ : "border-stone-200 bg-stone-50 text-stone-600 hover:bg-stone-100 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300 dark:hover:bg-stone-800"+ }`}+ >+ <span>{label}</span>+ </button>+ );+}export default function EvaluationsPage() {+ const m = getMessage();const posthog = usePostHog();const router = useRouter();+ const searchParams = useSearchParams();+ const [activeTab, setActiveTabState] = useState<Tab>(() =>+ coerceTab(searchParams.get("tab"))+ );useEffect(() => {posthog?.capture(CLIENT_EVENTS.EVALUATIONS_PAGE_VISITED);- router.replace("/evaluations/settings");- }, []);+ }, [posthog]);++ useEffect(() => {+ setActiveTabState(coerceTab(searchParams.get("tab")));+ }, [searchParams]);++ const setActiveTab = useCallback(+ (tab: Tab) => {+ setActiveTabState(tab);+ const params = new URLSearchParams(window.location.search);+ if (tab === "analytics") {+ params.delete("tab");+ } else {+ params.set("tab", tab);+ }+ const qs = params.toString();+ router.replace(`/evaluations${qs ? `?${qs}` : ""}`, { scroll: false });+ },+ [router]+ );++ return (+ <div className="flex h-full w-full flex-col overflow-hidden">+ <FeaturePageHeader+ eyebrow={m.SIDEBAR_MONITOR}+ title={m.EVALUATION_ENGINE_TITLE}+ icon={<MonitorCog className="h-4 w-4" />}+ tone={EVALUATION_HEADER_TONE}+ actions={+ <div className="flex flex-wrap items-center gap-2">+ <EvaluationPillTab+ active={activeTab === "analytics"}+ label={m.EVALUATION_TAB_ANALYTICS}+ onClick={() => setActiveTab("analytics")}+ />+ <EvaluationPillTab+ active={activeTab === "evaluators"}+ label={m.EVALUATION_TAB_EVALUATORS}+ onClick={() => setActiveTab("evaluators")}+ />+ <EvaluationPillTab+ active={activeTab === "configuration"}+ label={m.EVALUATION_TAB_CONFIGURATION}+ onClick={() => setActiveTab("configuration")}+ />+ </div>+ }+ />- return null;+ {activeTab === "analytics" ? (+ <section className="flex min-h-0 flex-1 flex-col overflow-hidden">+ <div className="flex items-center justify-end border-b border-stone-200 px-4 py-2 dark:border-stone-800">+ <Filter />+ </div>+ <div className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-4">+ <EvaluationAnalytics+ onConfigure={() => setActiveTab("configuration")}+ />+ </div>+ </section>+ ) : activeTab === "evaluators" ? (+ <section className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-4">+ <EvaluationTypesSection />+ </section>+ ) : (+ <section className="flex min-h-0 flex-1 flex-col gap-4 overflow-y-auto p-4">+ <EvaluationConfiguration />+ </section>+ )}+ </div>+ );}diff --git a/src/client/src/app/(playground)/evaluations/settings/page.tsx b/src/client/src/app/(playground)/evaluations/settings/page.tsxindex f8451cfa..a75b1545 100644--- a/src/client/src/app/(playground)/evaluations/settings/page.tsx+++ b/src/client/src/app/(playground)/evaluations/settings/page.tsx@@ -1,453 +1,14 @@"use client";-import SecretForm from "@/components/(playground)/vault/form";-import { Button } from "@/components/ui/button";-import { EVALUATION_ENGINES } from "@/constants/evaluation-engines";-import { CLIENT_EVENTS } from "@/constants/events";-import getMessage from "@/constants/messages";-import { EvaluationConfig } from "@/types/evaluation";-import useFetchWrapper from "@/utils/hooks/useFetchWrapper";-import { usePostHog } from "posthog-js/react";-import { useEffect, useMemo, useState } from "react";-import { toast } from "sonner";-import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";-import { Label } from "@/components/ui/label";-import { Switch } from "@/components/ui/switch";-import {- Select,- SelectContent,- SelectItem,- SelectTrigger,- SelectValue,-} from "@/components/ui/select";-import { Input } from "@/components/ui/input";-import Link from "next/link";-import { Zap, Settings2, Clock, Play, Info, SettingsIcon, Layers } from "lucide-react";-import { ProviderMetadata, ModelMetadata } from "@/types/openground";-import { Badge } from "@/components/ui/badge";-import { evalSampleRateToPercent } from "@/constants/evaluation-sampling";-import FeaturePageHeader from "@/components/(playground)/feature-page-header";--const EvaluationVaultCreate = ({- successCallback,-}: {- successCallback: () => void;-}) => (- <div className="flex flex-wrap items-center gap-1 w-full text-stone-600 dark:text-stone-400 text-sm mt-1">- {getMessage().EVALUATION_VAULT_KEY_NOT_FOUND}- <SecretForm successCallback={successCallback}>- <Button- variant="ghost"- className="py-0 h-auto px-0 text-xs text-primary hover:bg-transparent"- >- {getMessage().EVALUATION_CREATE_NEW}- </Button>- </SecretForm>- </div>-);--const EVALUATION_TOAST_ID = "evaluation-config";+import { useEffect } from "react";+import { useRouter } from "next/navigation";export default function EvaluationSettingsPage() {- const [provider, setProvider] = useState("");- const [model, setModel] = useState("");- // Engine is always "vercel" — kept in state for saving to meta (future-proofing)- const [engine] = useState<string>(EVALUATION_ENGINES[0].id);- const [autoEvaluation, setAutoEvaluation] = useState(false);- const [recurringTime, setRecurringTime] = useState("");- const [sampleRatePercent, setSampleRatePercent] = useState(- evalSampleRateToPercent(undefined)- );- const [vaultId, setVaultId] = useState("");- const [vaultKeys, setVaultKeys] = useState<{ label: string; value: string }[]>(- []- );-- const {- fireRequest: getConfig,- data: config,- isLoading: isLoadingConfig,- } = useFetchWrapper<EvaluationConfig>();- const { fireRequest: saveConfig, isLoading: isSaving } = useFetchWrapper();- const { fireRequest: getVaultKeys } = useFetchWrapper();- const { data: providers, fireRequest: fetchProviders } =- useFetchWrapper<ProviderMetadata[]>();- const posthog = usePostHog();-- useEffect(() => {- getConfig({- requestType: "GET",- url: "/api/evaluation/config",- responseDataKey: "data",- });- getVaultKeys({- requestType: "POST",- url: "/api/vault/get",- responseDataKey: "data",- body: JSON.stringify({}),- successCb: (res: any) => {- setVaultKeys(- res?.length ? res.map((k: any) => ({ label: k.key, value: k.id })) : []- );- },- });- fetchProviders({- requestType: "GET",- url: "/api/openground/providers",- });- }, []);+ const router = useRouter();useEffect(() => {- if (config) {- setProvider(config.provider || "");- setModel(config.model || "");- setAutoEvaluation(config.auto || false);- setRecurringTime(config.recurringTime || "");- setVaultId(config.vaultId || "");- const meta = JSON.parse(config.meta || "{}");- setSampleRatePercent(evalSampleRateToPercent(meta.evalSampleRate));- }- }, [config]);-- const selectedProviderMeta = useMemo(- () => (providers || []).find((p) => p.providerId === provider),- [providers, provider]- );- const modelOptions = useMemo(- () => selectedProviderMeta?.supportedModels || [],- [selectedProviderMeta]- );-- const handleSave = () => {- if (!provider || !model || !vaultId) {- toast.error(getMessage().EVALUATION_CONFIG_INVALID, {- id: EVALUATION_TOAST_ID,- });- return;- }-- const parsedSampleRate = Number.parseFloat(sampleRatePercent);- if (autoEvaluation) {- if (- !Number.isFinite(parsedSampleRate) ||- parsedSampleRate < 0 ||- parsedSampleRate > 100- ) {- toast.error(getMessage().EVALUATION_SAMPLE_RATE_PERCENT_INVALID, {- id: EVALUATION_TOAST_ID,- });- return;- }- }-- toast.loading(getMessage().EVALUATION_CONFIG_MODIFYING, {- id: EVALUATION_TOAST_ID,- });-- const meta = JSON.parse(config?.meta || "{}");- meta.engine = engine;- if (autoEvaluation) {- meta.evalSampleRate = parsedSampleRate / 100;- }-- saveConfig({- body: JSON.stringify({- id: config?.id,- provider,- model,- vaultId,- auto: autoEvaluation,- recurringTime: recurringTime || "* * * * *",- meta: JSON.stringify(meta),- }),- requestType: "POST",- url: "/api/evaluation/config",- responseDataKey: "data",- successCb: () => {- toast.success(- config?.id ? getMessage().EVALUATION_UPDATED : getMessage().EVALUATION_CREATED,- { id: EVALUATION_TOAST_ID }- );- getConfig({- requestType: "GET",- url: "/api/evaluation/config",- responseDataKey: "data",- });- posthog?.capture(- config?.id- ? CLIENT_EVENTS.EVALUATION_CONFIG_UPDATED_SUCCESS- : CLIENT_EVENTS.EVALUATION_CONFIG_CREATED_SUCCESS- );- },- failureCb: (err?: string) => {- toast.error(err || getMessage().EVALUATION_CONFIG_UPDATING_FAILED, {- id: EVALUATION_TOAST_ID,- });- },- });- };-- const header = (- <FeaturePageHeader- eyebrow="Configuration"- title={getMessage().EVALUATION_ENGINE_TITLE}- icon={<Settings2 className="h-4 w-4" />}- tone="border-orange-200 bg-orange-50 text-orange-700 dark:border-orange-900/70 dark:bg-orange-950/40 dark:text-orange-300"- actions={- <Button asChild variant="outline" size="sm" className="h-8">- <Link href="/evaluations/types">- <Layers className="mr-1.5 h-3.5 w-3.5" />- Evaluation Types- </Link>- </Button>- }- />- );-- if (isLoadingConfig && !config) {- return (- <div className="flex flex-col h-full w-full overflow-hidden">- {header}- <div className="flex flex-1 w-full p-4">- <div className="animate-pulse grid gap-4 w-full max-w-3xl">- <div className="h-40 bg-stone-100 dark:bg-stone-900" />- </div>- </div>- </div>- );- }-- return (- <div className="flex flex-col h-full w-full overflow-hidden">- {header}- <div className="flex flex-col flex-1 w-full overflow-auto gap-4 p-4">- <Card className="border-blue-200 dark:border-blue-900/50 bg-blue-50/50 dark:bg-blue-950/20">- <CardContent className="pt-6">- <div className="flex gap-3">- <Info className="size-5 text-blue-600 dark:text-blue-500 shrink-0 mt-0.5" />- <div className="text-sm text-stone-600 dark:text-stone-400">- <p className="font-medium text-stone-700 dark:text-stone-300 mb-1">- {getMessage().EVALUATION_MANUAL_AND_AUTO}- </p>- <p>{getMessage().EVALUATION_MANUAL_AND_AUTO_DESCRIPTION}</p>- </div>- </div>- </CardContent>- </Card>- <div className="grid grid-cols-3 gap-4">- <div className="col-span-2 grid gap-4">- {/* Configuration Card */}- <Card className="border-stone-200 dark:border-stone-800 shadow-sm">- <CardHeader className="pb-4">- <CardTitle className="text-base flex items-center gap-2">- <Settings2 className="size-4" />- {getMessage().EVALUATION_CONFIG_SECTION}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">- {getMessage().EVALUATION_ENGINE_DESCRIPTION}- </p>- </CardHeader>- <CardContent className="space-y-4">- <div className="flex items-center gap-2 text-xs text-stone-500 dark:text-stone-400 bg-stone-100 dark:bg-stone-800 rounded-md px-3 py-2">- <Info className="size-3.5 shrink-0" />- <span>Powered by <span className="font-medium text-stone-700 dark:text-stone-300">Vercel AI SDK</span></span>- </div>-- <div className="grid gap-3">- <div className="flex items-center justify-between">- <Label>{getMessage().EVALUATION_PROVIDER_LABEL}</Label>- <Link href="/manage-models">- <Button variant="ghost" size="sm" className="h-7 text-xs px-2">- <SettingsIcon className="h-3 w-3 mr-1" />- {getMessage().OPENGROUND_MANAGE_MODELS}- </Button>- </Link>- </div>- <Select- value={provider}- onValueChange={(v) => {- setProvider(v);- setModel("");- }}- >- <SelectTrigger>- <SelectValue placeholder={getMessage().EVALUATION_SELECT_PROVIDER} />- </SelectTrigger>- <SelectContent>- {(providers || []).map((p) => (- <SelectItem key={p.providerId} value={p.providerId}>- {p.displayName}- </SelectItem>- ))}- </SelectContent>- </Select>- </div>- <div className="grid gap-3">- <Label>{getMessage().EVALUATION_MODEL_LABEL}</Label>- <Select- value={model}- onValueChange={setModel}- disabled={!provider}- >- <SelectTrigger className="h-auto">- <SelectValue- placeholder={- provider- ? getMessage().EVALUATION_MODEL_PLACEHOLDER- : getMessage().EVALUATION_SELECT_PROVIDER_FIRST- }- />- </SelectTrigger>- <SelectContent>- {modelOptions.map((m) => (- <SelectItem key={m.id} value={m.id}>- <div className="flex flex-col items-start gap-0.5">- <span className="text-sm">{m.displayName}</span>- <div className="flex gap-1.5 text-xs text-stone-500">- <Badge variant="secondary" className="text-xs h-4 px-1">- {m.contextWindow.toLocaleString()}- </Badge>- <Badge variant="outline" className="text-xs h-4 px-1">- ${m.inputPricePerMToken}/M- </Badge>- </div>- </div>- </SelectItem>- ))}- </SelectContent>- </Select>- <p className="text-xs text-stone-500 dark:text-stone-400">- {getMessage().EVALUATION_MODEL_CUSTOM_HINT}- </p>- </div>-- <div className="grid gap-3">- <Label>{getMessage().EVALUATION_API_KEY_VAULT}</Label>- <Select value={vaultId} onValueChange={setVaultId}>- <SelectTrigger>- <SelectValue placeholder={getMessage().EVALUATION_SELECT_VAULT_KEY} />- </SelectTrigger>- <SelectContent>- {vaultKeys.map((o) => (- <SelectItem key={o.value} value={o.value}>- {o.label}- </SelectItem>- ))}- </SelectContent>- </Select>- <EvaluationVaultCreate- successCallback={() =>- getVaultKeys({- requestType: "POST",- url: "/api/vault/get",- responseDataKey: "data",- body: JSON.stringify({}),- successCb: (res: any) => {- setVaultKeys(- res?.length- ? res.map((k: any) => ({- label: k.key,- value: k.id,- }))- : []- );- },- })- }- />- </div>- </CardContent>- </Card>- </div>- <div className="grid gap-4">- {/* Auto Evaluation Card */}- <Card className="border-stone-200 dark:border-stone-800 shadow-sm">- <CardHeader className="pb-4">- <CardTitle className="text-base flex items-center gap-2">- <Zap className="size-4" />- {getMessage().EVALUATION_AUTO_TITLE}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">- {getMessage().EVALUATION_AUTO_DESCRIPTION}- </p>- </CardHeader>- <CardContent className="space-y-4">- <div className="flex items-center justify-between rounded-lg border border-stone-200 dark:border-stone-700 p-4">- <div className="space-y-0.5">- <Label>{getMessage().EVALUATION_ENABLE_AUTO}</Label>- <p className="text-xs text-stone-500 dark:text-stone-400">- {getMessage().EVALUATION_ENABLE_AUTO_DESCRIPTION}- </p>- </div>- <Switch- checked={autoEvaluation}- onCheckedChange={setAutoEvaluation}- />- </div>- {autoEvaluation && (- <div className="grid gap-3">- <Label className="flex items-center gap-2">- <Clock className="size-3.5" />- {getMessage().EVALUATION_CRON_SCHEDULE}- </Label>- <Input- placeholder={getMessage().EVALUATION_CRON_PLACEHOLDER}- value={recurringTime}- onChange={(e) => setRecurringTime(e.target.value)}- />- <p className="text-xs text-stone-500 dark:text-stone-400">- {getMessage().EVALUATION_CRON_HELP}- </p>- <Label>{getMessage().EVALUATION_SAMPLE_RATE_LABEL}</Label>- <Input- type="number"- min={0}- max={100}- step={1}- value={sampleRatePercent}- onChange={(e) => setSampleRatePercent(e.target.value)}- />- <p className="text-xs text-stone-500 dark:text-stone-400">- {getMessage().EVALUATION_SAMPLE_RATE_DESCRIPTION}- </p>- </div>- )}- </CardContent>- </Card>-- <Button onClick={handleSave} disabled={isSaving} className="w-full sm:w-auto bg-primary dark:bg-primary text-white dark:text-white hover:bg-primary/90 dark:hover:bg-primary/90">- {isSaving- ? getMessage().EVALUATION_SAVING- : config?.id- ? getMessage().EVALUATION_SAVE_CHANGES- : getMessage().EVALUATION_CREATE_CONFIG}- </Button>+ router.replace("/evaluations?tab=configuration");+ }, [router]);- {/* Manual Evaluation Section */}- <Card className="border-stone-200 dark:border-stone-800 shadow-sm">- <CardHeader className="pb-4">- <CardTitle className="text-base flex items-center gap-2">- <Play className="size-4" />- {getMessage().EVALUATION_MANUAL_TITLE}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">- {getMessage().EVALUATION_MANUAL_DESCRIPTION}- </p>- </CardHeader>- <CardContent className="space-y-4">- <ol className="list-decimal list-inside space-y-2 text-sm text-stone-600 dark:text-stone-400 mb-3">- <li>{getMessage().EVALUATION_MANUAL_STEP_1}</li>- <li>{getMessage().EVALUATION_MANUAL_STEP_2}</li>- <li>{getMessage().EVALUATION_MANUAL_STEP_3}</li>- </ol>- <Link href="/telemetry">- <Button variant="default">{getMessage().EVALUATION_GO_TO_TRACES}</Button>- </Link>- </CardContent>- </Card>- </div>- </div>- </div>- </div>- );+ return null;}diff --git a/src/client/src/app/(playground)/evaluations/types/[id]/page.tsx b/src/client/src/app/(playground)/evaluations/types/[id]/page.tsxindex 0e077ade..43044b08 100644--- a/src/client/src/app/(playground)/evaluations/types/[id]/page.tsx+++ b/src/client/src/app/(playground)/evaluations/types/[id]/page.tsx@@ -22,6 +22,7 @@ import { Link2, Plus, Trash2, ExternalLink, ArrowLeft, Sparkles, Layers } from "import { Button } from "@/components/ui/button";import Link from "next/link";import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import getMessage from "@/constants/messages";interface RuleWithPriority {ruleId: string;@@ -47,9 +48,11 @@ export default function EvaluationTypeDetailPage() {const et = EVALUATION_TYPES.find((e) => e.id === typeId);const [config, setConfig] = useState<EvaluationTypeConfig | null>(null);- const { fireRequest: getType, data: typeResponse } = useFetchWrapper<{- data?: EvaluationTypeConfig;- }>();+ const [notFound, setNotFound] = useState(false);+ const { fireRequest: getType, data: typeResponse, error: typeError, isFetched: typeFetched } =+ useFetchWrapper<{+ data?: EvaluationTypeConfig;+ }>();const { fireRequest: getRules, data: rules } = useFetchWrapper<Rule[]>();const { fireRequest: getEvalEntities, data: evalEntities } = useFetchWrapper<Array<{ rule_id: string; entity_id: string }>@@ -62,10 +65,12 @@ export default function EvaluationTypeDetailPage() {.map((e) => e.rule_id);useEffect(() => {+ setNotFound(false);+ setConfig(null);posthog?.capture(CLIENT_EVENTS.EVALUATION_TYPE_EDIT_PAGE_VISITED);getType({requestType: "GET",- url: `/api/evaluation/types/${typeId}`,+ url: `/api/evaluation/types/${encodeURIComponent(typeId)}`,responseDataKey: "data",});getRules({ requestType: "GET", url: "/api/rule-engine/rules" });@@ -79,6 +84,7 @@ export default function EvaluationTypeDetailPage() {const res = typeResponse as { data?: EvaluationTypeConfig } | undefined;const data = res?.data ?? (typeResponse as EvaluationTypeConfig | undefined);if (data?.id) {+ setNotFound(false);setConfig({id: data.id,enabled: data.enabled ?? false,@@ -89,7 +95,11 @@ export default function EvaluationTypeDetailPage() {prompt: data.prompt,defaultPrompt: data.defaultPrompt,});- } else if (et) {+ return;+ }++ if (et) {+ setNotFound(false);setConfig({id: et.id,enabled: et.enabledByDefault,@@ -97,8 +107,13 @@ export default function EvaluationTypeDetailPage() {prompt: data?.prompt,defaultPrompt: data?.defaultPrompt,});+ return;}- }, [typeResponse, et]);++ if (typeFetched && (typeError || !data?.id)) {+ setNotFound(true);+ }+ }, [typeResponse, et, typeFetched, typeError]);const handleToggle = (enabled: boolean) => {setConfig((prev) => (prev ? { ...prev, enabled } : null));@@ -183,7 +198,7 @@ export default function EvaluationTypeDetailPage() {responseDataKey: "data",successCb: () => {toast.success("Custom evaluation type deleted");- router.push("/evaluations/types");+ router.push("/evaluations?tab=evaluators");},failureCb: (err?: string) => toast.error(err || "Failed to delete"),});@@ -194,30 +209,58 @@ export default function EvaluationTypeDetailPage() {const displayLabel = et?.label || config?.label || typeId;const displayDescription = et?.description || config?.description || "";+ const evaluatorsHref = "/evaluations?tab=evaluators";+ const backLabel = getMessage().EVALUATION_BACK_TO_TYPES;+const header = (<FeaturePageHeader- eyebrow="Configuration"+ eyebrow={getMessage().FEATURE_EVALS}title={displayLabel}icon={<Layers className="h-4 w-4" />}tone="border-orange-200 bg-orange-50 text-orange-700 dark:border-orange-900/70 dark:bg-orange-950/40 dark:text-orange-300"+ leading={+ <Button+ asChild+ variant="outline"+ size="sm"+ className="h-7 w-7 shrink-0 p-0"+ >+ <Link href={evaluatorsHref} title={backLabel} aria-label={backLabel}>+ <ArrowLeft className="h-3.5 w-3.5" />+ </Link>+ </Button>+ }actions={- <div className="flex flex-wrap items-center gap-2">- {isCustom && (- <Badge variant="outline" className="h-8 gap-1 border-primary/30 text-primary">- <Sparkles className="size-3" />- Custom- </Badge>- )}+ isCustom ? (+ <Badge variant="outline" className="h-8 gap-1 border-primary/30 text-primary">+ <Sparkles className="size-3" />+ Custom+ </Badge>+ ) : null+ }+ />+ );++ if (notFound) {+ return (+ <div className="flex h-full w-full flex-col overflow-hidden">+ {header}+ <div className="flex flex-1 w-full flex-col items-center justify-center gap-3 p-4 text-center">+ <p className="text-base font-medium text-stone-800 dark:text-stone-200">+ {getMessage().EVALUATION_TYPE_NOT_FOUND}+ </p>+ <p className="max-w-md text-sm text-stone-500 dark:text-stone-400">+ {getMessage().EVALUATION_TYPE_NOT_FOUND_DESCRIPTION}+ </p><Button asChild variant="outline" size="sm" className="h-8">- <Link href="/evaluations/types">- <ArrowLeft className="mr-1.5 h-3.5 w-3.5" />- Evaluation Types+ <Link href="/evaluations?tab=evaluators">+ {getMessage().EVALUATION_BACK_TO_TYPES}</Link></Button></div>- }- />- );+ </div>+ );+ }if (!et && !config) {return (diff --git a/src/client/src/app/(playground)/evaluations/types/new/page.tsx b/src/client/src/app/(playground)/evaluations/types/new/page.tsxindex 406ca74c..ef405d41 100644--- a/src/client/src/app/(playground)/evaluations/types/new/page.tsx+++ b/src/client/src/app/(playground)/evaluations/types/new/page.tsx@@ -13,8 +13,12 @@ import useFetchWrapper from "@/utils/hooks/useFetchWrapper";import { usePostHog } from "posthog-js/react";import { CLIENT_EVENTS } from "@/constants/events";import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import getMessage from "@/constants/messages";++const EVALUATORS_HREF = "/evaluations?tab=evaluators";export default function CreateCustomEvaluationTypePage() {+ const m = getMessage();const posthog = usePostHog();const router = useRouter();const [typeId, setTypeId] = useState("");@@ -98,15 +102,23 @@ export default function CreateCustomEvaluationTypePage() {return (<div className="flex h-full w-full flex-col overflow-hidden"><FeaturePageHeader- eyebrow="Configuration"+ eyebrow={m.FEATURE_EVALS}title="Create Custom Evaluation Type"icon={<Sparkles className="h-4 w-4" />}tone="border-orange-200 bg-orange-50 text-orange-700 dark:border-orange-900/70 dark:bg-orange-950/40 dark:text-orange-300"- actions={- <Button asChild variant="outline" size="sm" className="h-8">- <Link href="/evaluations/types">- <ArrowLeft className="mr-1.5 h-3.5 w-3.5" />- Evaluation Types+ leading={+ <Button+ asChild+ variant="outline"+ size="sm"+ className="h-7 w-7 shrink-0 p-0"+ >+ <Link+ href={EVALUATORS_HREF}+ title={m.EVALUATION_BACK_TO_TYPES}+ aria-label={m.EVALUATION_BACK_TO_TYPES}+ >+ <ArrowLeft className="h-3.5 w-3.5" /></Link></Button>}diff --git a/src/client/src/app/(playground)/evaluations/types/page.tsx b/src/client/src/app/(playground)/evaluations/types/page.tsxindex 52d5a616..032a3857 100644--- a/src/client/src/app/(playground)/evaluations/types/page.tsx+++ b/src/client/src/app/(playground)/evaluations/types/page.tsx@@ -1,214 +1,14 @@"use client";-import { Card, CardHeader, CardTitle } from "@/components/ui/card";-import { Badge } from "@/components/ui/badge";-import useFetchWrapper from "@/utils/hooks/useFetchWrapper";-import { useEffect, useState } from "react";-import { EVALUATION_TYPES } from "@/constants/evaluation-types";-import { usePostHog } from "posthog-js/react";-import { CLIENT_EVENTS } from "@/constants/events";-import { Rule } from "@/types/rule-engine";-import { Layers, CheckCircle2, ChevronRight, Settings2, Plus, Sparkles } from "lucide-react";-import Link from "next/link";-import { Button } from "@/components/ui/button";-import FeaturePageHeader from "@/components/(playground)/feature-page-header";--interface EvaluationTypeDisplay {- id: string;- label: string;- description: string;- enabledByDefault: boolean;- enabled: boolean;- isCustom?: boolean;- rules?: Array<{ ruleId: string; priority: number }>;- prompt?: string;-}+import { useEffect } from "react";+import { useRouter } from "next/navigation";export default function EvaluationTypesPage() {- const posthog = usePostHog();- const { fireRequest: getTypes, data: typesResponse } = useFetchWrapper<- EvaluationTypeDisplay[] | { data: any[] }- >();- const { fireRequest: getRules, data: rules } = useFetchWrapper<Rule[]>();- const { fireRequest: getEvalEntities, data: evalEntities } = useFetchWrapper<- Array<{ rule_id: string; entity_id: string }>- >();-- const [allTypes, setAllTypes] = useState<EvaluationTypeDisplay[]>([]);-- const rulesLinkedToType = (evalEntities || []).reduce<- Record<string, string[]>- >((acc, e) => {- if (!acc[e.entity_id]) acc[e.entity_id] = [];- if (!acc[e.entity_id].includes(e.rule_id)) acc[e.entity_id].push(e.rule_id);- return acc;- }, {});-- useEffect(() => {- posthog?.capture(CLIENT_EVENTS.EVALUATION_TYPES_PAGE_VISITED);- getTypes({- requestType: "GET",- url: "/api/evaluation/types",- responseDataKey: "data",- successCb: (response: EvaluationTypeDisplay[] | { data?: any[] }) => {- const list = Array.isArray(response)- ? response- : response?.data;- posthog?.capture(CLIENT_EVENTS.EVALUATION_TYPES_LIST, {- count: Array.isArray(list) ? list.length : 0,- });- },- });- getRules({ requestType: "GET", url: "/api/rule-engine/rules" });- getEvalEntities({- requestType: "GET",- url: "/api/rule-engine/entities?entity_type=evaluation",- });- }, []);+ const router = useRouter();useEffect(() => {- const savedTypes = Array.isArray(typesResponse)- ? typesResponse- : (typesResponse as { data?: any[] })?.data;- if (savedTypes !== undefined && Array.isArray(savedTypes)) {- setAllTypes(savedTypes);- }- }, [typesResponse]);-- const builtInIds = new Set<string>(EVALUATION_TYPES.map((et) => et.id));- const builtInTypes = allTypes.filter((t) => builtInIds.has(t.id));- const customTypes = allTypes.filter((t) => !builtInIds.has(t.id));-- const displayBuiltIn = EVALUATION_TYPES.map((et) => {- const saved = builtInTypes.find((t) => t.id === et.id);- return {- ...et,- enabled: saved?.enabled ?? et.enabledByDefault,- isCustom: false,- rules: saved?.rules || [],- };- });-- const renderTypeCard = (type: EvaluationTypeDisplay, isCustomType: boolean) => {- const hasRules =- (type.rules?.length ?? 0) > 0 ||- (rulesLinkedToType[type.id]?.length ?? 0) > 0;- return (- <Link key={type.id} href={`/evaluations/types/${type.id}`}>- <Card className={`shadow-sm overflow-hidden transition-colors cursor-pointer ${- isCustomType- ? "border-primary/20 dark:border-primary/15 hover:border-primary/40 dark:hover:border-primary/30"- : "border-stone-200 dark:border-stone-800 hover:border-stone-300 dark:hover:border-stone-700"- }`}>- <CardHeader className="pb-3">- <div className="flex items-start justify-between gap-4">- <div className="flex-1 min-w-0">- <CardTitle className="text-base flex items-center gap-2 flex-wrap text-stone-900 dark:text-stone-100">- {type.label}- {isCustomType && (- <Badge variant="outline" className="text-xs font-normal gap-1 shrink-0 border-primary/30 text-primary dark:border-primary/20">- <Sparkles className="size-3" />- Custom- </Badge>- )}- {hasRules && (- <Badge- variant="outline"- className="text-xs font-normal gap-1 shrink-0"- >- <CheckCircle2 className="size-3" />- Rule engine- </Badge>- )}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 mt-1">- {type.description}- </p>- {(rulesLinkedToType[type.id]?.length ?? 0) > 0 && (- <div className="flex items-center gap-1.5 mt-2 flex-wrap">- <span className="text-xs text-stone-500 dark:text-stone-400">- Linked from rules:- </span>- {rulesLinkedToType[type.id].slice(0, 3).map((rid) => {- const r = (rules || []).find((x) => x.id === rid);- return (- <span- key={rid}- className="inline-flex items-center gap-1 text-xs text-primary"- >- {r?.name || rid.slice(0, 8)}- </span>- );- })}- {rulesLinkedToType[type.id].length > 3 && (- <span className="text-xs text-stone-400 dark:text-stone-500">- +{rulesLinkedToType[type.id].length - 3} more- </span>- )}- </div>- )}- </div>- <div className="flex items-center gap-2 shrink-0">- <Badge variant={type.enabled ? "default" : "secondary"}>- {type.enabled ? "Enabled" : "Disabled"}- </Badge>- <ChevronRight className="size-4 text-stone-400 dark:text-stone-500" />- </div>- </div>- </CardHeader>- </Card>- </Link>- );- };-- return (- <div className="flex h-full w-full flex-col overflow-hidden">- <FeaturePageHeader- eyebrow="Configuration"- title="Evaluation Types"- icon={<Layers className="h-4 w-4" />}- tone="border-orange-200 bg-orange-50 text-orange-700 dark:border-orange-900/70 dark:bg-orange-950/40 dark:text-orange-300"- actions={- <div className="flex flex-wrap items-center gap-2">- <Button asChild variant="outline" size="sm" className="h-8">- <Link href="/evaluations/settings">- <Settings2 className="mr-1.5 h-3.5 w-3.5" />- Settings- </Link>- </Button>- <Button asChild size="sm" className="h-8">- <Link href="/evaluations/types/new">- <Plus className="mr-1.5 h-3.5 w-3.5" />- Create Custom Type- </Link>- </Button>- </div>- }- />-- <div className="flex-1 w-full overflow-auto p-4">- <div className="w-full space-y-4">- {/* Built-in types */}- <div className="grid grid-cols-1 md:grid-cols-2 gap-4">- {displayBuiltIn.map((et) => renderTypeCard(et, false))}- </div>+ router.replace("/evaluations?tab=evaluators");+ }, [router]);- {/* Custom types */}- {customTypes.length > 0 && (- <>- <div className="flex items-center gap-2 pt-2">- <Sparkles className="size-4 text-primary" />- <h3 className="text-sm font-semibold text-stone-700 dark:text-stone-300">- Custom Evaluation Types- </h3>- </div>- <div className="grid grid-cols-1 md:grid-cols-2 gap-4">- {customTypes.map((ct) => renderTypeCard(ct, true))}- </div>- </>- )}- </div>- </div>- </div>- );+ return null;}diff --git a/src/client/src/app/(playground)/fleet-hub/[id]/agent.tsx b/src/client/src/app/(playground)/fleet-hub/[id]/agent.tsxindex c040fd97..e1807848 100644--- a/src/client/src/app/(playground)/fleet-hub/[id]/agent.tsx+++ b/src/client/src/app/(playground)/fleet-hub/[id]/agent.tsx@@ -52,9 +52,9 @@ export default function AgentDetail({ agent, fetchAgentInfo }: AgentDetailProps)return (<div className="flex h-full w-full flex-col overflow-hidden"><FeaturePageHeader- eyebrow={getMessage().FEATURE_FLEET_HUB}+ eyebrow={getMessage().SIDEBAR_MONITOR}title={serviceName}- icon={<OpenTelemetrySvg className="h-5 w-5" />}+ icon={<OpenTelemetrySvg className="h-4 w-4" />}tone="border-sky-200 bg-sky-50 text-sky-700 dark:border-sky-900/70 dark:bg-sky-950/40 dark:text-sky-300"leading={(<Buttondiff --git a/src/client/src/app/(playground)/fleet-hub/list.tsx b/src/client/src/app/(playground)/fleet-hub/list.tsxindex 37e7f7e9..2a45018e 100644--- a/src/client/src/app/(playground)/fleet-hub/list.tsx+++ b/src/client/src/app/(playground)/fleet-hub/list.tsx@@ -140,7 +140,7 @@ export default function List({ agents, isLoading, isFetched }: {return (<div className="flex flex-col w-full h-full">- <FeaturePageHeader eyebrow={getMessage().FEATURE_FLEET_HUB} title={getMessage().FEATURE_FLEET_HUB} icon={<OpenTelemetrySvg className="h-5 w-5" />} tone="border-sky-200 bg-sky-50 text-sky-700 dark:border-sky-900/70 dark:bg-sky-950/40 dark:text-sky-300" actions={<VisibilityColumns columns={columns} pageName={"fleethub"} />} />+ <FeaturePageHeader eyebrow={getMessage().SIDEBAR_MONITOR} title={getMessage().FEATURE_FLEET_HUB} icon={<OpenTelemetrySvg className="h-4 w-4" />} tone="border-sky-200 bg-sky-50 text-sky-700 dark:border-sky-900/70 dark:bg-sky-950/40 dark:text-sky-300" actions={<VisibilityColumns columns={columns} pageName={"fleethub"} />} /><div className="flex flex-col w-full h-full p-4 overflow-hidden"><DataTablecolumns={columns}diff --git a/src/client/src/app/(playground)/manage-models/page.tsx b/src/client/src/app/(playground)/manage-models/page.tsxindex 1851af94..07136b4f 100644--- a/src/client/src/app/(playground)/manage-models/page.tsx+++ b/src/client/src/app/(playground)/manage-models/page.tsx@@ -1,402 +1,14 @@"use client";-import { useState, useEffect, useMemo } from "react";-import { usePostHog } from "posthog-js/react";-import { CLIENT_EVENTS } from "@/constants/events";-import { Button } from "@/components/ui/button";-import {- SettingsIcon,- Download,- Upload,- Copy,- Link as LinkIcon,- Code2,- PlusCircle,-} from "lucide-react";-import getMessage from "@/constants/messages";-import ModelListSidebar from "@/components/(playground)/openground/model-list-sidebar";-import ModelEditorPanel from "@/components/(playground)/openground/model-editor-panel";-import SdkUsageDialog from "@/components/(playground)/openground/sdk-usage-dialog";-import ProviderEditorDialog, {- ProviderFormData,-} from "@/components/(playground)/openground/provider-editor-dialog";-import { ProviderMetadata, ModelMetadata } from "@/types/openground";-import useFetchWrapper from "@/utils/hooks/useFetchWrapper";-import {- Dialog,- DialogContent,- DialogDescription,- DialogFooter,- DialogHeader,- DialogTitle,-} from "@/components/ui/dialog";-import { Textarea } from "@/components/ui/textarea";-import { toast } from "sonner";-import { useRootStore } from "@/store";-import { getDatabaseConfigList } from "@/selectors/database-config";-import { Input } from "@/components/ui/input";-import copy from "copy-to-clipboard";-import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip";-import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import { useEffect } from "react";+import { useRouter } from "next/navigation";-interface CustomModel extends ModelMetadata {- id: string;- customId?: string;- model_id: string;- provider?: string;- modelType?: string;-}--export default function ManageModelsPage() {- const m = getMessage();- const posthog = usePostHog();- const [selectedProvider, setSelectedProvider] = useState<string | null>(null);- const [selectedModel, setSelectedModel] = useState<ModelMetadata | null>(null);- const [isCustomModel, setIsCustomModel] = useState(false);- const [isAddingNew, setIsAddingNew] = useState(false);- const [showImport, setShowImport] = useState(false);- const [showSdkUsage, setShowSdkUsage] = useState(false);- const [showProviderEditor, setShowProviderEditor] = useState(false);- const [editingProvider, setEditingProvider] = useState<ProviderFormData | null>(null);- const [importJson, setImportJson] = useState("");-- // Pull the active database config from the zustand store- const databaseConfigList = useRootStore(getDatabaseConfigList) || [];- const activeDbConfig = useMemo(- () => databaseConfigList.find((item) => !!item.isCurrent),- [databaseConfigList]- );- const dbConfigId = activeDbConfig?.id || "";-- const {- data: providers,- fireRequest: fetchProviders,- isLoading: loadingProviders,- } = useFetchWrapper<ProviderMetadata[]>();- const {- data: customModels,- fireRequest: fetchCustomModels,- isLoading: loadingCustom,- } = useFetchWrapper<Record<string, CustomModel[]>>();- const { fireRequest: fireImport, isLoading: importing } = useFetchWrapper();-- useEffect(() => {- posthog?.capture(CLIENT_EVENTS.SETTINGS_MANAGE_MODELS_PAGE_VISITED);- }, []);+export default function ManageModelsRedirectPage() {+ const router = useRouter();useEffect(() => {- loadProviders();- loadAllCustomModels();- }, []);-- const loadProviders = () => {- fetchProviders({ requestType: "GET", url: "/api/openground/providers" });- };-- const loadAllCustomModels = () => {- fetchCustomModels({ requestType: "GET", url: "/api/openground/models" });- };-- const handleSelectModel = (- model: ModelMetadata,- provider: string,- _isCustom: boolean- ) => {- setSelectedModel(model);- setSelectedProvider(provider);- setIsCustomModel(true);- setIsAddingNew(false);- };-- const handleAddNew = (provider: string) => {- setSelectedProvider(provider);- setSelectedModel(null);- setIsCustomModel(true);- setIsAddingNew(true);- };-- const handleModelSaved = () => {- loadAllCustomModels();- setIsAddingNew(false);- };-- const handleModelDeleted = () => {- loadAllCustomModels();- setSelectedModel(null);- setSelectedProvider(null);- setIsCustomModel(false);- };-- const handleAddProvider = () => {- setEditingProvider(null);- setShowProviderEditor(true);- };-- const handleEditProvider = (provider: ProviderMetadata) => {- setEditingProvider({- providerId: provider.providerId,- displayName: provider.displayName,- description: provider.description || "",- requiresVault: provider.requiresVault,- });- setShowProviderEditor(true);- };-- const handleProviderSaved = () => {- loadProviders();- loadAllCustomModels();- };-- // Build the public pricing URL from the active db config- const pricingUrl = dbConfigId- ? `${typeof window !== "undefined" ? window.location.origin : ""}/api/pricing/export/${dbConfigId}`- : "";-- const copyUrl = () => {- if (!pricingUrl) return;- copy(pricingUrl);- toast.success(m.MANAGE_MODELS_PRICING_URL_COPIED, { id: "url-copy" });- };-- const handleImport = () => {- let parsed: any;- try {- parsed = JSON.parse(importJson);- } catch {- toast.error(m.MANAGE_MODELS_INVALID_JSON, { id: "import" });- return;- }-- fireImport({- requestType: "POST",- url: "/api/openground/models/import",- body: JSON.stringify(parsed),- successCb: (res: any) => {- toast.success(- `${m.MANAGE_MODELS_IMPORT_SUCCESS}: ${res?.imported ?? 0} imported, ${res?.skipped ?? 0} skipped`,- { id: "import" }- );- setShowImport(false);- setImportJson("");- loadAllCustomModels();- loadProviders();- },- failureCb: (err?: string) => {- toast.error(err || m.MANAGE_MODELS_IMPORT_FAILED, { id: "import" });- },- });- };-- return (- <div className="flex flex-col h-full w-full overflow-hidden">- <div className="shrink-0">- <FeaturePageHeader eyebrow="Configuration" title={m.OPENGROUND_MANAGE_MODELS} icon={<SettingsIcon className="h-4 w-4" />} tone="border-violet-200 bg-violet-50 text-violet-700 dark:border-violet-900/70 dark:bg-violet-950/40 dark:text-violet-300" actions={<div className="flex items-center gap-2">- <Tooltip>- <TooltipTrigger>- <Button- variant="outline"- size="sm"- className="gap-2"- onClick={handleAddProvider}- >- <PlusCircle className="h-4 w-4" />- </Button>- </TooltipTrigger>- <TooltipContent>- {m.MANAGE_PROVIDERS_ADD}- </TooltipContent>- </Tooltip>- {pricingUrl && (- <Tooltip>- <TooltipTrigger>- <Button- variant="outline"- size="sm"- className="gap-2"- onClick={() => setShowSdkUsage(true)}- >- <Code2 className="h-4 w-4" />- </Button>- </TooltipTrigger>- <TooltipContent>- {m.MANAGE_MODELS_SDK_USAGE}- </TooltipContent>- </Tooltip>- )}- <Tooltip>- <TooltipTrigger>- <Button- variant="outline"- size="sm"- className="gap-2"- onClick={() => setShowImport(true)}- >- <Upload className="h-4 w-4" />- </Button>- </TooltipTrigger>- <TooltipContent>- {m.MANAGE_MODELS_IMPORT_PRICING}- </TooltipContent>- </Tooltip>- <Tooltip>- <TooltipTrigger>- <Button- variant="outline"- size="sm"- className="gap-2"- onClick={() => {- window.open("/api/openground/models/export", "_blank");- }}- >- <Download className="h-4 w-4" />- </Button>- </TooltipTrigger>- <TooltipContent>- {m.MANAGE_MODELS_EXPORT_PRICING}- </TooltipContent>- </Tooltip>- </div>} />- </div>- <div className="flex flex-col grow overflow-hidden p-4">- <div className="flex flex-col w-full h-full border border-stone-200 dark:border-stone-800 rounded-md">- {/* SDK Pricing URL bar */}- {pricingUrl && (- <div className="flex items-center gap-3 px-4 py-2.5 bg-stone-50 dark:bg-stone-950 border-b border-stone-200 dark:border-stone-800 shrink-0 rounded-t-md">- <LinkIcon className="h-3.5 w-3.5 text-stone-400 shrink-0" />- <span className="text-xs font-medium text-stone-500 dark:text-stone-400 shrink-0">- {m.MANAGE_MODELS_PRICING_URL_LABEL}- </span>- <Input- readOnly- value={pricingUrl}- onFocus={(e) => e.target.select()}- className="font-mono text-xs h-8 flex-1 min-w-0"- />- <Button- variant="outline"- size="sm"- className="gap-1.5 shrink-0 h-8"- onClick={copyUrl}- >- <Copy className="h-3.5 w-3.5" />- {m.COPY}- </Button>- </div>- )}-- {/* Main Content - Split Layout */}- <div className="flex flex-1 overflow-hidden min-h-0">- <ModelListSidebar- providers={providers || []}- customModels={customModels || {}}- loading={loadingProviders || loadingCustom}- selectedModel={selectedModel}- selectedProvider={selectedProvider}- selectedIsCustom={isCustomModel}- onSelectModel={handleSelectModel}- onAddNew={handleAddNew}- onEditProvider={handleEditProvider}- />-- <div className="flex-1 overflow-auto bg-stone-50 dark:bg-stone-950">- {selectedModel || isAddingNew ? (- <ModelEditorPanel- model={selectedModel}- provider={selectedProvider}- isCustomModel={isCustomModel}- isAddingNew={isAddingNew}- onSave={handleModelSaved}- onDelete={handleModelDeleted}- onCancel={() => {- setSelectedModel(null);- setSelectedProvider(null);- setIsAddingNew(false);- }}- />- ) : (- <div className="flex flex-col items-center justify-center h-full p-8 text-center">- <SettingsIcon className="h-16 w-16 text-stone-300 dark:text-stone-700 mb-4" />- <h3 className="text-lg font-medium text-stone-900 dark:text-stone-100 mb-2">- {m.OPENGROUND_SELECT_MODEL_TO_VIEW}- </h3>- <p className="text-sm text-stone-500 dark:text-stone-400 max-w-md">- {m.OPENGROUND_SELECT_MODEL_TO_VIEW_DESCRIPTION}- </p>- {pricingUrl && (- <Button- variant="outline"- size="sm"- className="gap-2 mt-6"- onClick={() => setShowSdkUsage(true)}- >- <Code2 className="h-4 w-4" />- {m.MANAGE_MODELS_SDK_USAGE}- </Button>- )}- </div>- )}- </div>- </div>-- {/* Import Dialog */}- <Dialog open={showImport} onOpenChange={setShowImport}>- <DialogContent className="sm:max-w-lg">- <DialogHeader>- <DialogTitle>{m.MANAGE_MODELS_IMPORT_DIALOG_TITLE}</DialogTitle>- <DialogDescription>- {m.MANAGE_MODELS_IMPORT_DIALOG_DESCRIPTION}- </DialogDescription>- </DialogHeader>- <Textarea- placeholder={`{- "providers": [- {- "providerId": "my-provider",- "displayName": "My Provider",- "description": "Custom LLM provider"- }- ],- "models": [- {- "provider": "my-provider",- "model_id": "my-model-v1",- "displayName": "My Model v1",- "inputPricePerMToken": 2.5,- "outputPricePerMToken": 10- }- ]- }`}- value={importJson}- onChange={(e) => setImportJson(e.target.value)}- className="min-h-[200px] text-xs dark:text-stone-100 text-stone-800"- />- <DialogFooter>- <Button variant="outline" onClick={() => setShowImport(false)}>- {m.CANCEL}- </Button>- <Button onClick={handleImport} disabled={importing || !importJson.trim()}>- {importing ? m.SAVING : m.MANAGE_MODELS_IMPORT_PRICING}- </Button>- </DialogFooter>- </DialogContent>- </Dialog>-- {/* SDK Usage Dialog */}- <SdkUsageDialog- open={showSdkUsage}- onOpenChange={setShowSdkUsage}- pricingUrl={pricingUrl}- />+ router.replace("/costs?tab=models");+ }, [router]);- {/* Provider Editor Dialog */}- <ProviderEditorDialog- open={showProviderEditor}- onOpenChange={setShowProviderEditor}- provider={editingProvider}- onSaved={handleProviderSaved}- />- </div>- </div>- </div>- );+ return null;}diff --git a/src/client/src/app/(playground)/pricing/page.tsx b/src/client/src/app/(playground)/pricing/page.tsxindex 9cd3fe19..1b82fb69 100644--- a/src/client/src/app/(playground)/pricing/page.tsx+++ b/src/client/src/app/(playground)/pricing/page.tsx@@ -1,233 +1,14 @@"use client";-import { Button } from "@/components/ui/button";-import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";-import { Input } from "@/components/ui/input";-import { Label } from "@/components/ui/label";-import { Switch } from "@/components/ui/switch";-import getMessage from "@/constants/messages";-import useFetchWrapper from "@/utils/hooks/useFetchWrapper";-import {- CircleDollarSign,- Clock,- Info,- Play,- Workflow,- Zap,-} from "lucide-react";-import Link from "next/link";-import { useEffect, useState } from "react";-import { toast } from "sonner";-import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import { useEffect } from "react";+import { useRouter } from "next/navigation";-interface PricingConfig {- id?: string;- auto?: boolean;- recurringTime?: string;- meta?: string;-}--const PRICING_TOAST_ID = "pricing-config";--export default function PricingPage() {- const m = getMessage();- const [autoPricing, setAutoPricing] = useState(false);- const [recurringTime, setRecurringTime] = useState("");-- const {- fireRequest: getConfig,- data: config,- isLoading: isLoadingConfig,- } = useFetchWrapper<PricingConfig>();- const { fireRequest: saveConfig, isLoading: isSaving } = useFetchWrapper();+export default function PricingRedirectPage() {+ const router = useRouter();useEffect(() => {- getConfig({- requestType: "GET",- url: "/api/pricing/config",- responseDataKey: "data",- });- }, []);-- useEffect(() => {- if (config) {- setAutoPricing(config.auto || false);- setRecurringTime(config.recurringTime || "");- }- }, [config]);-- const handleSave = () => {- toast.loading(m.SAVING, { id: PRICING_TOAST_ID });-- saveConfig({- body: JSON.stringify({- id: config?.id,- auto: autoPricing,- recurringTime: recurringTime || "* * * * *",- meta: config?.meta || "{}",- }),- requestType: "POST",- url: "/api/pricing/config",- responseDataKey: "data",- successCb: () => {- toast.success(m.PRICING_CONFIG_SAVED, { id: PRICING_TOAST_ID });- getConfig({- requestType: "GET",- url: "/api/pricing/config",- responseDataKey: "data",- });- },- failureCb: (err?: string) => {- toast.error(err || m.OPERATION_FAILED, {- id: PRICING_TOAST_ID,- });- },- });- };-- const header = (- <FeaturePageHeader eyebrow="Configuration" title={m.PRICING_TITLE} icon={<CircleDollarSign className="size-4" />} tone="border-emerald-200 bg-emerald-50 text-emerald-700 dark:border-emerald-900/70 dark:bg-emerald-950/40 dark:text-emerald-300" />- );-- if (isLoadingConfig && !config) {- return (- <div className="flex flex-col h-full w-full overflow-hidden">- {header}- <div className="flex flex-1 w-full p-4">- <div className="animate-pulse grid gap-4 w-full max-w-3xl">- <div className="h-40 bg-stone-100 dark:bg-stone-900" />- </div>- </div>- </div>- );- }-- return (- <div className="flex flex-col h-full w-full overflow-hidden">- {header}-- <div className="flex flex-col flex-1 w-full overflow-auto gap-4 p-4">-- {/* Single info bar (replaces the large left description card) */}- <Card className="border-blue-200 dark:border-blue-900/50 bg-blue-50/50 dark:bg-blue-950/20">- <CardContent className="py-3">- <div className="flex items-start gap-3">- <Info className="size-4 text-blue-600 dark:text-blue-500 shrink-0 mt-0.5" />- <p className="text-xs text-stone-600 dark:text-stone-400 leading-relaxed">- {m.PRICING_INFO_BAR}{" "}- <Link- href="/manage-models"- className="font-medium text-stone-700 dark:text-stone-300 underline"- >- {m.OPENGROUND_MANAGE_MODELS}- </Link>- .- </p>- </div>- </CardContent>- </Card>-- {/* Auto + Manual side-by-side to save vertical space */}- <div className="grid grid-cols-1 lg:grid-cols-2 gap-4">- <Card className="border-stone-200 dark:border-stone-800 shadow-sm flex flex-col">- <CardHeader className="pb-4">- <CardTitle className="text-base flex items-center gap-2">- <Zap className="size-4" />- {m.PRICING_AUTO_TITLE}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">- {m.PRICING_AUTO_DESCRIPTION}- </p>- </CardHeader>- <CardContent className="space-y-4 flex-1">- <div className="flex items-center justify-between rounded-lg border border-stone-200 dark:border-stone-700 p-4">- <div className="space-y-0.5">- <Label>{m.PRICING_AUTO_ENABLE_LABEL}</Label>- <p className="text-xs text-stone-500 dark:text-stone-400">- {m.PRICING_AUTO_ENABLE_HINT}- </p>- </div>- <Switch- checked={autoPricing}- onCheckedChange={setAutoPricing}- />- </div>- {autoPricing && (- <div className="grid gap-3">- <Label className="flex items-center gap-2">- <Clock className="size-3.5" />- {m.PRICING_AUTO_CRON_LABEL}- </Label>- <Input- placeholder={m.PRICING_AUTO_CRON_PLACEHOLDER}- value={recurringTime}- onChange={(e) => setRecurringTime(e.target.value)}- />- <p className="text-xs text-stone-500 dark:text-stone-400">- {m.PRICING_AUTO_CRON_HELP}- </p>- </div>- )}- <Button- onClick={handleSave}- disabled={isSaving}- className="bg-primary dark:bg-primary text-white dark:text-white hover:bg-primary/90 dark:hover:bg-primary/90"- >- {isSaving- ? m.SAVING- : config?.id- ? m.PRICING_UPDATE- : m.PRICING_SAVE}- </Button>- </CardContent>- </Card>-- <Card className="border-stone-200 dark:border-stone-800 shadow-sm flex flex-col">- <CardHeader className="pb-4">- <CardTitle className="text-base flex items-center gap-2">- <Play className="size-4" />- {m.PRICING_MANUAL_TITLE}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">- {m.PRICING_MANUAL_DESCRIPTION}- </p>- </CardHeader>- <CardContent className="space-y-4 flex-1">- <ol className="list-decimal list-inside space-y-2 text-sm text-stone-600 dark:text-stone-400">- <li>{m.PRICING_MANUAL_STEP_1}</li>- <li>{m.PRICING_MANUAL_STEP_2}</li>- <li>{m.PRICING_MANUAL_STEP_3}</li>- </ol>- <Link href="/telemetry" className="mt-3 inline-block">- <Button variant="default">{m.PRICING_GO_TO_TRACES}</Button>- </Link>- </CardContent>- </Card>- </div>+ router.replace("/costs?tab=configuration");+ }, [router]);- {/* How auto pricing works */}- <Card className="border-stone-200 dark:border-stone-800 shadow-sm">- <CardHeader className="pb-3">- <CardTitle className="text-base flex items-center gap-2">- <Workflow className="size-4" />- {m.PRICING_HOW_AUTO_WORKS_TITLE}- </CardTitle>- <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">- {m.PRICING_HOW_AUTO_WORKS_DESCRIPTION}- </p>- </CardHeader>- <CardContent>- <ol className="list-decimal list-inside space-y-2 text-sm text-stone-600 dark:text-stone-400 marker:text-stone-400">- <li>{m.PRICING_HOW_AUTO_WORKS_STEP_1}</li>- <li>{m.PRICING_HOW_AUTO_WORKS_STEP_2}</li>- <li>{m.PRICING_HOW_AUTO_WORKS_STEP_3}</li>- <li>{m.PRICING_HOW_AUTO_WORKS_STEP_4}</li>- <li>{m.PRICING_HOW_AUTO_WORKS_STEP_5}</li>- <li>{m.PRICING_HOW_AUTO_WORKS_STEP_6}</li>- </ol>- </CardContent>- </Card>- </div>- </div>- );+ return null;}diff --git a/src/client/src/app/(playground)/prompt-hub/[id]/page.tsx b/src/client/src/app/(playground)/prompt-hub/[id]/page.tsxindex 3ac937b9..ac136c96 100644--- a/src/client/src/app/(playground)/prompt-hub/[id]/page.tsx+++ b/src/client/src/app/(playground)/prompt-hub/[id]/page.tsx@@ -151,7 +151,7 @@ export default function PromptHub() {if (!isFetched || (!(data as any)?.promptId && isLoading)) {return (<div className="flex h-full w-full flex-col overflow-hidden">- <FeaturePageHeader eyebrow="Resources" title={m.LOADING} icon={<Component className="h-4 w-4" />} tone={promptHeaderTone} />+ <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={m.LOADING} icon={<Component className="h-4 w-4" />} tone={promptHeaderTone} /><div className="flex flex-col w-full h-full overflow-hidden gap-6 items-center justify-center"><div className="h-4 w-1/5 bg-secondary/[0.9] rounded" /><div className="h-4 w-3/5 bg-secondary/[0.9] rounded" />@@ -165,7 +165,7 @@ export default function PromptHub() {if (!data || !(data as any)?.promptId) {return (<div className="flex h-full w-full flex-col overflow-hidden">- <FeaturePageHeader eyebrow="Resources" title={m.FEATURE_PROMPTS} icon={<Component className="h-4 w-4" />} tone={promptHeaderTone} />+ <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={m.FEATURE_PROMPTS} icon={<Component className="h-4 w-4" />} tone={promptHeaderTone} /><div className="flex w-full h-full overflow-hidden items-center justify-center text-stone-600 dark:text-stone-400">{m.PROMPT_HUB_NO_PROMPT_EXISTS}</div>@@ -181,7 +181,7 @@ export default function PromptHub() {) {return (<div className="flex h-full w-full flex-col overflow-hidden">- <FeaturePageHeader eyebrow="Resources" title={m.FEATURE_PROMPTS} icon={<Component className="h-4 w-4" />} tone={promptHeaderTone} />+ <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={m.FEATURE_PROMPTS} icon={<Component className="h-4 w-4" />} tone={promptHeaderTone} /><div className="flex w-full h-full overflow-hidden items-center justify-center text-stone-600 dark:text-stone-400">{m.PROMPT_HUB_NO_VERSION_EXISTS}{" "}<span className="bg-secondary text-primary px-2 text-sm mx-3">diff --git a/src/client/src/app/(playground)/rule-engine/[id]/page.tsx b/src/client/src/app/(playground)/rule-engine/[id]/page.tsxindex c36cc00d..00a0080f 100644--- a/src/client/src/app/(playground)/rule-engine/[id]/page.tsx+++ b/src/client/src/app/(playground)/rule-engine/[id]/page.tsx@@ -172,7 +172,7 @@ export default function RuleDetailPage() {if (isLoading && !rule) {return (<div className="flex h-full w-full flex-col overflow-hidden">- <FeaturePageHeader eyebrow="Resources" title={messages.LOADING} icon={<SlidersHorizontal className="h-4 w-4" />} tone={ruleHeaderTone} />+ <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={messages.LOADING} icon={<SlidersHorizontal className="h-4 w-4" />} tone={ruleHeaderTone} /><div className="flex flex-col w-full h-full overflow-hidden gap-4 items-center justify-center"><div className="h-4 w-1/5 bg-stone-200 dark:bg-stone-700 rounded animate-pulse" /><div className="h-4 w-3/5 bg-stone-200 dark:bg-stone-700 rounded animate-pulse" />@@ -189,7 +189,7 @@ export default function RuleDetailPage() {return (<div className="flex h-full w-full flex-col overflow-hidden"><FeaturePageHeader- eyebrow="Resources"+ eyebrow={getMessage().SIDEBAR_DEVELOP}title={r.name}icon={<SlidersHorizontal className="h-4 w-4" />}tone={ruleHeaderTone}diff --git a/src/client/src/app/(playground)/settings/evaluation/page.tsx b/src/client/src/app/(playground)/settings/evaluation/page.tsxindex 270e48d4..f73daafe 100644--- a/src/client/src/app/(playground)/settings/evaluation/page.tsx+++ b/src/client/src/app/(playground)/settings/evaluation/page.tsx@@ -1,4 +1,5 @@"use client";+import { useEffect } from "react";import { usePostHog } from "posthog-js/react";import { CLIENT_EVENTS } from "@/constants/events";@@ -10,8 +11,8 @@ export default function SettingsEvaluationPage() {useEffect(() => {posthog?.capture(CLIENT_EVENTS.SETTINGS_EVALUATION_PAGE_VISITED);- router.replace("/evaluations/settings");- }, []);+ router.replace("/evaluations?tab=configuration");+ }, [posthog, router]);return null;}diff --git a/src/client/src/app/(playground)/telemetry/page.tsx b/src/client/src/app/(playground)/telemetry/page.tsxindex 06c2aba5..4e0a8b5f 100644--- a/src/client/src/app/(playground)/telemetry/page.tsx+++ b/src/client/src/app/(playground)/telemetry/page.tsx@@ -14,6 +14,7 @@ import { prepareObservabilitySignalChange } from "@/helpers/client/observabilityimport { getUpdateConfig, getUpdateFilter } from "@/selectors/filter";import { useRootStore } from "@/store";import FeaturePageHeader from "@/components/(playground)/feature-page-header";+import getMessage from "@/constants/messages";export default function TelemetryPage() {const searchParams = useSearchParams();@@ -59,10 +60,10 @@ export default function TelemetryPage() {return (<div className="flex h-full w-full flex-col overflow-hidden"><FeaturePageHeader- eyebrow={"Telemetry"}+ eyebrow={getMessage().SIDEBAR_MONITOR}title={activeConfig.label}icon={<ActiveIcon className="h-4 w-4" />}- tone={`rounded-md border p-1.5 ${activeConfig.tone}`}+ tone={activeConfig.tone}actions={(<div className="grid grid-cols-2 gap-2 md:flex md:flex-wrap md:justify-end">{/* The Coding Sessions + Coding Users signaldiff --git a/src/client/src/app/api/evaluation/analytics/[evaluatorId]/route.ts b/src/client/src/app/api/evaluation/analytics/[evaluatorId]/route.tsnew file mode 100644index 00000000..f5745a5d--- /dev/null+++ b/src/client/src/app/api/evaluation/analytics/[evaluatorId]/route.ts@@ -0,0 +1,47 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getEvaluationEvaluatorAnalytics } from "@/lib/platform/evaluation/analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";+import { NextRequest } from "next/server";++export async function POST(+ request: NextRequest,+ { params }: { params: { evaluatorId: string } }+) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const evaluatorId = decodeURIComponent(params.evaluatorId || "").trim();+ if (!evaluatorId) {+ return Response.json("Evaluator id missing!", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const metricParams: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ metricParams,+ validateMetricsRequestType.GET_EVALUATION_ANALYTICS+ );++ if (!validationParam.success) {+ return Response.json(validationParam.err, {+ status: 400,+ });+ }++ const res = await getEvaluationEvaluatorAnalytics(metricParams, evaluatorId);+ if (res.configured && !res.found) {+ return Response.json(res, { status: 404 });+ }+ return Response.json(res);+}diff --git a/src/client/src/app/api/evaluation/analytics/route.ts b/src/client/src/app/api/evaluation/analytics/route.tsnew file mode 100644index 00000000..56d5daec--- /dev/null+++ b/src/client/src/app/api/evaluation/analytics/route.ts@@ -0,0 +1,36 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getEvaluationAnalytics } from "@/lib/platform/evaluation/analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";+import { NextRequest } from "next/server";++export async function POST(request: NextRequest) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.GET_EVALUATION_ANALYTICS+ );++ if (!validationParam.success) {+ return Response.json(validationParam.err, {+ status: 400,+ });+ }++ const res = await getEvaluationAnalytics(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/cost/summary/route.ts b/src/client/src/app/api/metrics/cost/summary/route.tsnew file mode 100644index 00000000..e57f3251--- /dev/null+++ b/src/client/src/app/api/metrics/cost/summary/route.ts@@ -0,0 +1,34 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getCostSummary } from "@/lib/platform/cost/summary";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.COST_SUMMARY+ );++ if (!validationParam.success)+ return Response.json(validationParam.err, {+ status: 400,+ });++ const res = await getCostSummary(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/llm/cost/model/route.ts b/src/client/src/app/api/metrics/llm/cost/model/route.tsnew file mode 100644index 00000000..c5c0b9b3--- /dev/null+++ b/src/client/src/app/api/metrics/llm/cost/model/route.ts@@ -0,0 +1,34 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getCostByModel } from "@/lib/platform/llm/cost";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.COST_BY_MODEL+ );++ if (!validationParam.success)+ return Response.json(validationParam.err, {+ status: 400,+ });++ const res = await getCostByModel(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/llm/cost/provider/route.ts b/src/client/src/app/api/metrics/llm/cost/provider/route.tsnew file mode 100644index 00000000..bd33a2ea--- /dev/null+++ b/src/client/src/app/api/metrics/llm/cost/provider/route.ts@@ -0,0 +1,34 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getCostByProvider } from "@/lib/platform/llm/cost";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.COST_BY_PROVIDER+ );++ if (!validationParam.success)+ return Response.json(validationParam.err, {+ status: 400,+ });++ const res = await getCostByProvider(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/llm/cost/time/route.ts b/src/client/src/app/api/metrics/llm/cost/time/route.tsnew file mode 100644index 00000000..1db46887--- /dev/null+++ b/src/client/src/app/api/metrics/llm/cost/time/route.ts@@ -0,0 +1,34 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getCostPerTime } from "@/lib/platform/llm/cost";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.COST_PER_TIME+ );++ if (!validationParam.success)+ return Response.json(validationParam.err, {+ status: 400,+ });++ const res = await getCostPerTime(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/openground/cost/provider/route.ts b/src/client/src/app/api/metrics/openground/cost/provider/route.tsnew file mode 100644index 00000000..2e682412--- /dev/null+++ b/src/client/src/app/api/metrics/openground/cost/provider/route.ts@@ -0,0 +1,34 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getOpengroundCostByProvider } from "@/lib/platform/openground/cost-analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.OPENGROUND_COST_ANALYTICS+ );++ if (!validationParam.success)+ return Response.json(validationParam.err, {+ status: 400,+ });++ const res = await getOpengroundCostByProvider(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/openground/cost/route.ts b/src/client/src/app/api/metrics/openground/cost/route.tsnew file mode 100644index 00000000..f858c115--- /dev/null+++ b/src/client/src/app/api/metrics/openground/cost/route.ts@@ -0,0 +1,34 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getOpengroundCostAnalytics } from "@/lib/platform/openground/cost-analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.OPENGROUND_COST_ANALYTICS+ );++ if (!validationParam.success)+ return Response.json(validationParam.err, {+ status: 400,+ });++ const res = await getOpengroundCostAnalytics(params);+ return Response.json(res);+}diff --git a/src/client/src/app/api/metrics/request/attribute-keys/route.ts b/src/client/src/app/api/metrics/request/attribute-keys/route.tsindex 0dff24db..d5b3bdb9 100644--- a/src/client/src/app/api/metrics/request/attribute-keys/route.ts+++ b/src/client/src/app/api/metrics/request/attribute-keys/route.ts@@ -13,6 +13,7 @@ async function POSTHandler(request: Request) {const params: MetricParams = {timeLimit,selectedConfig: formData.selectedConfig,+ statusCode: formData.statusCode,};const validationParam = validateMetricsRequest(diff --git a/src/client/src/app/api/metrics/request/config/route.ts b/src/client/src/app/api/metrics/request/config/route.tsindex 3e54c78a..e3fbc893 100644--- a/src/client/src/app/api/metrics/request/config/route.ts+++ b/src/client/src/app/api/metrics/request/config/route.ts@@ -17,6 +17,7 @@ async function POSTHandler(request: Request) {limit,offset,selectedConfig: formData.selectedConfig,+ statusCode: formData.statusCode,};const validationParam = validateMetricsRequest(diff --git a/src/client/src/app/api/pricing/cron-analytics/route.ts b/src/client/src/app/api/pricing/cron-analytics/route.tsnew file mode 100644index 00000000..8990fb45--- /dev/null+++ b/src/client/src/app/api/pricing/cron-analytics/route.ts@@ -0,0 +1,35 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getPricingCronAnalytics } from "@/lib/platform/pricing/cron-analytics";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.TOTAL_COST+ );++ if (!validationParam.success) {+ return Response.json(validationParam.err, { status: 400 });+ }++ const limit =+ typeof formData.limit === "number" ? formData.limit : undefined;+ const res = await getPricingCronAnalytics(params, { limit });+ return Response.json(res);+}diff --git a/src/client/src/app/api/pricing/guidance/route.ts b/src/client/src/app/api/pricing/guidance/route.tsnew file mode 100644index 00000000..6f0ac7e6--- /dev/null+++ b/src/client/src/app/api/pricing/guidance/route.ts@@ -0,0 +1,33 @@+import { MetricParams, TimeLimit } from "@/lib/platform/common";+import { getCostPricingGuidance } from "@/lib/platform/pricing/guidance";+import {+ validateMetricsRequest,+ validateMetricsRequestType,+} from "@/helpers/server/platform";++export async function POST(request: Request) {+ let formData: Record<string, unknown>;+ try {+ formData = await request.json();+ } catch {+ return Response.json("Invalid JSON body", { status: 400 });+ }++ const timeLimit = formData.timeLimit as TimeLimit;+ const params: MetricParams = {+ timeLimit,+ selectedConfig: formData.selectedConfig,+ };++ const validationParam = validateMetricsRequest(+ params,+ validateMetricsRequestType.TOTAL_COST+ );++ if (!validationParam.success) {+ return Response.json(validationParam.err, { status: 400 });+ }++ const data = await getCostPricingGuidance(params);+ return Response.json({ data });+}diff --git a/src/client/src/clickhouse/seed-data/openlit-dashboard-LLM-dashboard-layout.json b/src/client/src/clickhouse/seed-data/openlit-dashboard-LLM-dashboard-layout.jsonindex 230bf1af..6edd243f 100644--- a/src/client/src/clickhouse/seed-data/openlit-dashboard-LLM-dashboard-layout.json+++ b/src/client/src/clickhouse/seed-data/openlit-dashboard-LLM-dashboard-layout.json@@ -36,7 +36,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\t\nSELECT\n\t\t\tSpanAttributes['gen_ai.request.model'] AS model,\n\t\t\tCAST(COUNT(SpanAttributes['gen_ai.request.model']) AS INTEGER) AS model_count\n\t\tFROM\n\t\t\totel_traces\n\t\tWHERE \n\t\tSpanAttributes['gen_ai.operation.name'] != 'vectordb'\n\t\tAND notEmpty(SpanAttributes['gen_ai.request.model'])\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY\n\t\t\tmodel\n\t\tORDER BY\n\t\t\tmodel_count DESC\n\t\tLIMIT 3"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\t\nSELECT\n\t\t\tSpanAttributes['gen_ai.request.model'] AS model,\n\t\t\tCAST(COUNT(SpanAttributes['gen_ai.request.model']) AS INTEGER) AS model_count\n\t\tFROM\n\t\t\totel_traces\n\t\tWHERE \n\t\tSpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n\t\tAND notEmpty(SpanAttributes['gen_ai.request.model'])\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY\n\t\t\tmodel\n\t\tORDER BY\n\t\t\tmodel_count DESC\n\t\tLIMIT 3"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -53,7 +53,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])), 5) AS completion_tokens,\n\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])), 5) AS completion_tokens_previous,\n\n round(\n if(\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])) = 0,\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])) * 100.0,\n (\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens']))\n -\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens']))\n )\n /\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])), 5) AS completion_tokens,\n\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])), 5) AS completion_tokens_previous,\n\n round(\n if(\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])) = 0,\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])) * 100.0,\n (\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens']))\n -\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens']))\n )\n /\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.output_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.output_tokens'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -69,7 +69,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT \n SpanAttributes['gen_ai.system'] AS provider,\n CAST(COUNT(*) AS INTEGER) AS count\n FROM\n otel_traces\n WHERE \n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND notEmpty(SpanAttributes['gen_ai.system'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= start_time\n AND Timestamp <= end_time\n GROUP BY provider;"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT \n if(notEmpty(SpanAttributes['gen_ai.provider.name']), SpanAttributes['gen_ai.provider.name'], SpanAttributes['gen_ai.system']) AS provider,\n CAST(COUNT(*) AS INTEGER) AS count\n FROM\n otel_traces\n WHERE \n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND (notEmpty(SpanAttributes['gen_ai.provider.name']) OR notEmpty(SpanAttributes['gen_ai.system']))\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= start_time\n AND Timestamp <= end_time\n GROUP BY provider;"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -94,7 +94,7 @@]},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\n\tSELECT\n\t\tCAST(SUM(toInt64OrZero(SpanAttributes['gen_ai.usage.total_tokens'])) AS INTEGER) AS total_tokens,\n\t\tCAST(SUM(toInt64OrZero(SpanAttributes['gen_ai.usage.input_tokens'])) AS INTEGER) AS prompt_tokens,\n\t\tCAST(SUM(toInt64OrZero(SpanAttributes['gen_ai.usage.output_tokens'])) AS INTEGER) AS completion_tokens,\n\t\tformatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\n\t\tFROM otel_traces \n\t\tWHERE \n\t\tSpanAttributes['gen_ai.operation.name'] != 'vectordb'\n\t\tAND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY request_time\n\t\tORDER BY request_time;"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\n\tSELECT\n\t\tCAST(SUM(toInt64OrZero(SpanAttributes['gen_ai.usage.total_tokens'])) AS INTEGER) AS total_tokens,\n\t\tCAST(SUM(toInt64OrZero(SpanAttributes['gen_ai.usage.input_tokens'])) AS INTEGER) AS prompt_tokens,\n\t\tCAST(SUM(toInt64OrZero(SpanAttributes['gen_ai.usage.output_tokens'])) AS INTEGER) AS completion_tokens,\n\t\tformatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\n\t\tFROM otel_traces \n\t\tWHERE \n\t\tSpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n\t\tAND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY request_time\n\t\tORDER BY request_time;"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -110,7 +110,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\nSELECT\n CAST(COUNT(*) AS INTEGER) AS total,\n formatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n\t\tAND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= start_time AND Timestamp <= end_time\nGROUP BY request_time\nORDER BY request_time"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\nSELECT\n CAST(COUNT(*) AS INTEGER) AS total,\n formatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n\t\tAND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= start_time AND Timestamp <= end_time\nGROUP BY request_time\nORDER BY request_time"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -126,7 +126,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT \n\t\t\tDISTINCT ResourceAttributes['service.name'] as applicationName, \n\t\t\tROUND(SUM(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost'])), 5) AS cost\n\t\tFROM otel_traces \n\t\tWHERE \n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']) \n AND notEmpty(ResourceAttributes['service.name'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY applicationName;"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT \n\t\t\tDISTINCT ResourceAttributes['service.name'] as applicationName, \n\t\t\tROUND(SUM(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost'])), 5) AS cost\n\t\tFROM otel_traces \n\t\tWHERE \n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']) \n AND notEmpty(ResourceAttributes['service.name'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY applicationName;"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -142,7 +142,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\nSELECT \n ARRAY_AGG(model) AS models,\n ARRAY_AGG(model_count) AS model_counts,\n CAST(SUM(model_count) AS INTEGER) AS total_model_count,\n request_time\nFROM (\n SELECT\n SpanAttributes['gen_ai.request.model'] AS model,\n COUNT(*) AS model_count,\n formatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\n FROM otel_traces\n WHERE\n Timestamp >= start_time AND Timestamp <= end_time\n AND SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND notEmpty(SpanAttributes['gen_ai.request.model'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n GROUP BY\n model, request_time\n) AS sub\nGROUP BY\n request_time\nORDER BY\n request_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\nSELECT \n ARRAY_AGG(model) AS models,\n ARRAY_AGG(model_count) AS model_counts,\n CAST(SUM(model_count) AS INTEGER) AS total_model_count,\n request_time\nFROM (\n SELECT\n SpanAttributes['gen_ai.request.model'] AS model,\n COUNT(*) AS model_count,\n formatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\n FROM otel_traces\n WHERE\n Timestamp >= start_time AND Timestamp <= end_time\n AND SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND notEmpty(SpanAttributes['gen_ai.request.model'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n GROUP BY\n model, request_time\n) AS sub\nGROUP BY\n request_time\nORDER BY\n request_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -161,7 +161,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time), 7) AS total_usage_cost,\n\n round(sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time), 7) AS total_usage_cost_previous,\n\n round(\n if(\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time) = 0,\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time) * 100.0,\n (\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time)\n -\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time)\n )\n /\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time), 7) AS total_usage_cost,\n\n round(sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time), 7) AS total_usage_cost_previous,\n\n round(\n if(\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time) = 0,\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time) * 100.0,\n (\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time)\n -\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time)\n )\n /\n sumIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -195,7 +195,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])), 5) AS prompt_tokens,\n\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])), 5) AS prompt_tokens_previous,\n\n round(\n if(\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])) = 0,\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])) * 100.0,\n (\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens']))\n -\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens']))\n )\n /\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])), 5) AS prompt_tokens,\n\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])), 5) AS prompt_tokens_previous,\n\n round(\n if(\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])) = 0,\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])) * 100.0,\n (\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens']))\n -\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens']))\n )\n /\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.input_tokens']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.input_tokens'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -229,7 +229,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= start_time AND Timestamp <= end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])), 4) AS total_tokens,\n\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])), 4) AS total_tokens_previous,\n\n round(\n if(\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])) = 0,\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= start_time AND Timestamp <= end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])) * 100.0,\n (\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= start_time AND Timestamp <= end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens']))\n -\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens']))\n )\n /\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= start_time AND Timestamp <= end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])), 4) AS total_tokens,\n\n round(avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])), 4) AS total_tokens_previous,\n\n round(\n if(\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])) = 0,\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= start_time AND Timestamp <= end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])) * 100.0,\n (\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= start_time AND Timestamp <= end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens']))\n -\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens']))\n )\n /\n avgIf(toInt32OrZero(SpanAttributes['gen_ai.usage.total_tokens']), \n Timestamp >= prev_start_time AND Timestamp <= prev_end_time \n AND notEmpty(SpanAttributes['gen_ai.usage.total_tokens'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -245,7 +245,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT \n\t\t\tDISTINCT ResourceAttributes['deployment.environment'] as environment, \n\t\t\tSUM(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost'])) AS cost\n\t\tFROM otel_traces \n\t\tWHERE \n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']) \n AND notEmpty(ResourceAttributes['deployment.environment'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY environment;"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT \n\t\t\tDISTINCT ResourceAttributes['deployment.environment'] as environment, \n\t\t\tSUM(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost'])) AS cost\n\t\tFROM otel_traces \n\t\tWHERE \n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']) \n AND notEmpty(ResourceAttributes['deployment.environment'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n\t\tAND Timestamp >= start_time\n AND Timestamp <= end_time\n\t\tGROUP BY environment;"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -263,7 +263,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])), 7) AS avg_usage_cost,\n\n round(avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])), 7) AS avg_usage_cost_previous,\n\n round(\n if(\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])) = 0,\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])) * 100.0,\n (\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']))\n -\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']))\n )\n /\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])), 7) AS avg_usage_cost,\n\n round(avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])), 7) AS avg_usage_cost_previous,\n\n round(\n if(\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])) = 0,\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])) * 100.0,\n (\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= start_time AND Timestamp <= end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']))\n -\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost']))\n )\n /\n avgIf(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']),\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n AND notEmpty(SpanAttributes['gen_ai.usage.cost'])) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -279,7 +279,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT\n SpanAttributes['gen_ai.operation.name'] AS category,\n CAST(COUNT(*) AS INTEGER) AS count\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND notEmpty(SpanAttributes['gen_ai.operation.name'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= start_time\n AND Timestamp <= end_time\nGROUP BY category"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time\n\nSELECT\n SpanAttributes['gen_ai.operation.name'] AS category,\n CAST(COUNT(*) AS INTEGER) AS count\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND notEmpty(SpanAttributes['gen_ai.operation.name'])\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= start_time\n AND Timestamp <= end_time\nGROUP BY category"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -296,7 +296,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n CAST(countIf(\n Timestamp >= start_time AND Timestamp <= end_time\n ) AS INTEGER) AS total_request,\n\n CAST(countIf(\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n ) AS INTEGER) AS total_request_previous,\n\n round(\n if(countIf(Timestamp >= prev_start_time AND Timestamp <= prev_end_time) = 0, \n (countIf(Timestamp >= start_time AND Timestamp <= end_time) * 100.0), \n ((countIf(Timestamp >= start_time AND Timestamp <= end_time) - countIf(Timestamp >= prev_start_time AND Timestamp <= prev_end_time)) \n / countIf(Timestamp >= prev_start_time AND Timestamp <= prev_end_time))*100.0\n ), 4\n ) AS rate\n\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time;"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n CAST(countIf(\n Timestamp >= start_time AND Timestamp <= end_time\n ) AS INTEGER) AS total_request,\n\n CAST(countIf(\n Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n ) AS INTEGER) AS total_request_previous,\n\n round(\n if(countIf(Timestamp >= prev_start_time AND Timestamp <= prev_end_time) = 0, \n (countIf(Timestamp >= start_time AND Timestamp <= end_time) * 100.0), \n ((countIf(Timestamp >= start_time AND Timestamp <= end_time) - countIf(Timestamp >= prev_start_time AND Timestamp <= prev_end_time)) \n / countIf(Timestamp >= prev_start_time AND Timestamp <= prev_end_time))*100.0\n ), 4\n ) AS rate\n\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time;"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -314,7 +314,7 @@"trendSuffix": "%"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(multiply(\n avgIf(Duration, Timestamp >= start_time AND Timestamp <= end_time),\n 10e-10\n ), 4) AS average_duration,\n\n round(multiply(\n avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time),\n 10e-10\n ), 4) AS average_duration_previous,\n\n round(\n if(\n avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time) = 0,\n avgIf(Duration, Timestamp >= start_time AND Timestamp <= end_time) * 100.0,\n (\n avgIf(Duration, Timestamp >= start_time AND Timestamp <= end_time) -\n avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time)\n ) / avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND isFinite(Duration)\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time\n\nSELECT\n round(multiply(\n avgIf(Duration, Timestamp >= start_time AND Timestamp <= end_time),\n 10e-10\n ), 4) AS average_duration,\n\n round(multiply(\n avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time),\n 10e-10\n ), 4) AS average_duration_previous,\n\n round(\n if(\n avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time) = 0,\n avgIf(Duration, Timestamp >= start_time AND Timestamp <= end_time) * 100.0,\n (\n avgIf(Duration, Timestamp >= start_time AND Timestamp <= end_time) -\n avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time)\n ) / avgIf(Duration, Timestamp >= prev_start_time AND Timestamp <= prev_end_time) * 100.0\n ), 4\n ) AS rate\nFROM otel_traces\nWHERE\n SpanAttributes['gen_ai.operation.name'] != 'vectordb'\n AND empty(SpanAttributes['coding_agent.client'])\n AND empty(ResourceAttributes['coding_agent.client'])\n AND empty(SpanAttributes['coding_agent.session.id'])\n AND empty(ResourceAttributes['coding_agent.session.id'])\n AND (empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')\n AND NOT startsWith(SpanName, 'coding_agent.')\n AND isFinite(Duration)\n AND StatusCode IN ('STATUS_CODE_OK', 'STATUS_CODE_UNSET', 'Ok', 'Unset')\n AND Timestamp >= prev_start_time AND Timestamp <= end_time\n"},"createdAt": "2025-06-16 07:39:17","updatedAt": "2025-06-16 07:39:17"@@ -451,4 +451,4 @@}]}-}\ No newline at end of file+}diff --git a/src/client/src/clickhouse/seed-data/openlit-dashboard-coding-agents-layout.json b/src/client/src/clickhouse/seed-data/openlit-dashboard-coding-agents-layout.jsonindex 2d16125a..ea476e90 100644--- a/src/client/src/clickhouse/seed-data/openlit-dashboard-coding-agents-layout.json+++ b/src/client/src/clickhouse/seed-data/openlit-dashboard-coding-agents-layout.json@@ -1,7 +1,7 @@{"id": "8f4d7b2e-6a91-4c52-9b3a-1e6c8f3d4a5b","title": "Coding Agents",- "description": "Coding-agent observability across vendors (Claude Code, Cursor, Codex). Sessions, cost, tools, users, classification, and governance — sourced live from otel_traces emitted by the openlit CLI hook.",+ "description": "Coding-agent observability across vendors (Claude Code, Cursor, Codex). Sessions, cost, tools, users, classification, and governance \u2014 sourced live from otel_traces emitted by the openlit CLI hook.","parentId": null,"isMainDashboard": false,"isPinned": true,@@ -11,7 +11,7 @@"9a1f1c20-3b27-4f88-9d51-1aaa11111111": {"id": "9a1f1c20-3b27-4f88-9d51-1aaa11111111","title": "Total sessions",- "description": "Distinct coding-agent sessions in the selected window. Optionally vendor- and user-scoped to match what the agents-hub card and the Sessions / Users tabs show.",+ "description": "Distinct coding-agent chats in the selected window (linked subagents fold under their parent; orphan Task sessions without parent_conversation_id still count separately). Matches the Sessions tab. Optionally vendor- and user-scoped.","type": "STAT_CARD","properties": {"value": "0.total_sessions",@@ -20,7 +20,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_session_now AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_session_prev AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(count()) FROM per_session_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(count()) FROM per_session_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(now_total AS INTEGER) AS total_sessions,\n CAST(prev_total AS INTEGER) AS total_sessions_previous,\n if(prev_total > 0, round((now_total - prev_total) * 100.0 / prev_total, 1), NULL) AS rate\nFROM totals\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_session_now AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_session_prev AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(count()) FROM per_session_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(count()) FROM per_session_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(now_total AS INTEGER) AS total_sessions,\n CAST(prev_total AS INTEGER) AS total_sessions_previous,\n if(prev_total > 0, round((now_total - prev_total) * 100.0 / prev_total, 1), NULL) AS rate\nFROM totals\n"},"createdAt": "2026-05-25 11:00:00","updatedAt": "2026-05-28 03:30:00"@@ -28,7 +28,7 @@"9a1f1c20-3b27-4f88-9d51-1aaa11111112": {"id": "9a1f1c20-3b27-4f88-9d51-1aaa11111112","title": "Active users",- "description": "Distinct users in the selected window, optionally vendor-scoped (gen_ai.user.name → service.name).",+ "description": "Distinct users in the selected window, optionally vendor-scoped (gen_ai.user.name \u2192 service.name).","type": "STAT_CARD","properties": {"value": "0.active_users",@@ -37,7 +37,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n -- Group by chat_id (= parent_id ?? session_id) so subagent\n -- spawns don't inflate the user count. Matches the\n -- CHAT_ID_EXPR used by the Total Sessions / Total Cost\n -- widgets and the Users tab so all surfaces agree.\n per_session_now AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_session_prev AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT uniqExact(user) FROM per_session_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT uniqExact(user) FROM per_session_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(now_total AS INTEGER) AS active_users,\n if(prev_total > 0, round((now_total - prev_total) * 100.0 / prev_total, 1), NULL) AS rate\nFROM totals\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n -- Group by chat_id (= parent_id ?? session_id) so subagent\n -- spawns don't inflate the user count. Matches the\n -- CHAT_ID_EXPR used by the Total Sessions / Total Cost\n -- widgets and the Users tab so all surfaces agree.\n per_session_now AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_session_prev AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT uniqExact(user) FROM per_session_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT uniqExact(user) FROM per_session_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(now_total AS INTEGER) AS active_users,\n if(prev_total > 0, round((now_total - prev_total) * 100.0 / prev_total, 1), NULL) AS rate\nFROM totals\n"},"createdAt": "2026-05-25 11:00:00","updatedAt": "2026-05-28 03:30:00"@@ -55,7 +55,7 @@"prefix": "$"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n -- Canonical per-session cost. Same formula listSessions /\n -- listCodingUsers / the agents-hub materializer use. Without\n -- the greatest() this widget silently under-counts: sessions\n -- whose only cost signal is the session-end stamp\n -- (coding_agent.session.cost_usd, set on Stop / SessionEnd by\n -- Claude Code and Codex) would report $0 because no per-turn\n -- gen_ai.usage.cost was ever emitted.\n cost_now AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toFloat64OrZero(any(SpanAttributes['coding_agent.session.cost_usd'])),\n sumOrNull(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']))\n ) AS session_cost\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n cost_prev AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toFloat64OrZero(any(SpanAttributes['coding_agent.session.cost_usd'])),\n sumOrNull(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']))\n ) AS session_cost\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT round(sumOrNull(session_cost), 4) FROM cost_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT round(sumOrNull(session_cost), 4) FROM cost_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n coalesce(now_total, 0) AS total_cost,\n coalesce(prev_total, 0) AS total_cost_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n -- Canonical per-session cost. Same formula listSessions /\n -- listCodingUsers / the agents-hub materializer use. Without\n -- the greatest() this widget silently under-counts: sessions\n -- whose only cost signal is the session-end stamp\n -- (coding_agent.session.cost_usd, set on Stop / SessionEnd by\n -- Claude Code and Codex) would report $0 because no per-turn\n -- gen_ai.usage.cost was ever emitted.\n cost_now AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toFloat64OrZero(any(SpanAttributes['coding_agent.session.cost_usd'])),\n sumOrNull(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']))\n ) AS session_cost\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n cost_prev AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toFloat64OrZero(any(SpanAttributes['coding_agent.session.cost_usd'])),\n sumOrNull(toFloat64OrZero(SpanAttributes['gen_ai.usage.cost']))\n ) AS session_cost\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT round(sumOrNull(session_cost), 4) FROM cost_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT round(sumOrNull(session_cost), 4) FROM cost_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n coalesce(now_total, 0) AS total_cost,\n coalesce(prev_total, 0) AS total_cost_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"},"createdAt": "2026-05-25 11:00:00","updatedAt": "2026-05-28 03:30:00"@@ -80,7 +80,7 @@"9a1f1c20-3b27-4f88-9d51-1aaa22222221": {"id": "9a1f1c20-3b27-4f88-9d51-1aaa22222221","title": "Sessions over time",- "description": "Distinct coding-agent sessions bucketed across the window. Optionally vendor- and user-scoped.",+ "description": "Distinct coding-agent chats (parent fold + conversation id) bucketed across the window. Matches Total sessions / the Sessions tab. Optionally vendor- and user-scoped.","type": "AREA_CHART","properties": {"xAxis": "request_time",@@ -92,7 +92,7 @@]},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\nSELECT\n CAST(uniq(coalesce(\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n nullIf(SpanAttributes['session.id'], ''),\n nullIf(ResourceAttributes['session.id'], '')\n )) AS INTEGER) AS sessions,\n formatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\nFROM otel_traces\nWHERE\n (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n {{#filter.vendor}}AND coalesce(nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], '')) = '{{filter.vendor}}'{{/filter.vendor}}\n {{#filter.user}}AND coalesce(\n nullIf(SpanAttributes['gen_ai.user.name'], ''),\n nullIf(ResourceAttributes['gen_ai.user.name'], ''),\n nullIf(ResourceAttributes['service.name'], ''),\n 'unknown'\n ) = '{{filter.user}}'{{/filter.user}}\nGROUP BY request_time\nORDER BY request_time"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n dateDiff('day', start_time, end_time) AS days_diff,\n dateDiff('year', start_time, end_time) AS years_diff,\n multiIf(\n years_diff >= 1, 'month',\n days_diff <= 1, 'hour',\n 'day'\n ) AS date_granularity\n\nSELECT\n CAST(uniq(coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n )) AS INTEGER) AS sessions,\n formatDateTime(DATE_TRUNC(date_granularity, Timestamp), '%Y/%m/%d %R') AS request_time\nFROM otel_traces\nWHERE\n (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n {{#filter.vendor}}AND coalesce(nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], '')) = '{{filter.vendor}}'{{/filter.vendor}}\n {{#filter.user}}AND coalesce(\n nullIf(SpanAttributes['gen_ai.user.name'], ''),\n nullIf(ResourceAttributes['gen_ai.user.name'], ''),\n nullIf(ResourceAttributes['service.name'], ''),\n 'unknown'\n ) = '{{filter.user}}'{{/filter.user}}\nGROUP BY request_time\nORDER BY request_time"},"createdAt": "2026-05-25 11:00:00","updatedAt": "2026-05-28 03:35:00"@@ -108,7 +108,7 @@"color": "#F36C06"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n per_session AS (\n SELECT\n coalesce(\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n nullIf(SpanAttributes['session.id'], ''),\n nullIf(ResourceAttributes['session.id'], '')\n ) AS sid,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY sid, vendor\n )\n\nSELECT vendor, CAST(count() AS INTEGER) AS sessions\nFROM per_session\nWHERE 1=1{{#filter.vendor}}\n AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}}\n AND user = '{{filter.user}}'{{/filter.user}}\nGROUP BY vendor\nORDER BY sessions DESC"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n per_session AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS sid,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY sid, vendor\n )\n\nSELECT vendor, CAST(count() AS INTEGER) AS sessions\nFROM per_session\nWHERE 1=1{{#filter.vendor}}\n AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}}\n AND user = '{{filter.user}}'{{/filter.user}}\nGROUP BY vendor\nORDER BY sessions DESC"},"createdAt": "2026-05-25 11:00:00","updatedAt": "2026-05-28 03:30:00"@@ -132,7 +132,7 @@"9a1f1c20-3b27-4f88-9d51-1aaa33333332": {"id": "9a1f1c20-3b27-4f88-9d51-1aaa33333332","title": "Top users",- "description": "Top users by session count, optionally vendor-scoped (gen_ai.user.name → service.name).",+ "description": "Top users by session count, optionally vendor-scoped (gen_ai.user.name \u2192 service.name).","type": "BAR_CHART","properties": {"xAxis": "user",@@ -196,7 +196,7 @@"9a1f1c20-3b27-4f88-9d51-1aaa55555551": {"id": "9a1f1c20-3b27-4f88-9d51-1aaa55555551","title": "Session outcomes",- "description": "Distribution of session outcomes — merged, committed, abandoned, cancelled — optionally vendor-scoped.",+ "description": "Distribution of session outcomes \u2014 merged, committed, abandoned, cancelled \u2014 optionally vendor-scoped.","type": "PIE_CHART","properties": {"labelPath": "outcome",@@ -212,7 +212,7 @@"9a1f1c20-3b27-4f88-9d51-1aaa55555552": {"id": "9a1f1c20-3b27-4f88-9d51-1aaa55555552","title": "Edit decisions",- "description": "Auto-applied vs user-reviewed edits — gauge user trust in the agent. Optionally vendor- and user-scoped.",+ "description": "Auto-applied vs user-reviewed edits \u2014 gauge user trust in the agent. Optionally vendor- and user-scoped.","type": "PIE_CHART","properties": {"labelPath": "decision",@@ -253,7 +253,7 @@"color": "#10b981"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_chat_now AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.lines.accepted'])),\n toInt64(sumIf(\n toInt64OrZero(SpanAttributes['coding_agent.edit.lines.added']),\n SpanName = 'coding_agent.edit.decision'\n AND SpanAttributes['coding_agent.edit.decision'] IN ('accept', 'auto_accepted')\n ))\n ) AS lines_accepted\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_chat_prev AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.lines.accepted'])),\n toInt64(sumIf(\n toInt64OrZero(SpanAttributes['coding_agent.edit.lines.added']),\n SpanName = 'coding_agent.edit.decision'\n AND SpanAttributes['coding_agent.edit.decision'] IN ('accept', 'auto_accepted')\n ))\n ) AS lines_accepted\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(sumOrNull(lines_accepted)) FROM per_chat_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(sumOrNull(lines_accepted)) FROM per_chat_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(coalesce(now_total, 0) AS INTEGER) AS lines_accepted,\n CAST(coalesce(prev_total, 0) AS INTEGER) AS lines_accepted_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_chat_now AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.lines.accepted'])),\n toInt64(sumIf(\n toInt64OrZero(SpanAttributes['coding_agent.edit.lines.added']),\n SpanName = 'coding_agent.edit.decision'\n AND SpanAttributes['coding_agent.edit.decision'] IN ('accept', 'auto_accepted')\n ))\n ) AS lines_accepted\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_chat_prev AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.lines.accepted'])),\n toInt64(sumIf(\n toInt64OrZero(SpanAttributes['coding_agent.edit.lines.added']),\n SpanName = 'coding_agent.edit.decision'\n AND SpanAttributes['coding_agent.edit.decision'] IN ('accept', 'auto_accepted')\n ))\n ) AS lines_accepted\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(sumOrNull(lines_accepted)) FROM per_chat_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(sumOrNull(lines_accepted)) FROM per_chat_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(coalesce(now_total, 0) AS INTEGER) AS lines_accepted,\n CAST(coalesce(prev_total, 0) AS INTEGER) AS lines_accepted_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"},"createdAt": "2026-05-28 11:00:00","updatedAt": "2026-05-28 11:00:00"@@ -288,7 +288,7 @@"color": "#6366f1"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_chat_now AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.commit_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.commit'))\n ) AS commits\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_chat_prev AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.commit_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.commit'))\n ) AS commits\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(sumOrNull(commits)) FROM per_chat_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(sumOrNull(commits)) FROM per_chat_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(coalesce(now_total, 0) AS INTEGER) AS commits,\n CAST(coalesce(prev_total, 0) AS INTEGER) AS commits_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_chat_now AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.commit_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.commit'))\n ) AS commits\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_chat_prev AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.commit_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.commit'))\n ) AS commits\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(sumOrNull(commits)) FROM per_chat_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(sumOrNull(commits)) FROM per_chat_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(coalesce(now_total, 0) AS INTEGER) AS commits,\n CAST(coalesce(prev_total, 0) AS INTEGER) AS commits_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"},"createdAt": "2026-05-28 11:00:00","updatedAt": "2026-05-28 11:00:00"@@ -305,7 +305,7 @@"color": "#6366f1"},"config": {- "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_chat_now AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.pr_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.pull_request'))\n ) AS prs\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_chat_prev AS (\n SELECT\n coalesce(\n nullIf(ResourceAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.agent.parent_id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.pr_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.pull_request'))\n ) AS prs\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(sumOrNull(prs)) FROM per_chat_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(sumOrNull(prs)) FROM per_chat_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(coalesce(now_total, 0) AS INTEGER) AS prs,\n CAST(coalesce(prev_total, 0) AS INTEGER) AS prs_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"+ "query": "WITH\n parseDateTimeBestEffort('{{filter.timeLimit.start}}') AS start_time,\n parseDateTimeBestEffort('{{filter.timeLimit.end}}') AS end_time,\n (end_time - start_time) AS duration,\n (start_time - duration) AS prev_start_time,\n (end_time - duration) AS prev_end_time,\n per_chat_now AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.pr_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.pull_request'))\n ) AS prs\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= start_time AND Timestamp <= end_time\n GROUP BY chat_id, vendor\n ),\n per_chat_prev AS (\n SELECT\n coalesce(\n nullIf(\n if(\n notEmpty(ResourceAttributes['coding_agent.agent.parent_id'])\n AND ResourceAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n ResourceAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(\n if(\n notEmpty(SpanAttributes['coding_agent.agent.parent_id'])\n AND SpanAttributes['coding_agent.agent.parent_id'] != coalesce(\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], ''),\n ''\n ),\n SpanAttributes['coding_agent.agent.parent_id'],\n ''\n ),\n ''\n ),\n nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['gen_ai.conversation.id'], ''),\n nullIf(SpanAttributes['coding_agent.session.id'], ''),\n nullIf(ResourceAttributes['coding_agent.session.id'], '')\n ) AS chat_id,\n coalesce( nullIf(SpanAttributes['coding_agent.client'], ''), nullIf(SpanAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['gen_ai.agent.name'], ''), nullIf(ResourceAttributes['service.name'], ''), 'unknown' ) AS vendor,\n coalesce(\n nullIf(any(SpanAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['gen_ai.user.name']), ''),\n nullIf(any(ResourceAttributes['service.name']), ''),\n 'unknown'\n ) AS user,\n greatest(\n toInt64OrZero(any(SpanAttributes['coding_agent.session.pr_count'])),\n toInt64(countIf(SpanName = 'coding_agent.git.pull_request'))\n ) AS prs\n FROM otel_traces\n WHERE (\n notEmpty(SpanAttributes['coding_agent.session.id'])\n OR (ResourceAttributes['service.name'] = 'claude-code'\n AND (notEmpty(SpanAttributes['session.id']) OR notEmpty(ResourceAttributes['session.id'])))\n )\n AND Timestamp >= prev_start_time AND Timestamp <= prev_end_time\n GROUP BY chat_id, vendor\n ),\n totals AS (\n SELECT\n (SELECT toInt64(sumOrNull(prs)) FROM per_chat_now WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS now_total,\n (SELECT toInt64(sumOrNull(prs)) FROM per_chat_prev WHERE 1=1{{#filter.vendor}} AND vendor = '{{filter.vendor}}'{{/filter.vendor}}{{#filter.user}} AND user = '{{filter.user}}'{{/filter.user}}) AS prev_total\n )\n\nSELECT\n CAST(coalesce(now_total, 0) AS INTEGER) AS prs,\n CAST(coalesce(prev_total, 0) AS INTEGER) AS prs_previous,\n if(coalesce(prev_total, 0) > 0, round((coalesce(now_total, 0) - coalesce(prev_total, 0)) * 100.0 / coalesce(prev_total, 0), 1), NULL) AS rate\nFROM totals\n"},"createdAt": "2026-05-28 11:00:00","updatedAt": "2026-05-28 11:00:00"diff --git a/src/client/src/components/(playground)/agents/agent-header.tsx b/src/client/src/components/(playground)/agents/agent-header.tsxindex 556e9394..3bbfbbea 100644--- a/src/client/src/components/(playground)/agents/agent-header.tsx+++ b/src/client/src/components/(playground)/agents/agent-header.tsx@@ -93,17 +93,17 @@ export default function AgentHeader({isCoding && hasCodingAgentVendorIcon(agent.coding_agent_vendor) ? (<CodingAgentVendorIconvendor={agent.coding_agent_vendor}- className="h-5 w-5"+ className="h-4 w-4"/>) : (- <Bot className="h-5 w-5" />+ <Bot className="h-4 w-4" />);const backLabel = getMessage().AGENTS_BACK_TO_HUB;return (<FeaturePageHeader- eyebrow={getMessage().FEATURE_AGENTS}+ eyebrow={getMessage().SIDEBAR_MONITOR}title={title}icon={icon}tone={AGENTS_HEADER_TONE}@@ -112,7 +112,7 @@ export default function AgentHeader({asChildvariant="outline"size="sm"- className="h-8 w-8 shrink-0 p-0"+ className="h-7 w-7 shrink-0 p-0"><Link href={backHref} title={backLabel} aria-label={backLabel}><ArrowLeft className="h-3.5 w-3.5" />@@ -136,7 +136,7 @@ export default function AgentHeader({? getMessage().AGENTS_REFRESHING: getMessage().AGENTS_REFRESH}- className="inline-flex h-8 w-8 items-center justify-center rounded-md border border-stone-200 bg-stone-50 text-stone-600 transition hover:bg-stone-100 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300 dark:hover:bg-stone-800 disabled:opacity-50"+ className="inline-flex h-7 w-7 items-center justify-center rounded-md border border-stone-200 bg-stone-50 text-stone-600 transition hover:bg-stone-100 dark:border-stone-800 dark:bg-stone-900 dark:text-stone-300 dark:hover:bg-stone-800 disabled:opacity-50"><RefreshCwclassName={`h-3.5 w-3.5 ${isRefreshing ? "animate-spin" : ""}`}diff --git a/src/client/src/components/(playground)/chat/chat-settings-form.tsx b/src/client/src/components/(playground)/chat/chat-settings-form.tsxindex 44888426..be0da43c 100644--- a/src/client/src/components/(playground)/chat/chat-settings-form.tsx+++ b/src/client/src/components/(playground)/chat/chat-settings-form.tsx@@ -234,7 +234,7 @@ export default function ChatSettingsForm() {<p className="text-xs text-stone-400 dark:text-stone-500">{m.CHAT_SETTINGS_MODEL_HINT}{" "}<a- href="/manage-models"+ href="/costs?tab=models"className="underline hover:text-stone-600 dark:hover:text-stone-300">{m.CHAT_SETTINGS_MANAGE_MODELS}diff --git a/src/client/src/components/(playground)/context/header.tsx b/src/client/src/components/(playground)/context/header.tsxindex 30007292..bf6e1f91 100644--- a/src/client/src/components/(playground)/context/header.tsx+++ b/src/client/src/components/(playground)/context/header.tsx@@ -32,5 +32,5 @@ export default function ContextHeader({</Button>) : null;- return <FeaturePageHeader eyebrow="Resources" title={pageHeaderTitle} icon={<BookOpen className="h-4 w-4" />} tone="border-blue-200 bg-blue-50 text-blue-700 dark:border-blue-900/70 dark:bg-blue-950/40 dark:text-blue-300" actions={<div className={className}>{extraButtons}{actions}</div>} />;+ return <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={pageHeaderTitle} icon={<BookOpen className="h-4 w-4" />} tone="border-blue-200 bg-blue-50 text-blue-700 dark:border-blue-900/70 dark:bg-blue-950/40 dark:text-blue-300" actions={<div className={className}>{extraButtons}{actions}</div>} />;}diff --git a/src/client/src/components/(playground)/costs/costs-analytics.tsx b/src/client/src/components/(playground)/costs/costs-analytics.tsxnew file mode 100644index 00000000..a22052ba--- /dev/null+++ b/src/client/src/components/(playground)/costs/costs-analytics.tsx@@ -0,0 +1,11 @@+"use client";++import CostsDashboard from "@/app/(playground)/dashboard/costs";++export default function CostsAnalytics({+ onConfigure,+}: {+ onConfigure: () => void;+}) {+ return <CostsDashboard onConfigure={onConfigure} />;+}diff --git a/src/client/src/components/(playground)/costs/costs-auto-pricing-banner.tsx b/src/client/src/components/(playground)/costs/costs-auto-pricing-banner.tsxnew file mode 100644index 00000000..b5c7e50a--- /dev/null+++ b/src/client/src/components/(playground)/costs/costs-auto-pricing-banner.tsx@@ -0,0 +1,78 @@+"use client";++import { useCallback, useEffect } from "react";+import { Info } from "lucide-react";+import { Button } from "@/components/ui/button";+import getMessage from "@/constants/messages";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import { getFilterDetails } from "@/selectors/filter";+import { getPingStatus } from "@/selectors/database-config";+import { useRootStore } from "@/store";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import type { CostPricingGuidance } from "@/lib/platform/pricing/guidance";++export default function CostsAutoPricingBanner({+ onConfigure,+}: {+ onConfigure: () => void;+}) {+ const m = getMessage();+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const { data, fireRequest, isFetched, isLoading } =+ useFetchWrapper<CostPricingGuidance>();++ const fetchData = useCallback(() => {+ fireRequest({+ body: JSON.stringify(getFilterParamsForDashboard(filter)),+ requestType: "POST",+ url: "/api/pricing/guidance",+ responseDataKey: "data",+ });+ }, [filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchData();+ }+ }, [filter, fetchData, pingStatus]);++ if (isLoading || !isFetched || pingStatus === "pending") {+ return null;+ }++ if (!data?.showBackfillBanner) {+ return null;+ }++ return (+ <div className="flex flex-col gap-3 rounded-md border border-emerald-200 bg-emerald-50/60 px-4 py-3 dark:border-emerald-900/70 dark:bg-emerald-950/30 sm:flex-row sm:items-center sm:justify-between">+ <div className="flex items-start gap-3 min-w-0">+ <Info className="mt-0.5 size-4 shrink-0 text-emerald-700 dark:text-emerald-300" />+ <div className="min-w-0">+ <p className="text-sm font-medium text-stone-900 dark:text-stone-100">+ {m.COSTS_BACKFILL_BANNER_TITLE}+ </p>+ <p className="mt-0.5 text-xs text-stone-600 dark:text-stone-400">+ {m.COSTS_BACKFILL_BANNER_DESCRIPTION}+ </p>+ <p className="mt-1 text-xs tabular-nums text-emerald-800 dark:text-emerald-200">+ {data.missingCostSpans.toLocaleString()}{" "}+ {m.COSTS_BACKFILL_BANNER_COUNT}+ </p>+ </div>+ </div>+ <Button+ size="sm"+ className="shrink-0"+ onClick={onConfigure}+ >+ {m.COSTS_BACKFILL_BANNER_CTA}+ </Button>+ </div>+ );+}diff --git a/src/client/src/components/(playground)/costs/costs-configuration.tsx b/src/client/src/components/(playground)/costs/costs-configuration.tsxnew file mode 100644index 00000000..418bd87c--- /dev/null+++ b/src/client/src/components/(playground)/costs/costs-configuration.tsx@@ -0,0 +1,208 @@+"use client";++import { Button } from "@/components/ui/button";+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";+import { Input } from "@/components/ui/input";+import { Label } from "@/components/ui/label";+import { Switch } from "@/components/ui/switch";+import getMessage from "@/constants/messages";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { Clock, Info, Play, Workflow, Zap } from "lucide-react";+import Link from "next/link";+import { useEffect, useState } from "react";+import { toast } from "sonner";++interface PricingConfig {+ id?: string;+ auto?: boolean;+ recurringTime?: string;+ meta?: string;+}++const PRICING_TOAST_ID = "pricing-config";++export default function CostsConfiguration() {+ const m = getMessage();+ const [autoPricing, setAutoPricing] = useState(false);+ const [recurringTime, setRecurringTime] = useState("");++ const {+ fireRequest: getConfig,+ data: config,+ isLoading: isLoadingConfig,+ } = useFetchWrapper<PricingConfig>();+ const { fireRequest: saveConfig, isLoading: isSaving } = useFetchWrapper();++ useEffect(() => {+ getConfig({+ requestType: "GET",+ url: "/api/pricing/config",+ responseDataKey: "data",+ });+ }, []);++ useEffect(() => {+ if (config) {+ setAutoPricing(config.auto || false);+ setRecurringTime(config.recurringTime || "");+ }+ }, [config]);++ const handleSave = () => {+ toast.loading(m.SAVING, { id: PRICING_TOAST_ID });++ saveConfig({+ body: JSON.stringify({+ id: config?.id,+ auto: autoPricing,+ recurringTime: recurringTime || "* * * * *",+ meta: config?.meta || "{}",+ }),+ requestType: "POST",+ url: "/api/pricing/config",+ responseDataKey: "data",+ successCb: () => {+ toast.success(m.PRICING_CONFIG_SAVED, { id: PRICING_TOAST_ID });+ getConfig({+ requestType: "GET",+ url: "/api/pricing/config",+ responseDataKey: "data",+ });+ },+ failureCb: (err?: string) => {+ toast.error(err || m.OPERATION_FAILED, {+ id: PRICING_TOAST_ID,+ });+ },+ });+ };++ if (isLoadingConfig && !config) {+ return (+ <div className="animate-pulse grid gap-4 w-full max-w-3xl">+ <div className="h-40 bg-stone-100 dark:bg-stone-900" />+ </div>+ );+ }++ return (+ <div className="flex flex-col w-full gap-4">+ <Card className="border-blue-200 dark:border-blue-900/50 bg-blue-50/50 dark:bg-blue-950/20">+ <CardContent className="py-3">+ <div className="flex items-start gap-3">+ <Info className="size-4 text-blue-600 dark:text-blue-500 shrink-0 mt-0.5" />+ <p className="text-xs text-stone-600 dark:text-stone-400 leading-relaxed">+ {m.PRICING_INFO_BAR}{" "}+ <Link+ href="/costs?tab=models"+ className="font-medium text-stone-700 dark:text-stone-300 underline"+ >+ {m.OPENGROUND_MANAGE_MODELS}+ </Link>+ .+ </p>+ </div>+ </CardContent>+ </Card>++ <div className="grid grid-cols-1 lg:grid-cols-2 gap-4">+ <Card className="border-stone-200 dark:border-stone-800 shadow-sm flex flex-col">+ <CardHeader className="pb-4">+ <CardTitle className="text-base flex items-center gap-2">+ <Zap className="size-4" />+ {m.PRICING_AUTO_TITLE}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">+ {m.PRICING_AUTO_DESCRIPTION}+ </p>+ </CardHeader>+ <CardContent className="space-y-4 flex-1">+ <div className="flex items-center justify-between rounded-lg border border-stone-200 dark:border-stone-700 p-4">+ <div className="space-y-0.5">+ <Label>{m.PRICING_AUTO_ENABLE_LABEL}</Label>+ <p className="text-xs text-stone-500 dark:text-stone-400">+ {m.PRICING_AUTO_ENABLE_HINT}+ </p>+ </div>+ <Switch+ checked={autoPricing}+ onCheckedChange={setAutoPricing}+ />+ </div>+ {autoPricing && (+ <div className="grid gap-3">+ <Label className="flex items-center gap-2">+ <Clock className="size-3.5" />+ {m.PRICING_AUTO_CRON_LABEL}+ </Label>+ <Input+ placeholder={m.PRICING_AUTO_CRON_PLACEHOLDER}+ value={recurringTime}+ onChange={(e) => setRecurringTime(e.target.value)}+ />+ <p className="text-xs text-stone-500 dark:text-stone-400">+ {m.PRICING_AUTO_CRON_HELP}+ </p>+ </div>+ )}+ <Button+ onClick={handleSave}+ disabled={isSaving}+ className="bg-primary dark:bg-primary text-white dark:text-white hover:bg-primary/90 dark:hover:bg-primary/90"+ >+ {isSaving+ ? m.SAVING+ : config?.id+ ? m.PRICING_UPDATE+ : m.PRICING_SAVE}+ </Button>+ </CardContent>+ </Card>++ <Card className="border-stone-200 dark:border-stone-800 shadow-sm flex flex-col">+ <CardHeader className="pb-4">+ <CardTitle className="text-base flex items-center gap-2">+ <Play className="size-4" />+ {m.PRICING_MANUAL_TITLE}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">+ {m.PRICING_MANUAL_DESCRIPTION}+ </p>+ </CardHeader>+ <CardContent className="space-y-4 flex-1">+ <ol className="list-decimal list-inside space-y-2 text-sm text-stone-600 dark:text-stone-400">+ <li>{m.PRICING_MANUAL_STEP_1}</li>+ <li>{m.PRICING_MANUAL_STEP_2}</li>+ <li>{m.PRICING_MANUAL_STEP_3}</li>+ </ol>+ <Link href="/telemetry" className="mt-3 inline-block">+ <Button variant="default">{m.PRICING_GO_TO_TRACES}</Button>+ </Link>+ </CardContent>+ </Card>+ </div>++ <Card className="border-stone-200 dark:border-stone-800 shadow-sm">+ <CardHeader className="pb-3">+ <CardTitle className="text-base flex items-center gap-2">+ <Workflow className="size-4" />+ {m.PRICING_HOW_AUTO_WORKS_TITLE}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">+ {m.PRICING_HOW_AUTO_WORKS_DESCRIPTION}+ </p>+ </CardHeader>+ <CardContent>+ <ol className="list-decimal list-inside space-y-2 text-sm text-stone-600 dark:text-stone-400 marker:text-stone-400">+ <li>{m.PRICING_HOW_AUTO_WORKS_STEP_1}</li>+ <li>{m.PRICING_HOW_AUTO_WORKS_STEP_2}</li>+ <li>{m.PRICING_HOW_AUTO_WORKS_STEP_3}</li>+ <li>{m.PRICING_HOW_AUTO_WORKS_STEP_4}</li>+ <li>{m.PRICING_HOW_AUTO_WORKS_STEP_5}</li>+ <li>{m.PRICING_HOW_AUTO_WORKS_STEP_6}</li>+ </ol>+ </CardContent>+ </Card>+ </div>+ );+}diff --git a/src/client/src/components/(playground)/costs/manage-models-section.tsx b/src/client/src/components/(playground)/costs/manage-models-section.tsxnew file mode 100644index 00000000..17b7755f--- /dev/null+++ b/src/client/src/components/(playground)/costs/manage-models-section.tsx@@ -0,0 +1,396 @@+"use client";++import { useState, useEffect, useMemo } from "react";+import { usePostHog } from "posthog-js/react";+import { CLIENT_EVENTS } from "@/constants/events";+import { Button } from "@/components/ui/button";+import {+ SettingsIcon,+ Download,+ Upload,+ Copy,+ Link as LinkIcon,+ Code2,+ PlusCircle,+} from "lucide-react";+import getMessage from "@/constants/messages";+import ModelListSidebar from "@/components/(playground)/openground/model-list-sidebar";+import ModelEditorPanel from "@/components/(playground)/openground/model-editor-panel";+import SdkUsageDialog from "@/components/(playground)/openground/sdk-usage-dialog";+import ProviderEditorDialog, {+ ProviderFormData,+} from "@/components/(playground)/openground/provider-editor-dialog";+import { ProviderMetadata, ModelMetadata } from "@/types/openground";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import {+ Dialog,+ DialogContent,+ DialogDescription,+ DialogFooter,+ DialogHeader,+ DialogTitle,+} from "@/components/ui/dialog";+import { Textarea } from "@/components/ui/textarea";+import { toast } from "sonner";+import { useRootStore } from "@/store";+import { getDatabaseConfigList } from "@/selectors/database-config";+import { Input } from "@/components/ui/input";+import copy from "copy-to-clipboard";+import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip";++interface CustomModel extends ModelMetadata {+ id: string;+ customId?: string;+ model_id: string;+ provider?: string;+ modelType?: string;+}++export default function ManageModelsSection() {+ const m = getMessage();+ const posthog = usePostHog();+ const [selectedProvider, setSelectedProvider] = useState<string | null>(null);+ const [selectedModel, setSelectedModel] = useState<ModelMetadata | null>(null);+ const [isCustomModel, setIsCustomModel] = useState(false);+ const [isAddingNew, setIsAddingNew] = useState(false);+ const [showImport, setShowImport] = useState(false);+ const [showSdkUsage, setShowSdkUsage] = useState(false);+ const [showProviderEditor, setShowProviderEditor] = useState(false);+ const [editingProvider, setEditingProvider] = useState<ProviderFormData | null>(null);+ const [importJson, setImportJson] = useState("");++ const databaseConfigList = useRootStore(getDatabaseConfigList) || [];+ const activeDbConfig = useMemo(+ () => databaseConfigList.find((item) => !!item.isCurrent),+ [databaseConfigList]+ );+ const dbConfigId = activeDbConfig?.id || "";++ const {+ data: providers,+ fireRequest: fetchProviders,+ isLoading: loadingProviders,+ } = useFetchWrapper<ProviderMetadata[]>();+ const {+ data: customModels,+ fireRequest: fetchCustomModels,+ isLoading: loadingCustom,+ } = useFetchWrapper<Record<string, CustomModel[]>>();+ const { fireRequest: fireImport, isLoading: importing } = useFetchWrapper();++ useEffect(() => {+ posthog?.capture(CLIENT_EVENTS.SETTINGS_MANAGE_MODELS_PAGE_VISITED);+ }, [posthog]);++ useEffect(() => {+ loadProviders();+ loadAllCustomModels();+ }, []);++ const loadProviders = () => {+ fetchProviders({ requestType: "GET", url: "/api/openground/providers" });+ };++ const loadAllCustomModels = () => {+ fetchCustomModels({ requestType: "GET", url: "/api/openground/models" });+ };++ const handleSelectModel = (+ model: ModelMetadata,+ provider: string,+ _isCustom: boolean+ ) => {+ setSelectedModel(model);+ setSelectedProvider(provider);+ setIsCustomModel(true);+ setIsAddingNew(false);+ };++ const handleAddNew = (provider: string) => {+ setSelectedProvider(provider);+ setSelectedModel(null);+ setIsCustomModel(true);+ setIsAddingNew(true);+ };++ const handleModelSaved = () => {+ loadAllCustomModels();+ setIsAddingNew(false);+ };++ const handleModelDeleted = () => {+ loadAllCustomModels();+ setSelectedModel(null);+ setSelectedProvider(null);+ setIsCustomModel(false);+ };++ const handleAddProvider = () => {+ setEditingProvider(null);+ setShowProviderEditor(true);+ };++ const handleEditProvider = (provider: ProviderMetadata) => {+ setEditingProvider({+ providerId: provider.providerId,+ displayName: provider.displayName,+ description: provider.description || "",+ requiresVault: provider.requiresVault,+ });+ setShowProviderEditor(true);+ };++ const handleProviderSaved = () => {+ loadProviders();+ loadAllCustomModels();+ };++ const pricingUrl = dbConfigId+ ? `${typeof window !== "undefined" ? window.location.origin : ""}/api/pricing/export/${dbConfigId}`+ : "";++ const copyUrl = () => {+ if (!pricingUrl) return;+ copy(pricingUrl);+ toast.success(m.MANAGE_MODELS_PRICING_URL_COPIED, { id: "url-copy" });+ };++ const handleImport = () => {+ let parsed: unknown;+ try {+ parsed = JSON.parse(importJson);+ } catch {+ toast.error(m.MANAGE_MODELS_INVALID_JSON, { id: "import" });+ return;+ }++ fireImport({+ requestType: "POST",+ url: "/api/openground/models/import",+ body: JSON.stringify(parsed),+ successCb: (res: { imported?: number; skipped?: number }) => {+ toast.success(+ `${m.MANAGE_MODELS_IMPORT_SUCCESS}: ${res?.imported ?? 0} imported, ${res?.skipped ?? 0} skipped`,+ { id: "import" }+ );+ setShowImport(false);+ setImportJson("");+ loadAllCustomModels();+ loadProviders();+ },+ failureCb: (err?: string) => {+ toast.error(err || m.MANAGE_MODELS_IMPORT_FAILED, { id: "import" });+ },+ });+ };++ const actionButtons = (+ <div className="flex items-center gap-1.5 shrink-0">+ {pricingUrl && (+ <Tooltip>+ <TooltipTrigger>+ <Button+ variant="outline"+ size="sm"+ className="h-8 w-8 p-0"+ onClick={copyUrl}+ >+ <Copy className="h-3.5 w-3.5" />+ </Button>+ </TooltipTrigger>+ <TooltipContent>{m.COPY}</TooltipContent>+ </Tooltip>+ )}+ <Tooltip>+ <TooltipTrigger>+ <Button+ variant="outline"+ size="sm"+ className="h-8 w-8 p-0"+ onClick={handleAddProvider}+ >+ <PlusCircle className="h-3.5 w-3.5" />+ </Button>+ </TooltipTrigger>+ <TooltipContent>{m.MANAGE_PROVIDERS_ADD}</TooltipContent>+ </Tooltip>+ {pricingUrl && (+ <Tooltip>+ <TooltipTrigger>+ <Button+ variant="outline"+ size="sm"+ className="h-8 w-8 p-0"+ onClick={() => setShowSdkUsage(true)}+ >+ <Code2 className="h-3.5 w-3.5" />+ </Button>+ </TooltipTrigger>+ <TooltipContent>{m.MANAGE_MODELS_SDK_USAGE}</TooltipContent>+ </Tooltip>+ )}+ <Tooltip>+ <TooltipTrigger>+ <Button+ variant="outline"+ size="sm"+ className="h-8 w-8 p-0"+ onClick={() => setShowImport(true)}+ >+ <Upload className="h-3.5 w-3.5" />+ </Button>+ </TooltipTrigger>+ <TooltipContent>{m.MANAGE_MODELS_IMPORT_PRICING}</TooltipContent>+ </Tooltip>+ <Tooltip>+ <TooltipTrigger>+ <Button+ variant="outline"+ size="sm"+ className="h-8 w-8 p-0"+ onClick={() => {+ window.open("/api/openground/models/export", "_blank");+ }}+ >+ <Download className="h-3.5 w-3.5" />+ </Button>+ </TooltipTrigger>+ <TooltipContent>{m.MANAGE_MODELS_EXPORT_PRICING}</TooltipContent>+ </Tooltip>+ </div>+ );++ return (+ <div className="flex flex-col w-full h-full min-h-0 overflow-hidden">+ <div className="flex flex-col w-full flex-1 min-h-0 border border-stone-200 dark:border-stone-800 rounded-md overflow-hidden">+ <div className="flex items-center gap-3 px-4 py-2.5 bg-stone-50 dark:bg-stone-950 border-b border-stone-200 dark:border-stone-800 shrink-0">+ {pricingUrl ? (+ <>+ <LinkIcon className="h-3.5 w-3.5 text-stone-400 shrink-0" />+ <span className="text-xs font-medium text-stone-500 dark:text-stone-400 shrink-0">+ {m.MANAGE_MODELS_PRICING_URL_LABEL}+ </span>+ <Input+ readOnly+ value={pricingUrl}+ onFocus={(e) => e.target.select()}+ className="font-mono text-xs h-8 flex-1 min-w-0"+ />+ </>+ ) : (+ <div className="flex-1" />+ )}+ {actionButtons}+ </div>++ <div className="flex flex-1 overflow-hidden min-h-0">+ <ModelListSidebar+ providers={providers || []}+ customModels={customModels || {}}+ loading={loadingProviders || loadingCustom}+ selectedModel={selectedModel}+ selectedProvider={selectedProvider}+ selectedIsCustom={isCustomModel}+ onSelectModel={handleSelectModel}+ onAddNew={handleAddNew}+ onEditProvider={handleEditProvider}+ />++ <div className="flex-1 overflow-auto bg-stone-50 dark:bg-stone-950">+ {selectedModel || isAddingNew ? (+ <ModelEditorPanel+ model={selectedModel}+ provider={selectedProvider}+ isCustomModel={isCustomModel}+ isAddingNew={isAddingNew}+ onSave={handleModelSaved}+ onDelete={handleModelDeleted}+ onCancel={() => {+ setSelectedModel(null);+ setSelectedProvider(null);+ setIsAddingNew(false);+ }}+ />+ ) : (+ <div className="flex flex-col items-center justify-center h-full p-8 text-center">+ <SettingsIcon className="h-16 w-16 text-stone-300 dark:text-stone-700 mb-4" />+ <h3 className="text-lg font-medium text-stone-900 dark:text-stone-100 mb-2">+ {m.OPENGROUND_SELECT_MODEL_TO_VIEW}+ </h3>+ <p className="text-sm text-stone-500 dark:text-stone-400 max-w-md">+ {m.OPENGROUND_SELECT_MODEL_TO_VIEW_DESCRIPTION}+ </p>+ {pricingUrl && (+ <Button+ variant="outline"+ size="sm"+ className="gap-2 mt-6"+ onClick={() => setShowSdkUsage(true)}+ >+ <Code2 className="h-4 w-4" />+ {m.MANAGE_MODELS_SDK_USAGE}+ </Button>+ )}+ </div>+ )}+ </div>+ </div>+ </div>++ <Dialog open={showImport} onOpenChange={setShowImport}>+ <DialogContent className="sm:max-w-lg">+ <DialogHeader>+ <DialogTitle>{m.MANAGE_MODELS_IMPORT_DIALOG_TITLE}</DialogTitle>+ <DialogDescription>+ {m.MANAGE_MODELS_IMPORT_DIALOG_DESCRIPTION}+ </DialogDescription>+ </DialogHeader>+ <Textarea+ placeholder={`{+ "providers": [+ {+ "providerId": "my-provider",+ "displayName": "My Provider",+ "description": "Custom LLM provider"+ }+ ],+ "models": [+ {+ "provider": "my-provider",+ "model_id": "my-model-v1",+ "displayName": "My Model v1",+ "inputPricePerMToken": 2.5,+ "outputPricePerMToken": 10+ }+ ]+ }`}+ value={importJson}+ onChange={(e) => setImportJson(e.target.value)}+ className="min-h-[200px] text-xs dark:text-stone-100 text-stone-800"+ />+ <DialogFooter>+ <Button variant="outline" onClick={() => setShowImport(false)}>+ {m.CANCEL}+ </Button>+ <Button onClick={handleImport} disabled={importing || !importJson.trim()}>+ {importing ? m.SAVING : m.MANAGE_MODELS_IMPORT_PRICING}+ </Button>+ </DialogFooter>+ </DialogContent>+ </Dialog>++ <SdkUsageDialog+ open={showSdkUsage}+ onOpenChange={setShowSdkUsage}+ pricingUrl={pricingUrl}+ />++ <ProviderEditorDialog+ open={showProviderEditor}+ onOpenChange={setShowProviderEditor}+ provider={editingProvider}+ onSaved={handleProviderSaved}+ />+ </div>+ );+}diff --git a/src/client/src/components/(playground)/evaluations/evaluation-analytics.tsx b/src/client/src/components/(playground)/evaluations/evaluation-analytics.tsxnew file mode 100644index 00000000..7f1f6d3a--- /dev/null+++ b/src/client/src/components/(playground)/evaluations/evaluation-analytics.tsx@@ -0,0 +1,333 @@+"use client";++import { useCallback, useEffect, useMemo } from "react";+import Link from "next/link";+import { round } from "lodash";+import { TrendingDown, TrendingUp, Minus } from "lucide-react";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { getFilterDetails } from "@/selectors/filter";+import { getPingStatus } from "@/selectors/database-config";+import { useRootStore } from "@/store";+import { getFilterParamsForDashboard } from "@/helpers/client/filter";+import getMessage from "@/constants/messages";+import { Card, CardContent } from "@/components/ui/card";+import { Skeleton } from "@/components/ui/skeleton";+import EvaluationOnboarding from "./evaluation-onboarding";+import EvaluationsDashboard from "@/app/(playground)/dashboard/evaluations";+import type {+ EvaluationAnalyticsByTypeRow,+ EvaluationAnalyticsResponse,+} from "@/types/evaluation";+import {+ displayEvaluationTypeName,+ resolveEvaluationType,+ type EvaluationTypeRef,+} from "@/helpers/client/evaluation-type";+import { Rule } from "@/types/rule-engine";+import { EVALUATION_TYPES } from "@/constants/evaluation-types";++function changePercent(current: number, previous: number) {+ return round(((current - previous) / (previous || 1)) * 100, 1);+}++function passRateClass(rate: number) {+ if (rate >= 90) return "text-emerald-600 dark:text-emerald-400";+ if (rate >= 75) return "text-lime-600 dark:text-lime-400";+ return "text-amber-600 dark:text-amber-400";+}++type ConfiguredType = EvaluationTypeRef & { enabled?: boolean };++type MergedTypeRow = {+ key: string;+ id: string;+ label: string;+ enabled: boolean;+ executions: number;+ passRate: number;+ previousPassRate: number;+};++/**+ * Keep only configured evaluators (built-in + custom). Drop random LLM+ * labels that do not map to a real evaluation type.+ */+function buildEvaluatorRows(+ rows: EvaluationAnalyticsByTypeRow[],+ configuredTypes: ConfiguredType[]+): MergedTypeRow[] {+ const statsById = new Map<string, MergedTypeRow>();++ for (const row of rows) {+ const resolved = resolveEvaluationType(row.evaluation, configuredTypes);+ if (!resolved) continue;++ const existing = statsById.get(resolved.id);+ if (!existing) {+ const configured = configuredTypes.find((t) => t.id === resolved.id);+ statsById.set(resolved.id, {+ key: resolved.id,+ id: resolved.id,+ label: resolved.label,+ enabled: configured?.enabled ?? true,+ executions: row.executions,+ passRate: row.passRate,+ previousPassRate: row.previousPassRate,+ });+ continue;+ }++ const totalExec = existing.executions + row.executions;+ existing.passRate =+ totalExec === 0+ ? 0+ : (existing.passRate * existing.executions ++ row.passRate * row.executions) /+ totalExec;+ existing.previousPassRate =+ totalExec === 0+ ? 0+ : (existing.previousPassRate * existing.executions ++ row.previousPassRate * row.executions) /+ totalExec;+ existing.executions = totalExec;+ }++ const rowsOut: MergedTypeRow[] = [];+ for (const type of configuredTypes) {+ const stats = statsById.get(type.id);+ if (stats) {+ rowsOut.push(stats);+ continue;+ }+ if (type.enabled) {+ rowsOut.push({+ key: type.id,+ id: type.id,+ label: type.label || displayEvaluationTypeName(type.id),+ enabled: true,+ executions: 0,+ passRate: 0,+ previousPassRate: 0,+ });+ }+ }++ return rowsOut.sort((a, b) => b.executions - a.executions);+}++export default function EvaluationAnalytics({+ onConfigure,+}: {+ onConfigure: () => void;+}) {+ const m = getMessage();+ const filter = useRootStore(getFilterDetails);+ const pingStatus = useRootStore(getPingStatus);+ const { data, isLoading, isFetched, fireRequest } =+ useFetchWrapper<EvaluationAnalyticsResponse>();+ const { data: typesResponse, fireRequest: getTypes } = useFetchWrapper<+ ConfiguredType[] | { data?: ConfiguredType[] }+ >();+ const { data: rules, fireRequest: getRules } = useFetchWrapper<Rule[]>();+ const { data: evalEntities, fireRequest: getEvalEntities } = useFetchWrapper<+ Array<{ rule_id: string; entity_id: string }>+ >();++ const fetchAnalytics = useCallback(() => {+ fireRequest({+ body: JSON.stringify(getFilterParamsForDashboard(filter)),+ requestType: "POST",+ url: "/api/evaluation/analytics",+ });+ }, [filter, fireRequest]);++ useEffect(() => {+ if (+ filter.timeLimit.start &&+ filter.timeLimit.end &&+ pingStatus === "success"+ ) {+ fetchAnalytics();+ }+ }, [filter, fetchAnalytics, pingStatus]);++ useEffect(() => {+ getTypes({+ requestType: "GET",+ url: "/api/evaluation/types",+ responseDataKey: "data",+ });+ getRules({ requestType: "GET", url: "/api/rule-engine/rules" });+ getEvalEntities({+ requestType: "GET",+ url: "/api/rule-engine/entities?entity_type=evaluation",+ });+ }, [getTypes, getRules, getEvalEntities]);++ const configuredTypes = useMemo(() => {+ const list = Array.isArray(typesResponse)+ ? typesResponse+ : typesResponse?.data;+ if (Array.isArray(list) && list.length > 0) {+ return list+ .filter((t) => t?.id)+ .map((t) => ({+ id: t.id,+ label: t.label || t.id,+ enabled: t.enabled ?? true,+ }));+ }+ return EVALUATION_TYPES.map((t) => ({+ id: t.id,+ label: t.label,+ enabled: t.enabledByDefault,+ }));+ }, [typesResponse]);++ const matcherByType = useMemo(() => {+ const map: Record<string, string[]> = {};+ for (const entity of evalEntities || []) {+ if (!map[entity.entity_id]) map[entity.entity_id] = [];+ if (!map[entity.entity_id].includes(entity.rule_id)) {+ map[entity.entity_id].push(entity.rule_id);+ }+ }+ return map;+ }, [evalEntities]);++ const loading = isLoading || !isFetched || pingStatus === "pending";++ if (isFetched && data && !data.configured) {+ return <EvaluationOnboarding onConfigure={onConfigure} />;+ }++ const byType = buildEvaluatorRows(+ (data?.byType || []) as EvaluationAnalyticsByTypeRow[],+ configuredTypes+ );++ return (+ <div className="flex flex-col gap-4 w-full">+ <EvaluationsDashboard />++ <Card className="overflow-hidden border-stone-200 shadow-sm dark:border-stone-800">+ <CardContent className="p-0">+ {loading ? (+ <div className="p-4">+ <Skeleton className="h-24 w-full" />+ </div>+ ) : byType.length === 0 ? (+ <div className="px-4 py-8 text-center text-sm text-stone-500 dark:text-stone-400">+ {m.EVALUATION_TABLE_EMPTY}+ </div>+ ) : (+ <div className="overflow-x-auto">+ <table className="w-full text-sm">+ <thead>+ <tr className="border-y border-stone-200 text-left text-[11px] uppercase tracking-wide text-stone-500 dark:border-stone-800 dark:text-stone-400">+ <th className="px-4 py-2 font-medium">+ {m.EVALUATION_TABLE_EVALUATION}+ </th>+ <th className="px-4 py-2 font-medium">+ {m.EVALUATION_TABLE_MATCHER}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.EVALUATION_TABLE_EXECUTIONS}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.EVALUATION_TABLE_PASS_RATE}+ </th>+ <th className="px-4 py-2 font-medium text-right">+ {m.EVALUATION_TABLE_PASS_RATE_TREND}+ </th>+ </tr>+ </thead>+ <tbody>+ {byType.map((row) => {+ const ruleIds = matcherByType[row.id] || [];+ const matcher =+ ruleIds.length === 0+ ? m.EVALUATION_TABLE_MATCHER_DEFAULT+ : ruleIds+ .slice(0, 2)+ .map((rid) => {+ const rule = (rules || []).find(+ (r) => r.id === rid+ );+ return rule?.name || rid.slice(0, 8);+ })+ .join(", ") ++ (ruleIds.length > 2+ ? ` +${ruleIds.length - 2}`+ : "");+ const trend = changePercent(+ row.passRate,+ row.previousPassRate+ );++ return (+ <tr+ key={row.key}+ className="border-b border-stone-100 hover:bg-stone-50 dark:border-stone-900 dark:hover:bg-stone-900/50"+ >+ <td className="px-4 py-2.5">+ <Link+ href={`/evaluations/evaluators/${encodeURIComponent(row.id)}`}+ className="inline-flex items-center gap-2 font-medium text-primary hover:underline"+ >+ <span+ className={`inline-block h-1.5 w-1.5 rounded-full ${+ row.enabled+ ? "bg-emerald-500"+ : "bg-stone-400"+ }`}+ />+ {row.label}+ </Link>+ </td>+ <td className="px-4 py-2.5 font-mono text-xs text-stone-500 dark:text-stone-400">+ {matcher}+ </td>+ <td className="px-4 py-2.5 text-right tabular-nums text-stone-800 dark:text-stone-200">+ {row.executions.toLocaleString()}+ </td>+ <td+ className={`px-4 py-2.5 text-right tabular-nums font-medium ${passRateClass(row.passRate)}`}+ >+ {round(row.passRate, 0)}%+ </td>+ <td className="px-4 py-2.5 text-right">+ <span+ className={`inline-flex items-center gap-0.5 text-xs ${+ trend === 0+ ? "text-stone-500"+ : trend > 0+ ? "text-emerald-600 dark:text-emerald-400"+ : "text-rose-600 dark:text-rose-400"+ }`}+ >+ {trend === 0 ? (+ <Minus className="h-3 w-3" />+ ) : trend > 0 ? (+ <TrendingUp className="h-3 w-3" />+ ) : (+ <TrendingDown className="h-3 w-3" />+ )}+ {trend === 0+ ? "0%"+ : `${trend > 0 ? "+" : ""}${trend}%`}+ </span>+ </td>+ </tr>+ );+ })}+ </tbody>+ </table>+ </div>+ )}+ </CardContent>+ </Card>+ </div>+ );+}diff --git a/src/client/src/components/(playground)/evaluations/evaluation-configuration.tsx b/src/client/src/components/(playground)/evaluations/evaluation-configuration.tsxnew file mode 100644index 00000000..336e7cd4--- /dev/null+++ b/src/client/src/components/(playground)/evaluations/evaluation-configuration.tsx@@ -0,0 +1,437 @@+"use client";++import SecretForm from "@/components/(playground)/vault/form";+import { Button } from "@/components/ui/button";+import { EVALUATION_ENGINES } from "@/constants/evaluation-engines";+import { CLIENT_EVENTS } from "@/constants/events";+import getMessage from "@/constants/messages";+import { EvaluationConfig } from "@/types/evaluation";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { usePostHog } from "posthog-js/react";+import { useEffect, useMemo, useState } from "react";+import { toast } from "sonner";+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";+import { Label } from "@/components/ui/label";+import { Switch } from "@/components/ui/switch";+import {+ Select,+ SelectContent,+ SelectItem,+ SelectTrigger,+ SelectValue,+} from "@/components/ui/select";+import { Input } from "@/components/ui/input";+import Link from "next/link";+import { Zap, Settings2, Clock, Play, Info, SettingsIcon } from "lucide-react";+import { ProviderMetadata } from "@/types/openground";+import { Badge } from "@/components/ui/badge";+import { evalSampleRateToPercent } from "@/constants/evaluation-sampling";+const EvaluationVaultCreate = ({+ successCallback,+}: {+ successCallback: () => void;+}) => (+ <div className="flex flex-wrap items-center gap-1 w-full text-stone-600 dark:text-stone-400 text-sm mt-1">+ {getMessage().EVALUATION_VAULT_KEY_NOT_FOUND}+ <SecretForm successCallback={successCallback}>+ <Button+ variant="ghost"+ className="py-0 h-auto px-0 text-xs text-primary hover:bg-transparent"+ >+ {getMessage().EVALUATION_CREATE_NEW}+ </Button>+ </SecretForm>+ </div>+);++const EVALUATION_TOAST_ID = "evaluation-config";++export default function EvaluationConfiguration() {+ const m = getMessage();+ const [provider, setProvider] = useState("");+ const [model, setModel] = useState("");+ const [engine] = useState<string>(EVALUATION_ENGINES[0].id);+ const [autoEvaluation, setAutoEvaluation] = useState(false);+ const [recurringTime, setRecurringTime] = useState("");+ const [sampleRatePercent, setSampleRatePercent] = useState(+ evalSampleRateToPercent(undefined)+ );+ const [vaultId, setVaultId] = useState("");+ const [vaultKeys, setVaultKeys] = useState<{ label: string; value: string }[]>(+ []+ );++ const {+ fireRequest: getConfig,+ data: config,+ isLoading: isLoadingConfig,+ } = useFetchWrapper<EvaluationConfig>();+ const { fireRequest: saveConfig, isLoading: isSaving } = useFetchWrapper();+ const { fireRequest: getVaultKeys } = useFetchWrapper();+ const { data: providers, fireRequest: fetchProviders } =+ useFetchWrapper<ProviderMetadata[]>();+ const posthog = usePostHog();++ useEffect(() => {+ getConfig({+ requestType: "GET",+ url: "/api/evaluation/config",+ responseDataKey: "data",+ });+ getVaultKeys({+ requestType: "POST",+ url: "/api/vault/get",+ responseDataKey: "data",+ body: JSON.stringify({}),+ successCb: (res: any) => {+ setVaultKeys(+ res?.length ? res.map((k: any) => ({ label: k.key, value: k.id })) : []+ );+ },+ });+ fetchProviders({+ requestType: "GET",+ url: "/api/openground/providers",+ });+ }, []);++ useEffect(() => {+ if (config) {+ setProvider(config.provider || "");+ setModel(config.model || "");+ setAutoEvaluation(config.auto || false);+ setRecurringTime(config.recurringTime || "");+ setVaultId(config.vaultId || "");+ const meta = JSON.parse(config.meta || "{}");+ setSampleRatePercent(evalSampleRateToPercent(meta.evalSampleRate));+ }+ }, [config]);++ const selectedProviderMeta = useMemo(+ () => (providers || []).find((p) => p.providerId === provider),+ [providers, provider]+ );+ const modelOptions = useMemo(+ () => selectedProviderMeta?.supportedModels || [],+ [selectedProviderMeta]+ );++ const handleSave = () => {+ if (!provider || !model || !vaultId) {+ toast.error(m.EVALUATION_CONFIG_INVALID, {+ id: EVALUATION_TOAST_ID,+ });+ return;+ }++ const parsedSampleRate = Number.parseFloat(sampleRatePercent);+ if (autoEvaluation) {+ if (+ !Number.isFinite(parsedSampleRate) ||+ parsedSampleRate < 0 ||+ parsedSampleRate > 100+ ) {+ toast.error(m.EVALUATION_SAMPLE_RATE_PERCENT_INVALID, {+ id: EVALUATION_TOAST_ID,+ });+ return;+ }+ }++ toast.loading(m.EVALUATION_CONFIG_MODIFYING, {+ id: EVALUATION_TOAST_ID,+ });++ const meta = JSON.parse(config?.meta || "{}");+ meta.engine = engine;+ if (autoEvaluation) {+ meta.evalSampleRate = parsedSampleRate / 100;+ }++ saveConfig({+ body: JSON.stringify({+ id: config?.id,+ provider,+ model,+ vaultId,+ auto: autoEvaluation,+ recurringTime: recurringTime || "* * * * *",+ meta: JSON.stringify(meta),+ }),+ requestType: "POST",+ url: "/api/evaluation/config",+ responseDataKey: "data",+ successCb: () => {+ toast.success(+ config?.id ? m.EVALUATION_UPDATED : m.EVALUATION_CREATED,+ { id: EVALUATION_TOAST_ID }+ );+ getConfig({+ requestType: "GET",+ url: "/api/evaluation/config",+ responseDataKey: "data",+ });+ posthog?.capture(+ config?.id+ ? CLIENT_EVENTS.EVALUATION_CONFIG_UPDATED_SUCCESS+ : CLIENT_EVENTS.EVALUATION_CONFIG_CREATED_SUCCESS+ );+ },+ failureCb: (err?: string) => {+ toast.error(err || m.EVALUATION_CONFIG_UPDATING_FAILED, {+ id: EVALUATION_TOAST_ID,+ });+ },+ });+ };++ if (isLoadingConfig && !config) {+ return (+ <div className="animate-pulse grid gap-4 w-full max-w-3xl">+ <div className="h-40 bg-stone-100 dark:bg-stone-900" />+ </div>+ );+ }++ return (+ <div className="flex flex-col w-full gap-4">+ <Card className="border-blue-200 dark:border-blue-900/50 bg-blue-50/50 dark:bg-blue-950/20">+ <CardContent className="pt-6">+ <div className="flex gap-3">+ <Info className="size-5 text-blue-600 dark:text-blue-500 shrink-0 mt-0.5" />+ <div className="text-sm text-stone-600 dark:text-stone-400">+ <p className="font-medium text-stone-700 dark:text-stone-300 mb-1">+ {m.EVALUATION_MANUAL_AND_AUTO}+ </p>+ <p>{m.EVALUATION_MANUAL_AND_AUTO_DESCRIPTION}</p>+ </div>+ </div>+ </CardContent>+ </Card>++ <div className="grid grid-cols-1 lg:grid-cols-3 gap-4">+ <div className="lg:col-span-2 grid gap-4">+ <Card className="border-stone-200 dark:border-stone-800 shadow-sm">+ <CardHeader className="pb-4">+ <CardTitle className="text-base flex items-center gap-2">+ <Settings2 className="size-4" />+ {m.EVALUATION_CONFIG_SECTION}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">+ {m.EVALUATION_ENGINE_DESCRIPTION}+ </p>+ </CardHeader>+ <CardContent className="space-y-4">+ <div className="grid gap-3">+ <div className="flex items-center justify-between">+ <Label>{m.EVALUATION_PROVIDER_LABEL}</Label>+ <Link href="/costs?tab=models">+ <Button+ variant="ghost"+ size="sm"+ className="h-7 text-xs px-2"+ >+ <SettingsIcon className="h-3 w-3 mr-1" />+ {m.OPENGROUND_MANAGE_MODELS}+ </Button>+ </Link>+ </div>+ <Select+ value={provider}+ onValueChange={(v) => {+ setProvider(v);+ setModel("");+ }}+ >+ <SelectTrigger>+ <SelectValue placeholder={m.EVALUATION_SELECT_PROVIDER} />+ </SelectTrigger>+ <SelectContent>+ {(providers || []).map((p) => (+ <SelectItem key={p.providerId} value={p.providerId}>+ {p.displayName}+ </SelectItem>+ ))}+ </SelectContent>+ </Select>+ </div>+ <div className="grid gap-3">+ <Label>{m.EVALUATION_MODEL_LABEL}</Label>+ <Select+ value={model}+ onValueChange={setModel}+ disabled={!provider}+ >+ <SelectTrigger className="h-auto">+ <SelectValue+ placeholder={+ provider+ ? m.EVALUATION_MODEL_PLACEHOLDER+ : m.EVALUATION_SELECT_PROVIDER_FIRST+ }+ />+ </SelectTrigger>+ <SelectContent>+ {modelOptions.map((modelOption) => (+ <SelectItem key={modelOption.id} value={modelOption.id}>+ <div className="flex flex-col items-start gap-0.5">+ <span className="text-sm">+ {modelOption.displayName}+ </span>+ <div className="flex gap-1.5 text-xs text-stone-500">+ <Badge+ variant="secondary"+ className="text-xs h-4 px-1"+ >+ {modelOption.contextWindow.toLocaleString()}+ </Badge>+ <Badge+ variant="outline"+ className="text-xs h-4 px-1"+ >+ ${modelOption.inputPricePerMToken}/M+ </Badge>+ </div>+ </div>+ </SelectItem>+ ))}+ </SelectContent>+ </Select>+ <p className="text-xs text-stone-500 dark:text-stone-400">+ {m.EVALUATION_MODEL_CUSTOM_HINT}+ </p>+ </div>++ <div className="grid gap-3">+ <Label>{m.EVALUATION_API_KEY_VAULT}</Label>+ <Select value={vaultId} onValueChange={setVaultId}>+ <SelectTrigger>+ <SelectValue placeholder={m.EVALUATION_SELECT_VAULT_KEY} />+ </SelectTrigger>+ <SelectContent>+ {vaultKeys.map((o) => (+ <SelectItem key={o.value} value={o.value}>+ {o.label}+ </SelectItem>+ ))}+ </SelectContent>+ </Select>+ <EvaluationVaultCreate+ successCallback={() =>+ getVaultKeys({+ requestType: "POST",+ url: "/api/vault/get",+ responseDataKey: "data",+ body: JSON.stringify({}),+ successCb: (res: any) => {+ setVaultKeys(+ res?.length+ ? res.map((k: any) => ({+ label: k.key,+ value: k.id,+ }))+ : []+ );+ },+ })+ }+ />+ </div>+ </CardContent>+ </Card>+ </div>++ <div className="grid gap-4 content-start">+ <Card className="border-stone-200 dark:border-stone-800 shadow-sm">+ <CardHeader className="pb-4">+ <CardTitle className="text-base flex items-center gap-2">+ <Zap className="size-4" />+ {m.EVALUATION_AUTO_TITLE}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">+ {m.EVALUATION_AUTO_DESCRIPTION}+ </p>+ </CardHeader>+ <CardContent className="space-y-4">+ <div className="flex items-center justify-between rounded-lg border border-stone-200 dark:border-stone-700 p-4">+ <div className="space-y-0.5">+ <Label>{m.EVALUATION_ENABLE_AUTO}</Label>+ <p className="text-xs text-stone-500 dark:text-stone-400">+ {m.EVALUATION_ENABLE_AUTO_DESCRIPTION}+ </p>+ </div>+ <Switch+ checked={autoEvaluation}+ onCheckedChange={setAutoEvaluation}+ />+ </div>+ {autoEvaluation && (+ <div className="grid gap-3">+ <Label className="flex items-center gap-2">+ <Clock className="size-3.5" />+ {m.EVALUATION_CRON_SCHEDULE}+ </Label>+ <Input+ placeholder={m.EVALUATION_CRON_PLACEHOLDER}+ value={recurringTime}+ onChange={(e) => setRecurringTime(e.target.value)}+ />+ <p className="text-xs text-stone-500 dark:text-stone-400">+ {m.EVALUATION_CRON_HELP}+ </p>+ <Label>{m.EVALUATION_SAMPLE_RATE_LABEL}</Label>+ <Input+ type="number"+ min={0}+ max={100}+ step={1}+ value={sampleRatePercent}+ onChange={(e) => setSampleRatePercent(e.target.value)}+ />+ <p className="text-xs text-stone-500 dark:text-stone-400">+ {m.EVALUATION_SAMPLE_RATE_DESCRIPTION}+ </p>+ </div>+ )}+ </CardContent>+ </Card>++ <Button+ onClick={handleSave}+ disabled={isSaving}+ className="w-full sm:w-auto bg-primary dark:bg-primary text-white dark:text-white hover:bg-primary/90 dark:hover:bg-primary/90"+ >+ {isSaving+ ? m.EVALUATION_SAVING+ : config?.id+ ? m.EVALUATION_SAVE_CHANGES+ : m.EVALUATION_CREATE_CONFIG}+ </Button>++ <Card className="border-stone-200 dark:border-stone-800 shadow-sm">+ <CardHeader className="pb-4">+ <CardTitle className="text-base flex items-center gap-2">+ <Play className="size-4" />+ {m.EVALUATION_MANUAL_TITLE}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 font-normal">+ {m.EVALUATION_MANUAL_DESCRIPTION}+ </p>+ </CardHeader>+ <CardContent className="space-y-4">+ <ol className="list-decimal list-inside space-y-2 text-sm text-stone-600 dark:text-stone-400 mb-3">+ <li>{m.EVALUATION_MANUAL_STEP_1}</li>+ <li>{m.EVALUATION_MANUAL_STEP_2}</li>+ <li>{m.EVALUATION_MANUAL_STEP_3}</li>+ </ol>+ <Link href="/telemetry">+ <Button variant="default">{m.EVALUATION_GO_TO_TRACES}</Button>+ </Link>+ </CardContent>+ </Card>+ </div>+ </div>++ </div>+ );+}diff --git a/src/client/src/components/(playground)/evaluations/evaluation-onboarding.tsx b/src/client/src/components/(playground)/evaluations/evaluation-onboarding.tsxnew file mode 100644index 00000000..de36d792--- /dev/null+++ b/src/client/src/components/(playground)/evaluations/evaluation-onboarding.tsx@@ -0,0 +1,52 @@+"use client";++import { Settings2 } from "lucide-react";+import { Button } from "@/components/ui/button";+import getMessage from "@/constants/messages";++export default function EvaluationOnboarding({+ onConfigure,+}: {+ onConfigure: () => void;+}) {+ const m = getMessage();++ return (+ <div className="flex flex-col items-center justify-center w-full py-16 px-4">+ <div className="max-w-xl w-full flex flex-col items-center">+ <div className="w-16 h-16 bg-orange-50 dark:bg-orange-950/40 rounded-full flex items-center justify-center mb-4 border border-orange-200 dark:border-orange-900/70">+ <Settings2 className="w-8 h-8 text-orange-600 dark:text-orange-300" />+ </div>+ <h2 className="text-2xl font-semibold text-stone-700 dark:text-stone-200 mb-2 text-center">+ {m.EVALUATION_ONBOARDING_TITLE}+ </h2>+ <p className="text-stone-500 dark:text-stone-400 text-center max-w-md mb-6">+ {m.EVALUATION_ONBOARDING_DESCRIPTION}+ </p>+ <ol className="w-full space-y-3 mb-8 text-sm text-stone-600 dark:text-stone-400">+ <li className="rounded-md border border-stone-200 dark:border-stone-800 bg-stone-50 dark:bg-stone-900 px-4 py-3">+ <span className="font-medium text-stone-800 dark:text-stone-200">+ 1.{" "}+ </span>+ {m.EVALUATION_ONBOARDING_STEP_1}+ </li>+ <li className="rounded-md border border-stone-200 dark:border-stone-800 bg-stone-50 dark:bg-stone-900 px-4 py-3">+ <span className="font-medium text-stone-800 dark:text-stone-200">+ 2.{" "}+ </span>+ {m.EVALUATION_ONBOARDING_STEP_2}+ </li>+ <li className="rounded-md border border-stone-200 dark:border-stone-800 bg-stone-50 dark:bg-stone-900 px-4 py-3">+ <span className="font-medium text-stone-800 dark:text-stone-200">+ 3.{" "}+ </span>+ {m.EVALUATION_ONBOARDING_STEP_3}+ </li>+ </ol>+ <Button onClick={onConfigure} className="h-9">+ {m.EVALUATION_ONBOARDING_CTA}+ </Button>+ </div>+ </div>+ );+}diff --git a/src/client/src/components/(playground)/evaluations/evaluation-types-section.tsx b/src/client/src/components/(playground)/evaluations/evaluation-types-section.tsxnew file mode 100644index 00000000..c7c1bc40--- /dev/null+++ b/src/client/src/components/(playground)/evaluations/evaluation-types-section.tsx@@ -0,0 +1,209 @@+"use client";++import { Card, CardHeader, CardTitle } from "@/components/ui/card";+import { Badge } from "@/components/ui/badge";+import useFetchWrapper from "@/utils/hooks/useFetchWrapper";+import { useEffect, useState } from "react";+import { EVALUATION_TYPES } from "@/constants/evaluation-types";+import { usePostHog } from "posthog-js/react";+import { CLIENT_EVENTS } from "@/constants/events";+import { Rule } from "@/types/rule-engine";+import { Layers, CheckCircle2, ChevronRight, Plus, Sparkles } from "lucide-react";+import Link from "next/link";+import { Button } from "@/components/ui/button";+import getMessage from "@/constants/messages";++interface EvaluationTypeDisplay {+ id: string;+ label: string;+ description: string;+ enabledByDefault: boolean;+ enabled: boolean;+ isCustom?: boolean;+ rules?: Array<{ ruleId: string; priority: number }>;+ prompt?: string;+}++export default function EvaluationTypesSection() {+ const m = getMessage();+ const posthog = usePostHog();+ const { fireRequest: getTypes, data: typesResponse } = useFetchWrapper<+ EvaluationTypeDisplay[] | { data: any[] }+ >();+ const { fireRequest: getRules, data: rules } = useFetchWrapper<Rule[]>();+ const { fireRequest: getEvalEntities, data: evalEntities } = useFetchWrapper<+ Array<{ rule_id: string; entity_id: string }>+ >();++ const [allTypes, setAllTypes] = useState<EvaluationTypeDisplay[]>([]);++ const rulesLinkedToType = (evalEntities || []).reduce<+ Record<string, string[]>+ >((acc, e) => {+ if (!acc[e.entity_id]) acc[e.entity_id] = [];+ if (!acc[e.entity_id].includes(e.rule_id)) acc[e.entity_id].push(e.rule_id);+ return acc;+ }, {});++ useEffect(() => {+ posthog?.capture(CLIENT_EVENTS.EVALUATION_TYPES_PAGE_VISITED);+ getTypes({+ requestType: "GET",+ url: "/api/evaluation/types",+ responseDataKey: "data",+ successCb: (response: EvaluationTypeDisplay[] | { data?: any[] }) => {+ const list = Array.isArray(response) ? response : response?.data;+ posthog?.capture(CLIENT_EVENTS.EVALUATION_TYPES_LIST, {+ count: Array.isArray(list) ? list.length : 0,+ });+ },+ });+ getRules({ requestType: "GET", url: "/api/rule-engine/rules" });+ getEvalEntities({+ requestType: "GET",+ url: "/api/rule-engine/entities?entity_type=evaluation",+ });+ }, []);++ useEffect(() => {+ const savedTypes = Array.isArray(typesResponse)+ ? typesResponse+ : (typesResponse as { data?: any[] })?.data;+ if (savedTypes !== undefined && Array.isArray(savedTypes)) {+ setAllTypes(savedTypes);+ }+ }, [typesResponse]);++ const builtInIds = new Set<string>(EVALUATION_TYPES.map((et) => et.id));+ const builtInTypes = allTypes.filter((t) => builtInIds.has(t.id));+ const customTypes = allTypes.filter((t) => !builtInIds.has(t.id));++ const displayBuiltIn = EVALUATION_TYPES.map((et) => {+ const saved = builtInTypes.find((t) => t.id === et.id);+ return {+ ...et,+ enabled: saved?.enabled ?? et.enabledByDefault,+ isCustom: false,+ rules: saved?.rules || [],+ };+ });++ const renderTypeCard = (type: EvaluationTypeDisplay, isCustomType: boolean) => {+ const hasRules =+ (type.rules?.length ?? 0) > 0 ||+ (rulesLinkedToType[type.id]?.length ?? 0) > 0;+ return (+ <Link key={type.id} href={`/evaluations/types/${type.id}`}>+ <Card+ className={`shadow-sm overflow-hidden transition-colors cursor-pointer ${+ isCustomType+ ? "border-primary/20 dark:border-primary/15 hover:border-primary/40 dark:hover:border-primary/30"+ : "border-stone-200 dark:border-stone-800 hover:border-stone-300 dark:hover:border-stone-700"+ }`}+ >+ <CardHeader className="pb-3">+ <div className="flex items-start justify-between gap-4">+ <div className="flex-1 min-w-0">+ <CardTitle className="text-base flex items-center gap-2 flex-wrap text-stone-900 dark:text-stone-100">+ {type.label}+ {isCustomType && (+ <Badge+ variant="outline"+ className="text-xs font-normal gap-1 shrink-0 border-primary/30 text-primary dark:border-primary/20"+ >+ <Sparkles className="size-3" />+ {m.EVALUATION_TYPE_CUSTOM_BADGE}+ </Badge>+ )}+ {hasRules && (+ <Badge+ variant="outline"+ className="text-xs font-normal gap-1 shrink-0"+ >+ <CheckCircle2 className="size-3" />+ {m.EVALUATION_TYPE_RULE_ENGINE_BADGE}+ </Badge>+ )}+ </CardTitle>+ <p className="text-sm text-stone-500 dark:text-stone-400 mt-1">+ {type.description}+ </p>+ {(rulesLinkedToType[type.id]?.length ?? 0) > 0 && (+ <div className="flex items-center gap-1.5 mt-2 flex-wrap">+ <span className="text-xs text-stone-500 dark:text-stone-400">+ {m.EVALUATION_TYPES_LINKED_FROM_RULES}+ </span>+ {rulesLinkedToType[type.id].slice(0, 3).map((rid) => {+ const r = (rules || []).find((x) => x.id === rid);+ return (+ <span+ key={rid}+ className="inline-flex items-center gap-1 text-xs text-primary"+ >+ {r?.name || rid.slice(0, 8)}+ </span>+ );+ })}+ {rulesLinkedToType[type.id].length > 3 && (+ <span className="text-xs text-stone-400 dark:text-stone-500">+ +{rulesLinkedToType[type.id].length - 3} more+ </span>+ )}+ </div>+ )}+ </div>+ <div className="flex items-center gap-2 shrink-0">+ <Badge variant={type.enabled ? "default" : "secondary"}>+ {type.enabled+ ? m.EVALUATION_TYPE_ENABLED+ : m.EVALUATION_TYPE_DISABLED}+ </Badge>+ <ChevronRight className="size-4 text-stone-400 dark:text-stone-500" />+ </div>+ </div>+ </CardHeader>+ </Card>+ </Link>+ );+ };++ return (+ <div className="w-full space-y-4">+ <div className="flex flex-wrap items-center justify-between gap-2">+ <div className="min-w-0">+ <h3 className="text-base font-semibold text-stone-900 dark:text-stone-100 flex items-center gap-2">+ <Layers className="size-4" />+ {m.EVALUATION_TYPES_SECTION_TITLE}+ </h3>+ <p className="text-sm text-stone-500 dark:text-stone-400 mt-0.5">+ {m.EVALUATION_TYPES_SECTION_DESCRIPTION}+ </p>+ </div>+ <Button asChild size="sm" className="h-8">+ <Link href="/evaluations/types/new">+ <Plus className="mr-1.5 h-3.5 w-3.5" />+ {m.EVALUATION_CREATE_CUSTOM_TYPE}+ </Link>+ </Button>+ </div>++ <div className="grid grid-cols-1 md:grid-cols-2 gap-4">+ {displayBuiltIn.map((et) => renderTypeCard(et, false))}+ </div>++ {customTypes.length > 0 && (+ <>+ <div className="flex items-center gap-2 pt-2">+ <Sparkles className="size-4 text-primary" />+ <h4 className="text-sm font-semibold text-stone-700 dark:text-stone-300">+ {m.EVALUATION_CUSTOM_TYPES_HEADING}+ </h4>+ </div>+ <div className="grid grid-cols-1 md:grid-cols-2 gap-4">+ {customTypes.map((ct) => renderTypeCard(ct, true))}+ </div>+ </>+ )}+ </div>+ );+}diff --git a/src/client/src/components/(playground)/feature-page-header.tsx b/src/client/src/components/(playground)/feature-page-header.tsxindex 2de5368b..ffcfadfa 100644--- a/src/client/src/components/(playground)/feature-page-header.tsx+++ b/src/client/src/components/(playground)/feature-page-header.tsx@@ -11,6 +11,13 @@ type FeaturePageHeaderProps = {actions?: ReactNode;};+/**+ * Shared page title bar for playground surfaces (Telemetry, Agents,+ * Dashboards, Resources, …). Keep this compact: Telemetry is the+ * visual reference. Call sites should pass `h-4 w-4` (or `size-4`)+ * icons; tone should be color classes only (border/bg/text) — padding+ * and rounding live here so every page matches.+ */export default function FeaturePageHeader({eyebrow,title,@@ -26,7 +33,9 @@ export default function FeaturePageHeader({<div className="min-w-0"><div className="flex items-center gap-2">{leading ? <div className="shrink-0">{leading}</div> : null}- <span className={`inline-flex shrink-0 items-center justify-center rounded-md border p-2 ${tone}`}>+ <span+ className={`inline-flex size-7 shrink-0 items-center justify-center rounded-md border p-1.5 ${tone}`}+ >{icon}</span><div className="min-w-0">@@ -44,7 +53,11 @@ export default function FeaturePageHeader({</div></div></div>- {actions ? <div className="shrink-0">{actions}</div> : null}+ {actions ? (+ <div className="flex min-w-0 shrink-0 flex-wrap items-center justify-end gap-2">+ {actions}+ </div>+ ) : null}</div></section>);diff --git a/src/client/src/components/(playground)/filter/traces-filter.tsx b/src/client/src/components/(playground)/filter/traces-filter.tsxindex 19153599..52de48df 100644--- a/src/client/src/components/(playground)/filter/traces-filter.tsx+++ b/src/client/src/components/(playground)/filter/traces-filter.tsx@@ -239,6 +239,7 @@ const DynamicFilters = ({configUrl,attributeKeysUrl,customAttributeTypes,+ pageName,}: {isVisibleFilters: boolean;filter: FilterType;@@ -246,6 +247,7 @@ const DynamicFilters = ({configUrl: string;attributeKeysUrl: string;customAttributeTypes: CustomFilterAttributeType[];+ pageName?: PAGE;}) => {const filterConfig = useRootStore(getFilterConfig);const pingStatus = useRootStore(getPingStatus);@@ -342,7 +344,16 @@ const DynamicFilters = ({const fetchConfig = useCallback(async (timeLimit: FilterType["timeLimit"]) => {fireRequest({- body: JSON.stringify({ timeLimit }),+ body: JSON.stringify({+ timeLimit,+ // Exceptions tab must discover filter options from Error+ // spans. The default config query only looks at OK/Unset+ // rows, so providers/services from failing LLM calls+ // (often the only traffic on Exceptions) never appear.+ ...(pageName === "exception"+ ? { statusCode: ["STATUS_CODE_ERROR", "Error"] }+ : {}),+ }),requestType: "POST",url: configUrl,successCb: (resp) => {@@ -355,7 +366,7 @@ const DynamicFilters = ({},});// eslint-disable-next-line react-hooks/exhaustive-deps- }, [configUrl]);+ }, [configUrl, pageName]);// Fetch filter config when time window changes or config is cleared after a range change.useEffect(() => {@@ -372,7 +383,12 @@ const DynamicFilters = ({const fetchAttributeKeys = useCallback(async (timeLimit: FilterType["timeLimit"]) => {fireAttrKeysRequest({- body: JSON.stringify({ timeLimit }),+ body: JSON.stringify({+ timeLimit,+ ...(pageName === "exception"+ ? { statusCode: ["STATUS_CODE_ERROR", "Error"] }+ : {}),+ }),requestType: "POST",url: attributeKeysUrl,successCb: (resp) => {@@ -391,7 +407,7 @@ const DynamicFilters = ({},});// eslint-disable-next-line react-hooks/exhaustive-deps- }, [attributeKeysUrl]);+ }, [attributeKeysUrl, pageName]);// Fetch attribute keys whenever the time window changes.// Use leaf primitives as deps – lodash merge mutates timeLimit in-place@@ -1280,6 +1296,7 @@ export default function TracesFilter({configUrl={configUrl}attributeKeysUrl={attributeKeysUrl}customAttributeTypes={customAttributeTypes}+ pageName={pageName}/>)}</div>diff --git a/src/client/src/components/(playground)/observability/signal-records.tsx b/src/client/src/components/(playground)/observability/signal-records.tsxindex 12d77c0c..11428dd6 100644--- a/src/client/src/components/(playground)/observability/signal-records.tsx+++ b/src/client/src/components/(playground)/observability/signal-records.tsx@@ -485,15 +485,7 @@ function SessionRecord({onOpen: (row: any) => void;}) {const show = (key: string) => visibilityColumns[key] !== false;- const sessionId = row.session_id || "";- // Cursor session ids are long UUIDs — show enough characters to disambiguate- // at a glance but keep the row visually compact.- const shortSessionId = sessionId- ? sessionId.length > 12- ? `${sessionId.slice(0, 12)}…`- : sessionId- : "—";- const vendor = (row.vendor || "").toLowerCase();+ const sessionId = row.session_id || "—";const totalTokens = Number(row.total_tokens ||Number(row.input_tokens || 0) + Number(row.output_tokens || 0)@@ -522,41 +514,29 @@ function SessionRecord({><div className="flex flex-col gap-3 lg:flex-row lg:items-center lg:justify-between"><div className="min-w-0">- <div className="flex min-w-0 items-start gap-2">- {hasCodingAgentVendorIcon(vendor) ? (- <CodingAgentVendorIcon- vendor={vendor}- className="mt-0.5 h-4 w-4 shrink-0"- />- ) : (- <Bot className="mt-0.5 h-4 w-4 shrink-0 text-violet-500" />+ {/* No vendor icon here — we're already on a vendor detail+ page, and the logo was pushing the session id / email+ out of alignment with the date row below. */}+ <h3+ className="break-all font-mono text-sm font-semibold text-stone-950 dark:text-stone-50"+ title={sessionId}+ >+ {sessionId}+ </h3>+ <div className="mt-0.5 flex flex-wrap items-center gap-x-2 gap-y-0.5 text-xs">+ {show("user") && row.user && (+ <span className="truncate text-xs text-stone-500 dark:text-stone-400">+ {row.user}+ </span>)}- <div className="min-w-0">- <h3- className="truncate font-mono text-sm font-semibold text-stone-950 dark:text-stone-50"- title={sessionId}+ {row.model && (+ <span+ className="truncate rounded bg-stone-100 px-1.5 py-0.5 text-[11px] text-stone-600 dark:bg-stone-900 dark:text-stone-300"+ title={row.model}>- {shortSessionId}- </h3>- <div className="mt-0.5 flex flex-wrap items-center gap-x-2 gap-y-0.5 text-xs">- {/* Vendor pill removed — the vendor icon- already sits to the left of the session id- so showing "cursor" here is redundant. */}- {show("user") && row.user && (- <span className="truncate text-xs text-stone-500 dark:text-stone-400">- {row.user}- </span>- )}- {row.model && (- <span- className="truncate rounded bg-stone-100 px-1.5 py-0.5 text-[11px] text-stone-600 dark:bg-stone-900 dark:text-stone-300"- title={row.model}- >- {row.model}- </span>- )}- </div>- </div>+ {row.model}+ </span>+ )}</div><div className="mt-1 flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-stone-500 dark:text-stone-400">{show("started") && <span>{formatDate(row.started_at)}</span>}@@ -586,6 +566,16 @@ function SessionRecord({running</span>)}+ {Number(row.subagent_event_count || 0) > 0 && (+ <span+ className="rounded px-1.5 py-0.5 text-[11px] font-medium bg-violet-100 text-violet-700 dark:bg-violet-950 dark:text-violet-300"+ title={m.CODING_AGENT_SESSION_SUBAGENTS_HINT}+ >+ {m.CODING_AGENT_SESSION_SUBAGENTS(+ Number(row.subagent_event_count || 0)+ )}+ </span>+ )}{show("classification") &&row.classification &&row.classification !== "unknown" && (diff --git a/src/client/src/components/(playground)/openground/header.tsx b/src/client/src/components/(playground)/openground/header.tsxindex 8d7bf39e..5d62b4ac 100644--- a/src/client/src/components/(playground)/openground/header.tsx+++ b/src/client/src/components/(playground)/openground/header.tsx@@ -34,7 +34,7 @@ export function OpengroundActions({return (<div className={className}>- <Link href="/manage-models">+ <Link href="/costs?tab=models"><Button variant="outline" size="sm" className="h-8"><SettingsIcon className="mr-1.5 size-3.5" />{getMessage().OPENGROUND_MANAGE_MODELS}@@ -68,7 +68,7 @@ export default function OpengroundHeader({return (<FeaturePageHeader- eyebrow="Resources"+ eyebrow={getMessage().SIDEBAR_DEVELOP}title={pageHeaderTitle}icon={<MonitorPlay className="size-4" />}tone="border-indigo-200 bg-indigo-50 text-indigo-700 dark:border-indigo-900/70 dark:bg-indigo-950/40 dark:text-indigo-300"diff --git a/src/client/src/components/(playground)/organisation/organisation-page.tsx b/src/client/src/components/(playground)/organisation/organisation-page.tsxindex 3e321fc1..169393c3 100644--- a/src/client/src/components/(playground)/organisation/organisation-page.tsx+++ b/src/client/src/components/(playground)/organisation/organisation-page.tsx@@ -356,10 +356,12 @@ export default function OrganisationSettingsPage() {</span><div><p className="text-[11px] uppercase tracking-wide text-stone-500 dark:text-stone-400">- {messages.ORGANISATION}+ {messages.SETTINGS}</p><h1 className="text-sm font-semibold leading-tight text-stone-950 dark:text-stone-50">- {messages.ORGANISATION_SETTINGS}+ {selectedTab === "projects"+ ? messages.PROJECTS+ : messages.ORGANISATION}</h1></div></div>diff --git a/src/client/src/components/(playground)/organisation/project-page-header.tsx b/src/client/src/components/(playground)/organisation/project-page-header.tsxindex 2edabbc1..0b55bf46 100644--- a/src/client/src/components/(playground)/organisation/project-page-header.tsx+++ b/src/client/src/components/(playground)/organisation/project-page-header.tsx@@ -26,7 +26,7 @@ export default function ProjectPageHeader({asChildvariant="outline"size="sm"- className="h-8 w-8 shrink-0 p-0"+ className="h-7 w-7 shrink-0 p-0"><Link href="/organisation" title={backLabel} aria-label={backLabel}><ArrowLeft className="h-3.5 w-3.5" />diff --git a/src/client/src/components/(playground)/prompt-hub/header.tsx b/src/client/src/components/(playground)/prompt-hub/header.tsxindex c3ffd959..43dc233f 100644--- a/src/client/src/components/(playground)/prompt-hub/header.tsx+++ b/src/client/src/components/(playground)/prompt-hub/header.tsx@@ -46,7 +46,7 @@ export default function PromptHubHeader({return (<FeaturePageHeader- eyebrow="Resources"+ eyebrow={getMessage().SIDEBAR_DEVELOP}title={pageHeaderTitle}icon={<Component className="h-4 w-4" />}tone="border-pink-200 bg-pink-50 text-pink-700 dark:border-pink-900/70 dark:bg-pink-950/40 dark:text-pink-300"diff --git a/src/client/src/components/(playground)/request/components/chat-view.tsx b/src/client/src/components/(playground)/request/components/chat-view.tsxindex 7b639cc3..3a5404a9 100644--- a/src/client/src/components/(playground)/request/components/chat-view.tsx+++ b/src/client/src/components/(playground)/request/components/chat-view.tsx@@ -37,6 +37,8 @@ import {Wrench,} from "lucide-react";import { useState } from "react";+import getMessage from "@/constants/messages";+import { cn } from "@/lib/utils";// OTel GenAI canonical key for the tool-call id. Kept local so the// chat view doesn't have to import the platform table-details (which@@ -101,9 +103,16 @@ interface ChatItem {function parseTimestampMs(ts?: string): number {if (!ts) return 0;- const withZ = ts.endsWith("Z") ? ts : ts + "Z";+ // ClickHouse JSONEachRow emits `YYYY-MM-DD HH:mm:ss.nnnnnnnnn` (space,+ // nanoseconds, no timezone). Append Z only after normalizing to ISO+ // so browsers that reject the space form still parse as UTC.+ const normalized = ts.includes("T") ? ts : ts.replace(" ", "T");+ const trimmed = normalized.replace(/(\.\d{3})\d+/, "$1");+ const withZ = trimmed.endsWith("Z") ? trimmed : `${trimmed}Z`;const ms = new Date(withZ).getTime();- return isNaN(ms) ? 0 : ms;+ if (!Number.isNaN(ms)) return ms;+ const fallback = new Date(ts).getTime();+ return Number.isNaN(fallback) ? 0 : fallback;}function tryParseJson(raw: unknown): any[] | null {@@ -873,7 +882,7 @@ const roleConfig = {label: "User",align: "items-end" as const,bubble:- "border border-primary/20 bg-primary/5 dark:bg-primary/10 text-stone-800 dark:text-stone-200 rounded-md",+ "border-2 border-sky-400/70 bg-sky-50 dark:border-sky-500/50 dark:bg-sky-950/40 text-stone-900 dark:text-stone-100 rounded-md shadow-sm",},assistant: {icon: <Bot className="h-3.5 w-3.5" />,@@ -917,7 +926,19 @@ export default function ChatView({}: {record: TraceHeirarchySpan;}) {+ const m = getMessage();const [request, updateRequest] = useRequest();+ const isCodingAgent =+ record.SpanName?.startsWith("coding_agent.") ||+ (record.children || []).some((c) =>+ c.SpanName?.startsWith("coding_agent.")+ );+ // Coding-agent sessions are dominated by tool/model-swap noise.+ // Default to Conversation so user/assistant prompts are visible;+ // "All activity" restores the full timeline.+ const [activityFilter, setActivityFilter] = useState<"conversation" | "all">(+ isCodingAgent ? "conversation" : "all"+ );// 1. Flatten treeconst allSpans: TraceHeirarchySpan[] = [];@@ -951,14 +972,9 @@ export default function ChatView({allItems.sort((a, b) => a.timestampMs - b.timestampMs);// 5. Deduplicate messages with identical role + content.- // In multi-step agent traces, each LLM call span carries the full- // conversation history in gen_ai.input.messages — so the same user- // prompt appears in every step. We keep only the first occurrence.- // Tool indicators, span indicators, thoughts, edits, and subagents- // are never deduped (they're unique actions).const seenMessages = new Set<string>();const dedupedItems = allItems.filter((item) => {- if (item.type !== "message") return true; // keep all non-message items+ if (item.type !== "message") return true;const key = `${item.role}::${item.content}`;if (seenMessages.has(key)) return false;seenMessages.add(key);@@ -966,14 +982,6 @@ export default function ChatView({});// 5b. Fold (Task tool-call → subagent) pairs into a single block.- // Claude Code emits PreToolUse(Task) as a separate tool span,- // then a SubagentStop span later. Both carry the same- // gen_ai.tool.call.id (newly stamped by setSubagentAttrs +- // PreToolUse caching in sessionstate). Rendering both would- // show the Task tool indicator AND the Subagent banner —- // redundant. Keep the subagent item, drop the tool indicator- // that pairs with it, and copy the tool's args onto the- // subagent so the user can still see what was requested.const subagentByCallId = new Map<string, ChatItem>();for (const it of dedupedItems) {if (it.type === "subagent" && it.toolCallId) {@@ -987,7 +995,6 @@ export default function ChatView({it.toolCallId &&subagentByCallId.has(it.toolCallId)) {- // Merge the tool args/result onto the subagent item once.const sa = subagentByCallId.get(it.toolCallId)!;if (!sa.toolArgs && it.toolArgs) sa.toolArgs = it.toolArgs;if (!sa.toolResult && it.toolResult) sa.toolResult = it.toolResult;@@ -996,13 +1003,51 @@ export default function ChatView({return true;});- // 5c. Identify the parent chat id we're currently viewing so the- // renderer only indents items belonging to a *different*- // parent (i.e. spans from a subagent that ran inside this- // chat). Items whose parent_id == the chat we're viewing are- // the chat's own spans and stay flush-left.- let viewerChatId = "";+ // 5c. Collapse consecutive identical model-swap pills. Cursor emits+ // model.changed on nearly every turn, burying user bubbles.+ const collapsed: ChatItem[] = [];for (const it of grouped) {+ const prev = collapsed[collapsed.length - 1];+ if (+ it.type === "model-swap" &&+ prev?.type === "model-swap" &&+ prev.fromValue === it.fromValue &&+ prev.toValue === it.toValue+ ) {+ prev.meta = String(Number(prev.meta || "1") + 1);+ continue;+ }+ if (it.type === "model-swap") {+ collapsed.push({ ...it, meta: "1" });+ } else {+ collapsed.push(it);+ }+ }++ const messageCount = collapsed.filter((i) => i.type === "message").length;+ const thoughtCount = collapsed.filter((i) => i.type === "thought").length;+ const toolCount = collapsed.filter(+ (i) => i.type === "tool-indicator"+ ).length;++ // Conversation keeps mid-session model/mode transitions visible —+ // those are part of the chat story. Tool/span noise stays in All.+ const visibleItems =+ activityFilter === "conversation"+ ? collapsed.filter((it) =>+ [+ "message",+ "thought",+ "edit",+ "subagent",+ "model-swap",+ "mode-transition",+ ].includes(it.type)+ )+ : collapsed;++ let viewerChatId = "";+ for (const it of collapsed) {const sa = it.span.SpanAttributes || {};const ra = it.span.ResourceAttributes || {};const ownId =@@ -1019,34 +1064,56 @@ export default function ChatView({const isSubagentItem = (it: ChatItem) =>!!it.parentId && it.parentId !== viewerChatId;- // 6. Empty state- if (grouped.length === 0) {+ if (collapsed.length === 0) {return (<div className="flex flex-col items-center justify-center h-full text-stone-400 dark:text-stone-500 gap-2 py-12"><MessageSquare className="h-8 w-8" />- <p className="text-sm">No chat messages found in this trace</p>- <p className="text-xs">- This view works with LLM spans that have prompt/response data- </p>+ <p className="text-sm">{m.OBSERVABILITY_CHAT_EMPTY}</p>+ <p className="text-xs">{m.OBSERVABILITY_CHAT_EMPTY_HINT}</p></div>);}return (<div className="flex flex-col gap-2 py-2">- {grouped.map((item, i) => {+ <div className="sticky top-0 z-10 mx-2 flex flex-wrap items-center gap-2 rounded-md border border-stone-200 bg-white/95 px-2 py-1.5 backdrop-blur dark:border-stone-800 dark:bg-stone-950/95">+ <div className="flex rounded-md border border-stone-200 p-0.5 dark:border-stone-800">+ {(+ [+ ["conversation", m.OBSERVABILITY_CHAT_FILTER_CONVERSATION],+ ["all", m.OBSERVABILITY_CHAT_FILTER_ALL],+ ] as const+ ).map(([key, label]) => (+ <button+ key={key}+ type="button"+ onClick={() => setActivityFilter(key)}+ className={cn(+ "rounded px-2 py-0.5 text-[11px] font-medium transition-colors",+ activityFilter === key+ ? "bg-primary text-white"+ : "text-stone-500 hover:bg-stone-100 dark:text-stone-400 dark:hover:bg-stone-800"+ )}+ >+ {label}+ </button>+ ))}+ </div>+ <span className="text-[11px] text-stone-500 dark:text-stone-400">+ {m.OBSERVABILITY_CHAT_SUMMARY(+ messageCount,+ thoughtCount,+ toolCount+ )}+ </span>+ </div>+ {visibleItems.map((item, i) => {const isSelected = request?.spanId === item.span.SpanId;const indented = isSubagentItem(item);- // Subagent items rendered indented with a thin left- // border so the user can see at a glance which part- // of the chat ran inside a subagent and which is the- // parent agent's own work. The border colour matches- // the SubagentIndicator's violet accent.const indentClass = indented? "ml-6 border-l-2 border-violet-300/60 pl-2 dark:border-violet-700/50": "";- // ── Mode transition pill ──if (item.type === "mode-transition") {return (<TransitionPill@@ -1058,19 +1125,22 @@ export default function ChatView({);}- // ── Model swap pill ──if (item.type === "model-swap") {+ const swapCount = Number(item.meta || "1");return (<TransitionPillkey={`${item.span.SpanId}-model-${i}`}item={item}- prefix="Model"+ prefix={+ swapCount > 1+ ? m.OBSERVABILITY_CHAT_MODEL_CHANGES(swapCount)+ : "Model"+ }indented={indented}/>);}- // ── Tool call indicator ──if (item.type === "tool-indicator") {return (<div key={`${item.span.SpanId}-tool-${i}`} className={indentClass}>@@ -1085,10 +1155,12 @@ export default function ChatView({);}- // ── Thinking / reasoning text ──if (item.type === "thought") {return (- <div key={`${item.span.SpanId}-thought-${i}`} className={indentClass}>+ <div+ key={`${item.span.SpanId}-thought-${i}`}+ className={indentClass}+ ><ThoughtIndicatoritem={item}isSelected={isSelected}@@ -1100,7 +1172,6 @@ export default function ChatView({);}- // ── Edit decision ──if (item.type === "edit") {return (<div key={`${item.span.SpanId}-edit-${i}`} className={indentClass}>@@ -1115,10 +1186,6 @@ export default function ChatView({);}- // ── Subagent — always shown flush-left even when it- // represents inner-subagent activity, because the- // block itself is the visual anchor for that- // "subagent triggered" moment.if (item.type === "subagent") {return (<SubagentIndicator@@ -1132,7 +1199,6 @@ export default function ChatView({);}- // ── Non-LLM span indicator ──if (item.type === "span-indicator") {return (<div@@ -1162,9 +1228,9 @@ export default function ChatView({);}- // ── Chat message bubble ──const role = item.role!;const config = roleConfig[role];+ if (!config) return null;const durationDisplay = getSpanDurationDisplay(item.span);const costDisplay = getSpanCostFormatted(item.span, 6);@@ -1173,7 +1239,6 @@ export default function ChatView({key={`${item.span.SpanId}-${role}-${i}`}className={`flex flex-col gap-0.5 px-2 ${config.align} ${indentClass}`}>- {/* Role label + span name */}<divclassName={`flex items-center gap-1.5 px-1 text-[10px] text-stone-400 dark:text-stone-500 ${role === "user" ? "flex-row-reverse" : ""@@ -1189,7 +1254,6 @@ export default function ChatView({</span></div>- {/* Bubble */}<divclassName={`max-w-[92%] cursor-pointer whitespace-pre-wrap break-words px-2.5 py-2 text-[12px] leading-relaxed transition-colors ${config.bubble@@ -1205,7 +1269,6 @@ export default function ChatView({{item.content}</div>- {/* Duration + cost for assistant messages */}{role === "assistant" && (<div className="flex items-center gap-2 text-[9px] text-stone-400 dark:text-stone-500 px-1 mt-0.5"><span className="flex items-center gap-0.5">diff --git a/src/client/src/components/(playground)/request/components/evaluations.tsx b/src/client/src/components/(playground)/request/components/evaluations.tsxindex f3568424..2d62617f 100644--- a/src/client/src/components/(playground)/request/components/evaluations.tsx+++ b/src/client/src/components/(playground)/request/components/evaluations.tsx@@ -533,7 +533,7 @@ export default function Evaluations({{getMessage().EVALUATION_CONFIG_NOT_SET}</div><Button variant="destructive" className="w-fit">- <Link href="/evaluations/settings">+ <Link href="/evaluations?tab=configuration">{getMessage().EVALUATION_CONFIG_SET}</Link></Button>diff --git a/src/client/src/components/(playground)/request/grouped-table.tsx b/src/client/src/components/(playground)/request/grouped-table.tsxindex 8fc970a4..d5e8ebe3 100644--- a/src/client/src/components/(playground)/request/grouped-table.tsx+++ b/src/client/src/components/(playground)/request/grouped-table.tsx@@ -13,7 +13,7 @@ import { PRIMARY_BACKGROUND } from "@/constants/common-classes";const PREDEFINED_FILTER_MAP: Record<string, { attributeType: CustomFilterAttributeType; key: string }> = {model: { attributeType: "SpanAttributes", key: "gen_ai.request.model" },- provider: { attributeType: "SpanAttributes", key: "gen_ai.system" },+ provider: { attributeType: "SpanAttributes", key: "gen_ai.provider.name" },spanName: { attributeType: "Field", key: "SpanName" },applicationName: { attributeType: "ResourceAttributes", key: "service.name" },};diff --git a/src/client/src/components/(playground)/rule-engine/header.tsx b/src/client/src/components/(playground)/rule-engine/header.tsxindex 19139f37..c48983b9 100644--- a/src/client/src/components/(playground)/rule-engine/header.tsx+++ b/src/client/src/components/(playground)/rule-engine/header.tsx@@ -33,5 +33,5 @@ export default function RuleEngineHeader({</div>);- return <FeaturePageHeader eyebrow="Resources" title={messages.RULE_ENGINE_BREADCRUMB} icon={<SlidersHorizontal className="h-4 w-4" />} tone="border-purple-200 bg-purple-50 text-purple-700 dark:border-purple-900/70 dark:bg-purple-950/40 dark:text-purple-300" actions={actions} />;+ return <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={messages.RULE_ENGINE_BREADCRUMB} icon={<SlidersHorizontal className="h-4 w-4" />} tone="border-purple-200 bg-purple-50 text-purple-700 dark:border-purple-900/70 dark:bg-purple-950/40 dark:text-purple-300" actions={actions} />;}diff --git a/src/client/src/components/(playground)/sidebar/index.tsx b/src/client/src/components/(playground)/sidebar/index.tsxindex e55bac0c..7590718a 100644--- a/src/client/src/components/(playground)/sidebar/index.tsx+++ b/src/client/src/components/(playground)/sidebar/index.tsx@@ -39,6 +39,14 @@ import { useSidebarLayout } from "../sidebar-layout-context";const isActive = (pathname: string, item: SidebarActionItem, currentUrl: string) => {if (!item.link) return false;if (item.link.includes("?")) return currentUrl.startsWith(item.link);+ // Organisation owns `/organisation` except the Projects tab, which has its+ // own Settings entry (`/organisation?tab=projects`).+ if (item.link === "/organisation") {+ return (+ pathname.startsWith("/organisation") &&+ !currentUrl.includes("tab=projects")+ );+ }if (item.link === "/dashboards") return pathname.startsWith("/dashboards") || pathname.startsWith("/d/");if (item.link === "/dashboard") return pathname.startsWith("/dashboard") && !pathname.startsWith("/dashboards");return pathname.startsWith(item.link);@@ -154,7 +162,7 @@ function SectionPanel({ section, pathname, currentUrl, onClose }: { section: Sid<div className="space-y-3">{groups.map((group) => (<div key={group.title} className="space-y-0.5">- <p className="px-2.5 pb-0.5 pt-1 text-xs font-semibold uppercase tracking-wide text-stone-400 dark:text-stone-500">{group.title}</p>+ <p className="px-2.5 pb-0.5 pt-1 text-xs font-semibold tracking-wide text-stone-400 dark:text-stone-500">{group.title}</p>{group.children.map((item) => <NavigationLink key={item.text} item={item} active={isActive(pathname, item, currentUrl)} onNavigate={onClose} />)}</div>))}diff --git a/src/client/src/components/(playground)/sidebar/my-apps.tsx b/src/client/src/components/(playground)/sidebar/my-apps.tsxindex 7c10383e..40d748d9 100644--- a/src/client/src/components/(playground)/sidebar/my-apps.tsx+++ b/src/client/src/components/(playground)/sidebar/my-apps.tsx@@ -240,7 +240,7 @@ export default function MyApps({return (<div key={group.title} className="mb-1.5">- <p className="px-2.5 pb-0.5 pt-1 text-xs font-medium uppercase tracking-wide text-stone-400 dark:text-stone-500">+ <p className="px-2.5 pb-0.5 pt-1 text-xs font-medium tracking-wide text-stone-400 dark:text-stone-500">{group.title}</p>{visible.map((item) =>diff --git a/src/client/src/components/(playground)/vault/header.tsx b/src/client/src/components/(playground)/vault/header.tsxindex 4a98b2e7..930f750b 100644--- a/src/client/src/components/(playground)/vault/header.tsx+++ b/src/client/src/components/(playground)/vault/header.tsx@@ -33,5 +33,5 @@ export default function VaultHeader({</div>);- return <FeaturePageHeader eyebrow="Resources" title={messages.FEATURE_VAULT} icon={<BookKey className="h-4 w-4" />} tone="border-amber-200 bg-amber-50 text-amber-700 dark:border-amber-900/70 dark:bg-amber-950/40 dark:text-amber-300" actions={actions} />;+ return <FeaturePageHeader eyebrow={getMessage().SIDEBAR_DEVELOP} title={messages.FEATURE_VAULT} icon={<BookKey className="h-4 w-4" />} tone="border-amber-200 bg-amber-50 text-amber-700 dark:border-amber-900/70 dark:bg-amber-950/40 dark:text-amber-300" actions={actions} />;}diff --git a/src/client/src/components/data-table/columns.tsx b/src/client/src/components/data-table/columns.tsxindex 642acbd0..4366aec0 100644--- a/src/client/src/components/data-table/columns.tsx+++ b/src/client/src/components/data-table/columns.tsx@@ -7,6 +7,8 @@ export type Columns<T extends string, TransformedT> = Partial<header: () => ReactNode;cell: ({ row, extraFunctions }: { row: TransformedT, extraFunctions: Record<string, any> }) => ReactNode;enableHiding?: boolean;+ /** Optional CSS grid track size (e.g. `2.75rem` or `minmax(10rem, 1.4fr)`). */+ width?: string;}>>;diff --git a/src/client/src/components/data-table/table.tsx b/src/client/src/components/data-table/table.tsxindex 6ee7df63..fddb0c1f 100644--- a/src/client/src/components/data-table/table.tsx+++ b/src/client/src/components/data-table/table.tsx@@ -88,7 +88,9 @@ export default function DataTable({const noData = !data?.length && !isLoading;const style: CSSProperties = {- gridTemplateColumns: `repeat(${visibleColumns.length}, minmax(0, 1fr))`,+ gridTemplateColumns: visibleColumns+ .map((column) => columns[column]?.width || "minmax(0, 1fr)")+ .join(" "),};const onClickHandler = (rowItem: any) =>@@ -100,7 +102,7 @@ export default function DataTable({{visibleColumns.map((column) => (<ColumnRowItemkey={column}- className={`group-last-of-type:border-b-0 bg-stone-100 text-stone-500 dark:bg-stone-900 dark:text-stone-400 text-sm overflow-hidden text-ellipsis whitespace-nowrap`}+ className={`group-last-of-type:border-b-0 bg-stone-100 text-stone-500 dark:bg-stone-900 dark:text-stone-400 text-sm whitespace-nowrap`}>{columns[column]?.header()}</ColumnRowItem>diff --git a/src/client/src/constants/messages/en.ts b/src/client/src/constants/messages/en.tsindex 3a635b12..91fef011 100644--- a/src/client/src/constants/messages/en.ts+++ b/src/client/src/constants/messages/en.ts@@ -85,7 +85,7 @@ export const EVALUATION_NO_RESULTS = "No evaluations";// Evaluation Settings pageexport const EVALUATION_VAULT_KEY_NOT_FOUND = "Unable to find the vault key.";export const EVALUATION_CREATE_NEW = "Create new";-export const EVALUATION_ENGINE_TITLE = "Evaluation Engine";+export const EVALUATION_ENGINE_TITLE = "Evaluations";export const EVALUATION_ENGINE_DESCRIPTION ="Choose the evaluation framework. Rule engine context and evaluation types are applied for both manual and auto runs.";export const EVALUATION_ENGINE_LABEL = "Engine";@@ -145,6 +145,70 @@ export const EVALUATION_FEEDBACK_COMMENT_PLACEHOLDER = "Optional comment...";export const EVALUATION_FEEDBACK_SUBMIT = "Submit Feedback";export const EVALUATION_FEEDBACK_SAVED = "Feedback saved!";+// Evaluation Analytics+export const EVALUATION_TAB_ANALYTICS = "Analytics";+export const EVALUATION_TAB_EVALUATORS = "Evaluators";+export const EVALUATION_TAB_CONFIGURATION = "Configuration";+export const EVALUATION_STAT_EVALUATIONS = "Evaluations";+export const EVALUATION_STAT_ACTIVE = "Active";+export const EVALUATION_STAT_TRACES_EVALUATED = "Traces evaluated";+export const EVALUATION_STAT_EXECUTIONS = "Executions";+export const EVALUATION_STAT_TOTAL_COST = "Total cost";+export const EVALUATION_STAT_AVG_PASS_RATE = "Avg pass rate";+export const EVALUATION_STAT_FAILED_SCORES = "Scores marked as failed";+export const EVALUATION_STAT_AUTO_EXECUTIONS = "Auto executions";+export const EVALUATION_STAT_EVALUATORS = "Evaluators";+export const EVALUATION_CHART_PASS_RATE = "Pass rate";+export const EVALUATION_CHART_EXECUTIONS = "Executions";+export const EVALUATION_TABLE_EVALUATION = "Evaluation";+export const EVALUATION_TABLE_MATCHER = "Matcher";+export const EVALUATION_TABLE_MATCHER_DEFAULT = "*";+export const EVALUATION_TABLE_EXECUTIONS = "Executions";+export const EVALUATION_TABLE_PASS_RATE = "Pass rate";+export const EVALUATION_TABLE_PASS_RATE_TREND = "Pass rate trend";+export const EVALUATION_TABLE_EMPTY = "No evaluation results in this time range.";+export const EVALUATION_TABLE_EVALUATORS_TITLE = "Evaluators";+export const EVALUATION_DETAIL_CONFIGURE = "Configure";+export const EVALUATION_DETAIL_ENABLED = "Enabled";+export const EVALUATION_DETAIL_DISABLED = "Disabled";+export const EVALUATION_DETAIL_RESULTS = "Results";+export const EVALUATION_DETAIL_RESULTS_DESCRIPTION =+ "Evaluation results for this evaluator, including pass rates and recent runs.";+export const EVALUATION_DETAIL_NO_RESULTS = "No recent results for this evaluator.";+export const EVALUATION_DETAIL_PASS_RATE_BY_EVALUATOR = "Pass rate";+export const EVALUATION_DETAIL_SAMPLE_META = "sample";+export const EVALUATION_DETAIL_PASS = "Pass";+export const EVALUATION_DETAIL_FAIL = "Fail";+export const EVALUATION_SCORE = "Score";+export const EVALUATION_BACK_TO_ANALYTICS = "Back to Analytics";+export const EVALUATION_ONBOARDING_TITLE = "Set up Evaluations";+export const EVALUATION_ONBOARDING_DESCRIPTION =+ "Configure an evaluation engine to score traces for hallucination, bias, toxicity, and custom criteria.";+export const EVALUATION_ONBOARDING_STEP_1 =+ "Choose a provider, model, and vault API key under Configuration.";+export const EVALUATION_ONBOARDING_STEP_2 =+ "Enable built-in types or create custom evaluators.";+export const EVALUATION_ONBOARDING_STEP_3 =+ "Turn on auto evaluation or run evaluations manually from Telemetry.";+export const EVALUATION_ONBOARDING_CTA = "Open Configuration";+export const EVALUATION_POWERED_BY_VERCEL = "Powered by";+export const EVALUATION_POWERED_BY_VERCEL_SDK = "Vercel AI SDK";+export const EVALUATION_TYPES_SECTION_TITLE = "Evaluation types";+export const EVALUATION_TYPES_SECTION_DESCRIPTION =+ "Enable built-in evaluators or create custom ones for your use case.";+export const EVALUATION_CREATE_CUSTOM_TYPE = "Create custom evaluator";+export const EVALUATION_CUSTOM_TYPES_HEADING = "Custom evaluation types";+export const EVALUATION_TYPES_LINKED_FROM_RULES = "Linked from rules:";+export const EVALUATION_TYPE_ENABLED = "Enabled";+export const EVALUATION_TYPE_DISABLED = "Disabled";+export const EVALUATION_TYPE_CUSTOM_BADGE = "Custom";+export const EVALUATION_TYPE_RULE_ENGINE_BADGE = "Rule engine";++export const EVALUATION_TYPE_NOT_FOUND = "Evaluation type not found";+export const EVALUATION_TYPE_NOT_FOUND_DESCRIPTION =+ "This evaluator id does not match a built-in or custom evaluation type. Open Configuration to manage evaluators.";+export const EVALUATION_BACK_TO_TYPES = "Back to evaluation types";+// Tracesexport const TRACE_NOT_FOUND = "Trace not found!";export const TRACE_FETCHING_ERROR = "Error fetching trace!";@@ -243,6 +307,18 @@ export const OBSERVABILITY_HIERARCHY_UNAVAILABLE ="Span hierarchy is not available for this span.";export const OBSERVABILITY_TREE = "Tree";export const OBSERVABILITY_CHAT = "Chat";+export const OBSERVABILITY_CHAT_FILTER_CONVERSATION = "Conversation";+export const OBSERVABILITY_CHAT_FILTER_ALL = "All activity";+export const OBSERVABILITY_CHAT_SUMMARY = (+ messages: number,+ thoughts: number,+ tools: number+) => `${messages} messages · ${thoughts} thoughts · ${tools} tools`;+export const OBSERVABILITY_CHAT_EMPTY = "No chat messages found in this trace";+export const OBSERVABILITY_CHAT_EMPTY_HINT =+ "This view works with LLM spans that have prompt/response data";+export const OBSERVABILITY_CHAT_MODEL_CHANGES = (count: number) =>+ count === 1 ? "1 model change" : `${count} model changes`;export const OBSERVABILITY_TIMELINE = "Timeline";export const OBSERVABILITY_GRAPH = "Graph";export const OBSERVABILITY_NO_SERVER_CONNECTION = "Cannot connect to server!";@@ -379,7 +455,7 @@ export const OPENGROUND_CUSTOM_MODEL = "Custom Model";export const OPENGROUND_ENTER_CUSTOM_MODEL_NAME = "Enter custom model name";export const OPENGROUND_USE_CUSTOM_MODEL = "Use Custom Model";export const OPENGROUND_OR_ENTER_CUSTOM = "or enter custom";-export const OPENGROUND_MANAGE_MODELS = "Manage Models";+export const OPENGROUND_MANAGE_MODELS = "Manage models";export const OPENGROUND_ADD_NEW_MODEL = "Add New Model";export const OPENGROUND_EDIT_MODEL = "Edit Model";export const OPENGROUND_MODEL_ID = "Model ID";@@ -446,7 +522,51 @@ export const MANAGE_PROVIDERS_DISPLAY_NAME = "Display Name";export const MANAGE_PROVIDERS_DESCRIPTION = "Description";export const MANAGE_PROVIDERS_REQUIRES_VAULT = "Requires API Key (Vault)";-// Pricing+// Costs+export const COSTS_TITLE = "Costs";+export const COSTS_TAB_ANALYTICS = "Analytics";+export const COSTS_TAB_MODELS = "Manage models";+export const COSTS_TAB_CONFIGURATION = "Configuration";+export const COSTS_USAGE_SECTION = "Cost usage";+export const COSTS_OPTIMIZATION_SECTION = "Cost optimization";+export const COSTS_AUTO_PRICING_SECTION = "Auto pricing runs";+export const COSTS_STAT_TOTAL_PLATFORM = "Total AI cost";+export const COSTS_STAT_LLM = "App LLM cost";+export const COSTS_STAT_CODING_AGENTS = "Coding agents";+export const COSTS_STAT_OTTER = "Otter";+export const COSTS_STAT_EVALUATIONS = "Evaluations cost";+export const COSTS_STAT_OPENGROUND = "Openground cost";+export const COSTS_STAT_AVG_REQUEST = "Avg cost per request";+export const COSTS_STAT_TOTAL_REQUESTS = "Total LLM requests";+export const COSTS_STAT_CRON_RUNS = "Cron runs";+export const COSTS_STAT_CRON_SUCCESS = "Successful runs";+export const COSTS_STAT_PRICING_APPLIED = "Costs applied";+export const COSTS_STAT_SPANS_SCANNED = "Spans scanned";+export const COSTS_CRON_TABLE_STARTED = "Started";+export const COSTS_CRON_TABLE_STATUS = "Status";+export const COSTS_CRON_TABLE_DURATION = "Duration";+export const COSTS_CRON_TABLE_UPDATED = "Costs applied";+export const COSTS_CRON_TABLE_SCANNED = "Scanned";+export const COSTS_CRON_TABLE_SKIPPED = "Skipped";+export const COSTS_CRON_TABLE_FAILED = "Failed";+export const COSTS_CRON_TABLE_EMPTY =+ "No Auto Pricing runs applied costs in this time range.";+export const COSTS_CRON_TABLE_EMPTY_CTA = "Enable Auto Pricing";+export const COSTS_CRON_TABLE_EMPTY_HINT =+ "Turn on Auto Pricing to backfill missing LLM costs on a schedule.";+export const COSTS_BACKFILL_BANNER_TITLE = "Some LLM traces are missing cost";+export const COSTS_BACKFILL_BANNER_DESCRIPTION =+ "Auto Pricing can backfill gen_ai.usage.cost for spans that arrived without a price. Existing spend charts still work for costs already on your traces.";+export const COSTS_BACKFILL_BANNER_CTA = "Configure Auto Pricing";+export const COSTS_BACKFILL_BANNER_COUNT = "spans missing cost in this range";+export const COSTS_CHART_BY_PROVIDER = "Cost by provider";+export const COSTS_CHART_BY_MODEL = "Cost by model";+export const COSTS_CHART_BY_ENVIRONMENT = "Cost by environment";+export const COSTS_CHART_BY_APPLICATION = "Cost by application";+export const COSTS_CHART_OVER_TIME = "Cost over time";+export const COSTS_CHART_OPENGROUND_BY_PROVIDER = "Openground spend by provider";++// Pricing (configuration tab copy)export const PRICING_TITLE = "Pricing";export const PRICING_PAGE_DESCRIPTION ="Recalculate the cost of LLM traces using the current model pricing in Manage Models. Updates the `gen_ai.usage.cost` attribute on existing traces.";@@ -507,6 +627,13 @@ export const FEATURE_VAULT = "Vault";export const FEATURE_FLEET_HUB = "Fleet Hub";export const FLEET_HUB_BACK_TO_LIST = "Back to Fleet Hub";export const FEATURE_AGENTS = "Agents";+export const FEATURE_EVALS = "Evaluations";+export const FEATURE_MANAGE_MODELS = "Manage models";++export const SIDEBAR_MONITOR = "Monitor";+export const SIDEBAR_DEVELOP = "Develop";+export const SIDEBAR_CONFIGURATION = "Configure";+export const SIDEBAR_PROJECTS = "Projects";// Agentsexport const AGENTS_FILTER_SYSTEM = "System";@@ -545,6 +672,10 @@ export const AGENTS_NO_CODING_AGENTS_DESCRIPTION = "Pick a tool to see the instaexport const CODING_AGENT_CONTENT_CAPTURE_NOTE_TITLE = "Content capture is off";export const CODING_AGENT_CONTENT_CAPTURE_NOTE_COMMAND ="openlit configure --content-capture full";+export const CODING_AGENT_SESSION_SUBAGENTS = (count: number) =>+ count === 1 ? "1 subagent" : `${count} subagents`;+export const CODING_AGENT_SESSION_SUBAGENTS_HINT =+ "Linked Task / background worker activity folded under this chat";// Agents - No Controllerexport const AGENTS_NO_CONTROLLERS_TITLE = "No controllers detected";@@ -573,6 +704,7 @@ export const AGENTS_PODS_ACK_PROGRESS = (ack: number, total: number) =>`Pods: ${ack}/${total} acknowledged`;export const AGENTS_COLUMN_SERVICE = "Name";export const AGENTS_COLUMN_SYSTEM = "System";+export const AGENTS_COLUMN_RUNTIME = "Runtime";export const AGENTS_COLUMN_PROVIDERS = "Providers";// Coding-agent stat / column labels intentionally omit a window// suffix (no "24h"). The underlying data is always evaluateddiff --git a/src/client/src/constants/messages/hi.ts b/src/client/src/constants/messages/hi.tsindex 53ebfb99..eb7e035f 100644--- a/src/client/src/constants/messages/hi.ts+++ b/src/client/src/constants/messages/hi.ts@@ -50,7 +50,7 @@ export const EVALUATION_RUN_FAILURE = "इवैल्यूएशन चला// Evaluation Settings pageexport const EVALUATION_VAULT_KEY_NOT_FOUND = "वॉल्ट कुंजी नहीं मिली।";export const EVALUATION_CREATE_NEW = "नया बनाएं";-export const EVALUATION_ENGINE_TITLE = "इवैल्यूएशन इंजन";+export const EVALUATION_ENGINE_TITLE = "मूल्यांकन";export const EVALUATION_ENGINE_DESCRIPTION ="इवैल्यूएशन फ्रेमवर्क चुनें। रूल इंजन कॉन्टेक्स्ट और इवैल्यूएशन टाइप मैनुअल और ऑटो दोनों रन के लिए लागू होते हैं।";export const EVALUATION_ENGINE_LABEL = "इंजन";diff --git a/src/client/src/constants/sidebar.tsx b/src/client/src/constants/sidebar.tsxindex 341ade66..ecb2f88f 100644--- a/src/client/src/constants/sidebar.tsx+++ b/src/client/src/constants/sidebar.tsx@@ -1,5 +1,5 @@-import OpenTelemetrySvg from "@/components/svg/opentelemetry";import { getEnterpriseSidebarItems } from "@/features/sidebar";+import getMessage from "@/constants/messages";import { SidebarItemProps } from "@/types/sidebar";import {Activity,@@ -11,7 +11,7 @@ import {Building2,CircleDollarSign,Component,- Cpu,+ FolderKanban,Home,Key,LayoutDashboard,@@ -24,6 +24,8 @@ import {export const ICON_CLASSES = "flex-shrink-0 size-4";+const m = getMessage();+export const SIDEBAR_ITEMS: SidebarItemProps[] = [{icon: <Home className={ICON_CLASSES} />,@@ -43,7 +45,7 @@ export const SIDEBAR_ITEMS: SidebarItemProps[] = [icon: <Boxes className={ICON_CLASSES} />,groups: [{- title: "Monitoring",+ title: m.SIDEBAR_MONITOR,children: [{icon: <Activity className={ICON_CLASSES} />,@@ -59,15 +61,22 @@ export const SIDEBAR_ITEMS: SidebarItemProps[] = [type: "action",},{- icon: <OpenTelemetrySvg className={ICON_CLASSES} />,- text: "Fleet Hub",- link: "/fleet-hub",+ icon: <MonitorCog className={ICON_CLASSES} />,+ text: m.FEATURE_EVALS,+ link: "/evaluations",type: "action",},+ {+ icon: <CircleDollarSign className={ICON_CLASSES} />,+ text: m.COSTS_TITLE,+ link: "/costs",+ type: "action",+ },+ ...getEnterpriseSidebarItems("configuration", ICON_CLASSES),],},{- title: "Resources",+ title: m.SIDEBAR_DEVELOP,children: [{icon: <Component className={ICON_CLASSES} />,@@ -101,36 +110,6 @@ export const SIDEBAR_ITEMS: SidebarItemProps[] = [},],},- {- title: "Configuration",- children: [- {- icon: <MonitorCog className={ICON_CLASSES} />,- text: "Evaluations",- link: "/evaluations",- type: "action",- },- {- icon: <Cpu className={ICON_CLASSES} />,- text: "Manage Models",- link: "/manage-models",- type: "action",- },- {- icon: <CircleDollarSign className={ICON_CLASSES} />,- text: "Pricing",- link: "/pricing",- type: "action",- },- {- icon: <Building2 className={ICON_CLASSES} />,- text: "Organisation",- link: "/organisation",- type: "action",- },- ...getEnterpriseSidebarItems("configuration", ICON_CLASSES),- ],- },],},{@@ -139,15 +118,27 @@ export const SIDEBAR_ITEMS: SidebarItemProps[] = [collapsible: true,icon: <SettingsIcon className={ICON_CLASSES} />,children: [+ {+ icon: <Building2 className={ICON_CLASSES} />,+ text: m.ORGANISATION,+ link: "/organisation",+ type: "action",+ },+ {+ icon: <FolderKanban className={ICON_CLASSES} />,+ text: m.SIDEBAR_PROJECTS,+ link: "/organisation?tab=projects",+ type: "action",+ },{icon: <User className={ICON_CLASSES} />,- text: "User Profile",+ text: m.USER_PROFILE,link: "/settings/profile",type: "action",},{icon: <Key className={ICON_CLASSES} />,- text: "Api Keys",+ text: m.API_KEYS,link: "/settings/api-keys",type: "action",},diff --git a/src/client/src/constants/traces.ts b/src/client/src/constants/traces.tsindex c3c1ab5e..ab10c632 100644--- a/src/client/src/constants/traces.ts+++ b/src/client/src/constants/traces.ts@@ -144,7 +144,10 @@ export const TraceMapping: Record<TraceMappingKeyType, TraceMappingValueType> =provider: {label: "Provider",type: "string",- path: "system",+ // OTel GenAI renamed gen_ai.system -> gen_ai.provider.name (1.30+).+ // Prefer the new key; keep system as a fallback for legacy spans.+ path: "provider.name",+ paths: ["provider.name", "system"],prefix: SpanAttributesGenAIPrefix,icon: PyramidIcon,},diff --git a/src/client/src/helpers/client/evaluation-type.ts b/src/client/src/helpers/client/evaluation-type.tsnew file mode 100644index 00000000..9dc1e875--- /dev/null+++ b/src/client/src/helpers/client/evaluation-type.ts@@ -0,0 +1,107 @@+import { EVALUATION_TYPES } from "@/constants/evaluation-types";++export type EvaluationTypeRef = {+ id: string;+ label: string;+};++/**+ * ClickHouse often stores the judge's evaluation name as the label+ * ("Hallucination") or even the full context header+ * ("Hallucination evaluation context"). Routes and config use snake_case ids.+ */+export function normalizeEvaluationStoredName(raw: string): string {+ return String(raw || "")+ .replace(/\s+evaluation\s+context$/i, "")+ .trim();+}++function formatFallbackLabel(value: string): string {+ if (!value) return value;+ if (value.includes(" ") || /[A-Z]/.test(value.slice(1))) {+ return value;+ }+ return value+ .split("_")+ .filter(Boolean)+ .map((w) => w.charAt(0).toUpperCase() + w.slice(1).toLowerCase())+ .join(" ");+}++export function resolveEvaluationType(+ stored: string,+ customTypes: EvaluationTypeRef[] = []+): EvaluationTypeRef | null {+ const cleaned = normalizeEvaluationStoredName(stored);+ if (!cleaned) return null;++ const lower = cleaned.toLowerCase();+ const compact = lower.replace(/[\s_-]+/g, "");++ const builtInById = EVALUATION_TYPES.find(+ (t) => t.id === cleaned || t.id === lower+ );+ if (builtInById) {+ return { id: builtInById.id, label: builtInById.label };+ }++ const builtInByLabel = EVALUATION_TYPES.find(+ (t) =>+ t.label.toLowerCase() === lower ||+ t.label.toLowerCase().replace(/[\s_-]+/g, "") === compact+ );+ if (builtInByLabel) {+ return { id: builtInByLabel.id, label: builtInByLabel.label };+ }++ const customById = customTypes.find(+ (t) => t.id === cleaned || t.id.toLowerCase() === lower+ );+ if (customById) {+ return {+ id: customById.id,+ label: customById.label || formatFallbackLabel(customById.id),+ };+ }++ const customByLabel = customTypes.find((t) => {+ const label = (t.label || "").toLowerCase();+ return (+ label === lower || label.replace(/[\s_-]+/g, "") === compact+ );+ });+ if (customByLabel) {+ return {+ id: customByLabel.id,+ label: customByLabel.label || formatFallbackLabel(customByLabel.id),+ };+ }++ return null;+}++export function displayEvaluationTypeName(+ stored: string,+ customTypes: EvaluationTypeRef[] = []+): string {+ const resolved = resolveEvaluationType(stored, customTypes);+ if (resolved) return resolved.label;+ return formatFallbackLabel(normalizeEvaluationStoredName(stored) || stored);+}++export function getEvaluationStoredNameVariants(+ id: string,+ label?: string+): string[] {+ const resolvedLabel =+ label ||+ EVALUATION_TYPES.find((t) => t.id === id)?.label ||+ id;+ const variants = new Set<string>([+ id,+ resolvedLabel,+ `${resolvedLabel} evaluation context`,+ `${id} evaluation context`,+ ]);+ return Array.from(variants).filter(Boolean);+}diff --git a/src/client/src/helpers/client/observability.ts b/src/client/src/helpers/client/observability.tsindex 83e6634d..de04e945 100644--- a/src/client/src/helpers/client/observability.ts+++ b/src/client/src/helpers/client/observability.ts@@ -15,13 +15,24 @@ export type UpdateConfigFn = (config?: FilterConfig) => void;// Leaving it set leaks ORDER BY clauses across signals and causes// the new tab to fall back to an inappropriate default ordering at// best — or a 500 at worst when the SQL column doesn't exist.-// Reset sort and pagination alongside the existing groupBy/config-// reset so each tab starts from a clean slate.+//+// selectedConfig must be cleared too. Each signal owns its own+// localStorage scope (`openlit_filter_v1:<signal>`), so preserving+// the previous tab's filters in the Zustand store defeats that and+// silently empties the new tab. The concrete failure mode: Traces+// `providers=openai` surviving into Exceptions, where every+// StatusCode=Error span from a failed LLM call has empty+// gen_ai.provider.name / gen_ai.system (the SDK never stamped them+// before the error) — the provider WHERE matches zero rows and the+// Exceptions tab renders empty despite hundreds of errors in CH.+// After the wipe, the new tab's useFilterUrlSync re-applies that+// signal's own persisted filters from localStorage / URL.export function prepareObservabilitySignalChange(updateConfig: UpdateConfigFn,updateFilter: UpdateFilterFn) {updateConfig(undefined);+ updateFilter("selectedConfig", {}, { clearFilter: true });updateFilter("groupBy", null);updateFilter("groupValue", null);updateFilter("sorting", DEFAULT_SORTING);diff --git a/src/client/src/helpers/server/platform.ts b/src/client/src/helpers/server/platform.tsindex f22fbbdb..051dd0ae 100644--- a/src/client/src/helpers/server/platform.ts+++ b/src/client/src/helpers/server/platform.ts@@ -6,7 +6,10 @@ import {differenceInDays,differenceInYears,} from "date-fns";-import { getTraceMappingKeyFullPath } from "../server/trace";+import {+ getTraceMappingKeyFullPath,+ getTraceMappingKeyFullPaths,+} from "../server/trace";import { FilterWhereConditionType } from "@/types/platform";import { buildVersionWhereClause } from "@/lib/platform/agents/version-where";@@ -14,6 +17,22 @@ function escapeClickHouseString(value: string) {return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");}+/**+ * Exclude coding-agent / openlit-cli hook telemetry from LLM / request+ * dashboards. Those spans belong on the Coding Agents dashboard only.+ *+ * Matches the materializer barriers: coding_agent.* attrs, distro name,+ * and coding_agent.* span names.+ */+export const EXCLUDE_CODING_AGENT_WHERE = [+ "empty(SpanAttributes['coding_agent.client'])",+ "empty(ResourceAttributes['coding_agent.client'])",+ "empty(SpanAttributes['coding_agent.session.id'])",+ "empty(ResourceAttributes['coding_agent.session.id'])",+ "(empty(ResourceAttributes['telemetry.distro.name']) OR ResourceAttributes['telemetry.distro.name'] != 'openlit-cli')",+ "NOT startsWith(SpanName, 'coding_agent.')",+].join(" AND ");+export const validateMetricsRequestType = {// RequestREQUEST_PER_TIME: "REQUEST_PER_TIME",@@ -27,6 +46,11 @@ export const validateMetricsRequestType = {AVERAGE_REQUEST_COST: "AVERAGE_REQUEST_COST",COST_BY_APPLICATION: "COST_BY_APPLICATION",COST_BY_ENVIRONMENT: "COST_BY_ENVIRONMENT",+ COST_BY_PROVIDER: "COST_BY_PROVIDER",+ COST_BY_MODEL: "COST_BY_MODEL",+ COST_PER_TIME: "COST_PER_TIME",+ COST_SUMMARY: "COST_SUMMARY",+ OPENGROUND_COST_ANALYTICS: "OPENGROUND_COST_ANALYTICS",// ModelTOP_MODELS: "TOP_MODELS",MODEL_PER_TIME: "MODEL_PER_TIME",@@ -64,6 +88,7 @@ export const validateMetricsRequestType = {GENERATION_BY_APPLICATION: "GENERATION_BY_APPLICATION",// evaluationGET_TOTAL_EVALUATION_DETECTED: "GET_TOTAL_EVALUATION_DETECTED",+ GET_EVALUATION_ANALYTICS: "GET_EVALUATION_ANALYTICS",};export const validateMetricsRequest = (@@ -80,6 +105,11 @@ export const validateMetricsRequest = (case validateMetricsRequestType.AVERAGE_REQUEST_COST:case validateMetricsRequestType.COST_BY_APPLICATION:case validateMetricsRequestType.COST_BY_ENVIRONMENT:+ case validateMetricsRequestType.COST_BY_PROVIDER:+ case validateMetricsRequestType.COST_BY_MODEL:+ case validateMetricsRequestType.COST_PER_TIME:+ case validateMetricsRequestType.COST_SUMMARY:+ case validateMetricsRequestType.OPENGROUND_COST_ANALYTICS:// Modelscase validateMetricsRequestType.MODEL_PER_TIME:@@ -111,6 +141,7 @@ export const validateMetricsRequest = (// Evaluationcase validateMetricsRequestType.GET_TOTAL_EVALUATION_DETECTED:+ case validateMetricsRequestType.GET_EVALUATION_ANALYTICS:if (!params.timeLimit?.start || !params.timeLimit?.end) {return {success: false,@@ -199,21 +230,22 @@ export const getFilterWhereCondition = (if (filter.selectedConfig.providers?.length) {// Mirror of the provider DISTINCT fold in// lib/platform/request/index.ts: the dropdown shows- // values from gen_ai.system + db.system +- // coding_agent.client, so the WHERE check must- // match across the same three attributes — otherwise- // unticking e.g. "cursor" would have no effect on- // the rendered rows.+ // values from gen_ai.provider.name (+ gen_ai.system+ // fallback) + db.system + coding_agent.client, so the+ // WHERE check must match across the same attributes —+ // otherwise unticking e.g. "cursor" would have no+ // effect on the rendered rows.const providerList = filter.selectedConfig.providers.map((provider) => `'${provider}'`).join(", ");- whereArray.push(- `(SpanAttributes['${getTraceMappingKeyFullPath(- "system"- )}'] IN (${providerList}) OR SpanAttributes['${getTraceMappingKeyFullPath(- "provider"- )}'] IN (${providerList}) OR SpanAttributes['coding_agent.client'] IN (${providerList}))`+ const providerClauses = [+ ...(getTraceMappingKeyFullPaths("provider") as string[]),+ getTraceMappingKeyFullPath("system") as string,+ "coding_agent.client",+ ].map(+ (path) => `SpanAttributes['${path}'] IN (${providerList})`);+ whereArray.push(`(${providerClauses.join(" OR ")})`);}if (filter.selectedConfig.traceTypes?.length) {@@ -252,6 +284,21 @@ export const getFilterWhereCondition = ();}+ // UI ComboDropdown persists the selection as `services`+ // (see traces-filter.tsx / URL `services=`). `serviceNames`+ // is the agent-detail scope lock. Honor both so a Services+ // filter on Traces/Exceptions actually reaches ClickHouse —+ // especially important on Exceptions where failed LLM spans+ // often lack gen_ai.provider.name and ServiceName is the+ // only useful discriminator.+ if (filter.selectedConfig.services?.length) {+ whereArray.push(+ `ServiceName IN (${filter.selectedConfig.services+ .map((serviceName: string) => `'${escapeClickHouseString(serviceName)}'`)+ .join(", ")})`+ );+ }+if (filter.selectedConfig.spanNames?.length) {whereArray.push(`SpanName IN (${filter.selectedConfig.spanNames@@ -346,6 +393,9 @@ export const getFilterWhereCondition = ("type")}'] != 'vectordb'`);+ // LLM / request dashboards must not mix in coding-agent+ // hook telemetry (separate Coding Agents dashboard).+ whereArray.push(EXCLUDE_CODING_AGENT_WHERE);}}} catch {}diff --git a/src/client/src/lib/platform/agents/aggregate-graph.ts b/src/client/src/lib/platform/agents/aggregate-graph.tsindex b4731f24..6ed7a46c 100644--- a/src/client/src/lib/platform/agents/aggregate-graph.ts+++ b/src/client/src/lib/platform/agents/aggregate-graph.ts@@ -24,6 +24,7 @@ import { buildVersionWhereClause } from "./version-filter";import { agentsLogger } from "./logger";import type { VersionFilter } from "@/types/platform";import { escapeClickHouseString } from "@/lib/clickhouse-escape";+import { deploymentEnvironmentSqlPredicate } from "./index";const escape = escapeClickHouseString;@@ -90,11 +91,10 @@ export async function getAggregateGraph(params: AggregateGraphParams): Promise<AggregateGraph> {const maxTraces = Math.max(50, params.maxTraces || DEFAULT_MAX_TRACES);- const env = params.environment || "default";- const envPredicate =- env === "default"- ? `(ResourceAttributes['deployment.environment'] = 'default' OR ResourceAttributes['deployment.environment'] = '')`- : `ResourceAttributes['deployment.environment'] = '${escape(env)}'`;+ const envPredicate = deploymentEnvironmentSqlPredicate(+ params.environment,+ escape+ );const versionClause = buildVersionWhereClause(params.versionFilter);const recentFallback =diff --git a/src/client/src/lib/platform/agents/index.ts b/src/client/src/lib/platform/agents/index.tsindex 26c88223..8bfb4eef 100644--- a/src/client/src/lib/platform/agents/index.ts+++ b/src/client/src/lib/platform/agents/index.ts@@ -42,7 +42,9 @@ function rowToAgent(row: Record<string, unknown>): UnifiedAgent {return {agent_key: String(row.agent_key),service_name: String(row.service_name),- environment: String(row.environment || "default"),+ environment: normalizeDeploymentEnvironment(+ String(row.environment || "default")+ ),cluster_id: String(row.cluster_id || "default"),workload_key: String(row.workload_key || ""),source: String(row.source || "sdk") as AgentSource,@@ -396,6 +398,12 @@ async function loadAgents(params: ListAgentsParams): Promise<ListAgentsResult> {);}+ // Drop placeholder / local-dev env labels — they fold into `default` at+ // materialize time, and leftover rows would otherwise show as duplicates.+ where.push(+ `lower(s.environment) NOT IN ('local', 'default_environment')`+ );+const query = `${ROLLUP_CTES}SELECT ${SELECT_COLUMNS}@@ -531,6 +539,38 @@ export function invalidateAgent(agentKey: string, dbConfigId?: string) {invalidate(`agents:detail:${dbConfigId || "default"}:${agentKey}`);}+/**+ * Collapse placeholder / local-dev environment labels to `default` so SDK+ * sample apps don't create a noisy `local` dimension on agent identity.+ */+export function normalizeDeploymentEnvironment(+ environment?: string | null+): string {+ const env = (environment || "").trim();+ if (+ !env ||+ env.toLowerCase() === "local" ||+ env === "default_environment"+ ) {+ return "default";+ }+ return env;+}++/**+ * ClickHouse predicate that treats empty / local-dev labels as `default`.+ */+export function deploymentEnvironmentSqlPredicate(+ environment: string | null | undefined,+ escape: (value: string) => string+): string {+ const env = normalizeDeploymentEnvironment(environment);+ if (env === "default") {+ return `(ResourceAttributes['deployment.environment'] IN ('default', 'local', 'default_environment', ''))`;+ }+ return `ResourceAttributes['deployment.environment'] = '${escape(env)}'`;+}+/*** Compute the deterministic agent_key used as the URL slug + primary key.* Matches the formula used by the materializer.@@ -541,7 +581,7 @@ export function computeAgentKey(serviceName: string): string {const cluster = clusterId || "default";- const env = environment || "default";+ const env = normalizeDeploymentEnvironment(environment);return createHash("sha1").update(`${cluster}|${env}|${serviceName}`).digest("hex")diff --git a/src/client/src/lib/platform/agents/materialize.ts b/src/client/src/lib/platform/agents/materialize.tsindex e9d7213f..6b1fbb7d 100644--- a/src/client/src/lib/platform/agents/materialize.ts+++ b/src/client/src/lib/platform/agents/materialize.ts@@ -14,7 +14,11 @@ import {CONTROLLER_SERVICES_TABLE,} from "@/lib/platform/controller/table-details";import type { AgentSource, CodingAgentVendor } from "@/types/agents";-import { computeAgentKey, invalidateAgent } from "./index";+import {+ computeAgentKey,+ invalidateAgent,+ normalizeDeploymentEnvironment,+} from "./index";import { invalidatePrefix } from "./cache";import { agentsLogger } from "./logger";import {@@ -22,10 +26,18 @@ import {getLatestVersionsBatch,upsertVersion,} from "./snapshot";-import { AGENTS_SUMMARY_TABLE } from "./table-details";+import { AGENTS_SUMMARY_TABLE, AGENT_VERSIONS_TABLE } from "./table-details";import { escapeClickHouseString } from "@/lib/clickhouse-escape";import { mergeProviders } from "./provider-normalize";+/** SQL: fold empty / local-dev env labels into `default` (matches normalizeDeploymentEnvironment). */+const DEPLOYMENT_ENV_SQL = `multiIf(+ ResourceAttributes['deployment.environment'] = '', 'default',+ lower(ResourceAttributes['deployment.environment']) = 'local', 'default',+ ResourceAttributes['deployment.environment'] = 'default_environment', 'default',+ ResourceAttributes['deployment.environment']+)`;+/*** Maximum agents the materializer touches in a single tick.*@@ -146,7 +158,7 @@ async function discoverAgents(cli_services AS (SELECT DISTINCTServiceName AS service_name,- if(ResourceAttributes['deployment.environment'] != '', ResourceAttributes['deployment.environment'], 'default') AS environment,+ ${DEPLOYMENT_ENV_SQL} AS environment,if(ResourceAttributes['k8s.cluster.name'] != '', ResourceAttributes['k8s.cluster.name'], 'default') AS cluster_idFROM ${OTEL_TRACES_TABLE_NAME}WHERE Timestamp >= now() - INTERVAL ${SDK_DISCOVERY_LOOKBACK_MINUTES} MINUTE@@ -159,7 +171,7 @@ async function discoverAgents(sdk_seen AS (SELECTServiceName AS service_name,- if(ResourceAttributes['deployment.environment'] != '', ResourceAttributes['deployment.environment'], 'default') AS environment,+ ${DEPLOYMENT_ENV_SQL} AS environment,if(ResourceAttributes['k8s.cluster.name'] != '', ResourceAttributes['k8s.cluster.name'], 'default') AS cluster_id,argMax(ResourceAttributes['service.workload.key'], Timestamp) AS workload_key,argMax(ResourceAttributes['telemetry.sdk.version'], Timestamp) AS sdk_version,@@ -187,7 +199,7 @@ async function discoverAgents(-- Barrier #1 reinforced: even if individual spans don't have-- the distro marker, the service_name as a whole must not be-- one that ever emitted a CLI span in this window.- AND (ServiceName, if(ResourceAttributes['deployment.environment'] != '', ResourceAttributes['deployment.environment'], 'default'), if(ResourceAttributes['k8s.cluster.name'] != '', ResourceAttributes['k8s.cluster.name'], 'default')) NOT IN (SELECT service_name, environment, cluster_id FROM cli_services)+ AND (ServiceName, ${DEPLOYMENT_ENV_SQL}, if(ResourceAttributes['k8s.cluster.name'] != '', ResourceAttributes['k8s.cluster.name'], 'default')) NOT IN (SELECT service_name, environment, cluster_id FROM cli_services)GROUP BY service_name, environment, cluster_id)SELECT@@ -332,7 +344,7 @@ async function discoverAgents(for (const row of sdkRows) {if (!row.service_name) continue;- const env = row.environment || "default";+ const env = normalizeDeploymentEnvironment(row.environment);const cluster = row.cluster_id || "default";const key = computeAgentKey(cluster, env, row.service_name);const workloadKey = row.workload_key || "";@@ -361,7 +373,9 @@ async function discoverAgents(if (!row.service_name) continue;const cluster = row.cluster_id || "default";const attrs = parseResourceAttributes(row.resource_attributes);- const env = attrs["deployment.environment"] || "default";+ const env = normalizeDeploymentEnvironment(+ attrs["deployment.environment"]+ );const workloadKey = row.workload_key || "";const ctrlKey = computeAgentKey(cluster, env, row.service_name);@@ -817,7 +831,7 @@ async function fetchRequestCounts(const query = `SELECTServiceName AS service_name,- if(ResourceAttributes['deployment.environment'] != '', ResourceAttributes['deployment.environment'], 'default') AS environment,+ ${DEPLOYMENT_ENV_SQL} AS environment,if(ResourceAttributes['k8s.cluster.name'] != '', ResourceAttributes['k8s.cluster.name'], 'default') AS cluster_id,count() AS request_count_24hFROM ${OTEL_TRACES_TABLE_NAME}@@ -927,7 +941,7 @@ export async function materializeAgents(let discovered: DiscoveredAgent[];if (scope?.serviceName) {- const env = scope.environment || "default";+ const env = normalizeDeploymentEnvironment(scope.environment);const cluster = scope.clusterId || "default";const key = computeAgentKey(cluster, env, scope.serviceName);// Look for the agent in BOTH discovery pipelines, not just the SDK@@ -1204,5 +1218,75 @@ export async function materializeAgents(invalidatePrefix(`agents:list:${dbConfigId || "default"}:`);}+ await purgePlaceholderEnvironmentRows(dbConfigId);+return { processed, newVersions, errors };}++/**+ * Remove summary (+ version) rows stamped with placeholder env labels that+ * now fold into `default`. Without this, ReplacingMergeTree keeps both the+ * old `local` agent_key and the new `default` one as separate apps.+ */+async function purgePlaceholderEnvironmentRows(+ dbConfigId?: string+): Promise<void> {+ const keysRes = await dataCollector(+ {+ query: `+ SELECT agent_key+ FROM ${AGENTS_SUMMARY_TABLE} FINAL+ WHERE lower(environment) IN ('local', 'default_environment')+ `,+ },+ "query",+ dbConfigId+ );+ if (keysRes.err) {+ agentsLogger.error("materializer_purge_placeholder_env_lookup_failed", {+ err: keysRes.err,+ });+ return;+ }++ const keys = Array.isArray(keysRes.data)+ ? (keysRes.data as Array<{ agent_key: string }>)+ .map((r) => r.agent_key)+ .filter(Boolean)+ : [];++ const summaryRes = await dataCollector(+ {+ query: `+ ALTER TABLE ${AGENTS_SUMMARY_TABLE}+ DELETE WHERE lower(environment) IN ('local', 'default_environment')+ `,+ },+ "query",+ dbConfigId+ );+ if (summaryRes.err) {+ agentsLogger.error("materializer_purge_placeholder_env_summary_failed", {+ err: summaryRes.err,+ });+ }++ if (!keys.length) return;++ const escaped = keys.map((k) => `'${escape(k)}'`).join(", ");+ const versionsRes = await dataCollector(+ {+ query: `+ ALTER TABLE ${AGENT_VERSIONS_TABLE}+ DELETE WHERE agent_key IN (${escaped})+ `,+ },+ "query",+ dbConfigId+ );+ if (versionsRes.err) {+ agentsLogger.error("materializer_purge_placeholder_env_versions_failed", {+ err: versionsRes.err,+ });+ }+}diff --git a/src/client/src/lib/platform/agents/snapshot.ts b/src/client/src/lib/platform/agents/snapshot.tsindex b6f9b86e..87194bb5 100644--- a/src/client/src/lib/platform/agents/snapshot.ts+++ b/src/client/src/lib/platform/agents/snapshot.ts@@ -41,7 +41,7 @@ import type {AgentTool,AgentVersion,} from "@/types/agents";-import { computeAgentKey } from "./index";+import { computeAgentKey, deploymentEnvironmentSqlPredicate, normalizeDeploymentEnvironment } from "./index";import { AGENT_VERSIONS_TABLE } from "./table-details";import { escapeClickHouseString } from "@/lib/clickhouse-escape";import { mergeProviders } from "./provider-normalize";@@ -241,7 +241,7 @@ export async function deriveSnapshot(1,Math.min(params.lookbackMinutes ?? DEFAULT_LOOKBACK_MINUTES, 60 * 24));- const env = params.environment || "default";+ const env = normalizeDeploymentEnvironment(params.environment);const cluster = params.clusterId || "default";const query = `@@ -659,11 +659,10 @@ export async function getVersionTimeline((a, b) => a.version_number - b.version_number);- const env = agentRow.environment || "default";- const envPredicate =- env === "default"- ? `(ResourceAttributes['deployment.environment'] = 'default' OR ResourceAttributes['deployment.environment'] = '')`- : `ResourceAttributes['deployment.environment'] = '${escape(env)}'`;+ const envPredicate = deploymentEnvironmentSqlPredicate(+ agentRow.environment,+ escape+ );// Count every span tied to the agent's ServiceName, not just LLM// chat-completions. Agent-framework workloads (CrewAI, LangGraph,diff --git a/src/client/src/lib/platform/agents/version-filter.ts b/src/client/src/lib/platform/agents/version-filter.tsindex fca6b470..8bc32800 100644--- a/src/client/src/lib/platform/agents/version-filter.ts+++ b/src/client/src/lib/platform/agents/version-filter.ts@@ -24,7 +24,7 @@ import type { VersionFilter } from "@/types/platform";import { swr, POLICY_VERSIONS } from "./cache";import { agentsLogger } from "./logger";import { getVersion } from "./snapshot";-import { getAgent } from "./index";+import { getAgent, deploymentEnvironmentSqlPredicate } from "./index";// Re-export the pure SQL builder so existing callers that import it from// `./version-filter` keep working. The actual implementation lives in// `version-where.ts` (zero side-effect imports) so leaf consumers don't@@ -72,11 +72,10 @@ async function probeAttributeStamping(): Promise<boolean> {const first = toClickHouseDateTime(params.firstSeen);const last = toClickHouseDateTime(params.lastSeen);- const env = params.environment || "default";- const envPredicate =- env === "default"- ? `(ResourceAttributes['deployment.environment'] = 'default' OR ResourceAttributes['deployment.environment'] = '')`- : `ResourceAttributes['deployment.environment'] = '${escape(env)}'`;+ const envPredicate = deploymentEnvironmentSqlPredicate(+ params.environment,+ escape+ );const query = `SELECT 1 AS stampedFROM ${OTEL_TRACES_TABLE_NAME}diff --git a/src/client/src/lib/platform/chat/cost.ts b/src/client/src/lib/platform/chat/cost.tsnew file mode 100644index 00000000..2604731a--- /dev/null+++ b/src/client/src/lib/platform/chat/cost.ts@@ -0,0 +1,62 @@+import { dataCollector, MetricParams } from "@/lib/platform/common";+import { getFilterPreviousParams } from "@/helpers/server/platform";+import {+ OPENLIT_CHAT_CONVERSATION_TABLE,+ OPENLIT_OTTER_RUNS_TABLE,+ OPENLIT_TRACE_ANALYSIS_TABLE,+} from "@/lib/platform/chat/table-details";+import Sanitizer from "@/utils/sanitizer";++function timeWhere(params: MetricParams) {+ const start = Sanitizer.sanitizeValue(String(params.timeLimit.start));+ const end = Sanitizer.sanitizeValue(String(params.timeLimit.end));+ return `+ created_at >= parseDateTimeBestEffort('${start}')+ AND created_at <= parseDateTimeBestEffort('${end}')+ `;+}++async function sumPeriod(params: MetricParams): Promise<number> {+ const where = timeWhere(params);+ // Cost columns are already Float64 — do not wrap with toFloat64OrZero+ // (ClickHouse OrZero/OrNull converters only accept String).+ const query = `+ SELECT+ CAST((+ (SELECT sum(ifNull(total_cost, 0)) FROM ${OPENLIT_CHAT_CONVERSATION_TABLE}+ WHERE conversation_type = 'chat' AND ${where})+ ++ (SELECT sum(ifNull(cost, 0)) FROM ${OPENLIT_TRACE_ANALYSIS_TABLE}+ WHERE analysis_type IN ('trace_analysis', 'span_analysis') AND ${where})+ ++ (SELECT sum(ifNull(cost, 0)) FROM ${OPENLIT_OTTER_RUNS_TABLE}+ WHERE run_type IN ('prompt_improvement') AND ${where})+ ) AS FLOAT) AS total_cost+ `;++ const { data, err } = await dataCollector({ query });+ if (err) return 0;+ const row = Array.isArray(data) ? data[0] : undefined;+ return Number(row?.total_cost) || 0;+}++/**+ * Otter / platform AI spend: chat conversations + trace/span analysis ++ * prompt improvement runs. These live in product tables, not otel_traces.+ */+export async function getOtterTotalCost(params: MetricParams) {+ const previous = getFilterPreviousParams(params);+ const [total, previousTotal] = await Promise.all([+ sumPeriod(params),+ sumPeriod(previous),+ ]);++ return {+ data: [+ {+ total_cost: total,+ previous_total_cost: previousTotal,+ },+ ],+ };+}diff --git a/src/client/src/lib/platform/chat/schema-context.ts b/src/client/src/lib/platform/chat/schema-context.tsindex ff046dbe..29070ddc 100644--- a/src/client/src/lib/platform/chat/schema-context.ts+++ b/src/client/src/lib/platform/chat/schema-context.ts@@ -66,7 +66,7 @@ URL mappings per entity type:- **context**: \`/context/{id}\`- **prompt**: \`/prompt-hub/{id}\`- **vault**: \`/vault\` (no ID in URL)-- **model**: \`/manage-models\` (no ID in URL)+- **model**: \`/costs?tab=models\` (no ID in URL)- **evaluation**: \`/evaluations\` (no ID in URL)Rules:diff --git a/src/client/src/lib/platform/coding-agents/cost.ts b/src/client/src/lib/platform/coding-agents/cost.tsnew file mode 100644index 00000000..dfea7cda--- /dev/null+++ b/src/client/src/lib/platform/coding-agents/cost.ts@@ -0,0 +1,78 @@+import { dataCollector, MetricParams, OTEL_TRACES_TABLE_NAME } from "@/lib/platform/common";+import { getFilterPreviousParams } from "@/helpers/server/platform";+import {+ CODING_AGENT_ATTR,+ CODING_AGENT_SPAN_SESSION,+ GEN_AI_ATTR,+} from "@/lib/platform/coding-agents/table-details";+import Sanitizer from "@/utils/sanitizer";++/**+ * Chat-thread id used for coding-agent rollups. Prefer parent when the+ * span is a subagent; otherwise use the session id. Mirrors CHAT_ID_EXPR+ * in coding-agents/queries.ts (simplified for cost aggregation).+ */+const CHAT_ID_EXPR = `+ coalesce(+ nullIf(ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'], ''),+ nullIf(SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'], ''),+ nullIf(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ nullIf(ResourceAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ nullIf(SpanAttributes['session.id'], '')+ )+`;++const CODING_AGENT_WHERE = `+ (+ notEmpty(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'])+ OR notEmpty(ResourceAttributes['${CODING_AGENT_ATTR.sessionId}'])+ OR startsWith(SpanName, 'coding_agent.')+ )+`;++function periodCostQuery(periodParams: MetricParams, joinKey: string) {+ const start = Sanitizer.sanitizeValue(String(periodParams.timeLimit.start));+ const end = Sanitizer.sanitizeValue(String(periodParams.timeLimit.end));+ const safeJoinKey = Sanitizer.sanitizeValue(joinKey);+ return `+ SELECT+ CAST(sum(ifNull(cost_usd, 0)) AS FLOAT) AS total_cost,+ '${safeJoinKey}' AS start_date+ FROM (+ SELECT+ ${CHAT_ID_EXPR} AS chat_id,+ coalesce(+ nullIf(toFloat64OrZero(anyIf(+ SpanAttributes['${CODING_AGENT_ATTR.sessionCostUsd}'],+ SpanName = '${CODING_AGENT_SPAN_SESSION}'+ )), 0),+ sumOrNull(if(+ SpanName != '${CODING_AGENT_SPAN_SESSION}',+ toFloat64OrZero(SpanAttributes['${GEN_AI_ATTR.usageCost}']),+ 0+ ))+ ) AS cost_usd+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE Timestamp >= parseDateTimeBestEffort('${start}')+ AND Timestamp <= parseDateTimeBestEffort('${end}')+ AND ${CODING_AGENT_WHERE}+ AND notEmpty(${CHAT_ID_EXPR})+ GROUP BY chat_id+ )+ `;+}++export async function getCodingAgentsTotalCost(params: MetricParams) {+ const previous = getFilterPreviousParams(params);+ const joinKey = String(params.timeLimit.start);+ const query = `+ SELECT+ CAST(current_data.total_cost AS FLOAT) AS total_cost,+ CAST(previous_day.total_cost AS FLOAT) AS previous_total_cost+ FROM (${periodCostQuery(params, joinKey)}) AS current_data+ JOIN (${periodCostQuery(previous, joinKey)}) AS previous_day+ ON current_data.start_date = previous_day.start_date+ `;++ return dataCollector({ query });+}diff --git a/src/client/src/lib/platform/coding-agents/queries.ts b/src/client/src/lib/platform/coding-agents/queries.tsindex 0e9bc971..38063605 100644--- a/src/client/src/lib/platform/coding-agents/queries.ts+++ b/src/client/src/lib/platform/coding-agents/queries.ts@@ -162,12 +162,12 @@ export interface CodingAgentSessionRow {acceptance_pct: number;commit_count: number;pr_count: number;- // True when the chat is a subagent of another chat — i.e. any span- // in the rollup has `coding_agent.agent.parent_id` set or the CLI- // stamped `coding_agent.session.is_subagent`. listSessions hides- // these by default; they fold under the parent chat via the- // CHAT_ID_EXPR coalesce.+ // True when this rollup is a subagent-only chat (no owner spans).+ // listSessions hides these by default; linked subagents fold under+ // the parent via CHAT_ID_EXPR and the parent row stays visible.is_subagent: 0 | 1;+ /** Count of `coding_agent.subagent` spans folded under this chat. */+ subagent_event_count: number;}/**@@ -278,10 +278,44 @@ const VENDOR_EXPR = `// launched inside a Cursor terminal) fold into TWO rows so each// agent's chat shows up under its own vendor regardless of which// editor hosted it.+// CHAT_ID_EXPR folds linked subagent spans under their parent chat.+// Only a *foreign* parent_id counts — Cursor often echoes+// parent_conversation_id == conversation.id on the parent chat itself+// ("self-parent"); treating that as a fold key is harmless for the+// id value but we skip it so chat_id matches conversation/session+// identity the same way is_subagent / the CLI foreignParentID do.const CHAT_ID_EXPR = `coalesce(- nullIf(ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'], ''),- nullIf(SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'], ''),+ nullIf(+ if(+ notEmpty(ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'])+ AND ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'] != coalesce(+ nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ nullIf(ResourceAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ ''+ ),+ ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'],+ ''+ ),+ ''+ ),+ nullIf(+ if(+ notEmpty(SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'])+ AND SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'] != coalesce(+ nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ nullIf(ResourceAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ ''+ ),+ SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'],+ ''+ ),+ ''+ ),nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),nullIf(SpanAttributes['gen_ai.conversation.id'], ''),nullIf(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),@@ -584,24 +618,55 @@ const SESSION_BASE_COLUMNS = `toInt64OrZero(any(SpanAttributes['${CODING_AGENT_ATTR.sessionPrCount}'])),toInt64(countIf(SpanName = '${CODING_AGENT_SPAN_GIT_PR}'))) AS pr_count,- -- Is this chat a subagent of another chat? True when ANY span in- -- the group has a non-empty parent_id (resource OR span attr) OR- -- the CLI explicitly stamped coding_agent.session.is_subagent- -- on the session-root resource. listSessions hides these by- -- default - they fold under the parent chat via CHAT_ID_EXPR.- -- The check is intentionally lenient so a partial linkage (e.g.- -- parent_id only on a few spans, missing on the root) still- -- classifies the chat as a subagent.+ -- Is this chat a subagent-only row (hide from Sessions by default)?+ --+ -- Historical bug: we used max(notEmpty(parent_id)), which also fired+ -- when Cursor echoed parent_conversation_id == conversation_id on+ -- parent-chat hooks ("self-parent"). Folded true subagents then+ -- marked the WHOLE parent group as is_subagent=1 and the Sessions+ -- tab hid the user's main chat while Overview still counted it.+ --+ -- Contract now:+ -- • Explicit coding_agent.session.is_subagent='true' ⇒ subagent+ -- • Foreign parent_id (differs from this span's own conversation+ -- / session id) ⇒ subagent span+ -- • Self-parent (parent_id == own conversation/session id) ⇒ ignore+ -- • Group is_subagent=1 only when there are ZERO owner spans+ -- (every span is flagged or has a foreign parent). Parent chats+ -- that absorbed linked subagents still have owner spans and stay+ -- visible; orphan subagent-only groups stay hidden.if(- max(- notEmpty(SpanAttributes['${CODING_AGENT_ATTR.agentParentId}']) OR- notEmpty(ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}']) OR- ResourceAttributes['coding_agent.session.is_subagent'] = 'true' OR- SpanAttributes['coding_agent.session.is_subagent'] = 'true'- ),+ countIf(+ NOT (+ ResourceAttributes['coding_agent.session.is_subagent'] = 'true'+ OR SpanAttributes['coding_agent.session.is_subagent'] = 'true'+ OR (+ notEmpty(ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'])+ AND ResourceAttributes['${CODING_AGENT_ATTR.agentParentId}'] != coalesce(+ nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ nullIf(ResourceAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ ''+ )+ )+ OR (+ notEmpty(SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'])+ AND SpanAttributes['${CODING_AGENT_ATTR.agentParentId}'] != coalesce(+ nullIf(ResourceAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['gen_ai.conversation.id'], ''),+ nullIf(SpanAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ nullIf(ResourceAttributes['${CODING_AGENT_ATTR.sessionId}'], ''),+ ''+ )+ )+ )+ ) = 0,1,0- ) AS is_subagent+ ) AS is_subagent,+ -- Linked subagent activity folded under this chat (for UI badge).+ toInt64(countIf(SpanName = 'coding_agent.subagent')) AS subagent_event_count`;/**diff --git a/src/client/src/lib/platform/coding-agents/query-builders.ts b/src/client/src/lib/platform/coding-agents/query-builders.tsindex 7d1c4634..10d56892 100644--- a/src/client/src/lib/platform/coding-agents/query-builders.ts+++ b/src/client/src/lib/platform/coding-agents/query-builders.ts@@ -73,10 +73,11 @@ export function buildSessionsHaving(opts: SessionsHavingOptions): string {if (opts.classification) {havingClauses.push(`classification = '${escape(opts.classification)}'`);}- // Hide subagent rows by default: they fold under the parent chat- // via CHAT_ID_EXPR; listing them at the top level produces- // duplicate-looking rows for one user-perceived chat. Callers- // debugging linkage gaps can opt back in.+ // Hide subagent-only rows by default. Linked subagents fold under+ // the parent chat via CHAT_ID_EXPR; the parent row stays visible+ // (is_subagent ignores Cursor's self-parent_id echo). Orphan+ // subagent-only groups are what we hide. Callers debugging+ // linkage gaps can opt back in with includeSubagents.if (!opts.includeSubagents) {havingClauses.push(`is_subagent = 0`);}diff --git a/src/client/src/lib/platform/cost/summary.ts b/src/client/src/lib/platform/cost/summary.tsnew file mode 100644index 00000000..402e6f32--- /dev/null+++ b/src/client/src/lib/platform/cost/summary.ts@@ -0,0 +1,77 @@+import { MetricParams } from "@/lib/platform/common";+import { getTotalCost } from "@/lib/platform/llm/cost";+import { getEvaluationAnalytics } from "@/lib/platform/evaluation/analytics";+import { getOpengroundTotalCost } from "@/lib/platform/openground/cost-analytics";+import { getCodingAgentsTotalCost } from "@/lib/platform/coding-agents/cost";+import { getOtterTotalCost } from "@/lib/platform/chat/cost";++function firstRow(data: unknown): Record<string, number> {+ if (Array.isArray(data) && data[0] && typeof data[0] === "object") {+ return data[0] as Record<string, number>;+ }+ return {};+}++/**+ * Company-wide AI cost rollup. Buckets are mutually exclusive:+ * - llm: instrumented app LLM traffic (otel_traces, coding agents excluded)+ * - coding_agents: coding-agent session rollups (otel_traces)+ * - otter: Otter chat + analysis + prompt improvement (product tables)+ * - evaluations: judge LLM spend (openlit_evaluation)+ * - openground: comparison run provider costs+ */+export async function getCostSummary(params: MetricParams) {+ const [llmRes, codingRes, otterRes, evaluationsRes, opengroundRes] =+ await Promise.all([+ getTotalCost(params),+ getCodingAgentsTotalCost(params),+ getOtterTotalCost(params),+ getEvaluationAnalytics(params),+ getOpengroundTotalCost(params),+ ]);++ const llmRow = firstRow(llmRes.data);+ const codingRow = firstRow(codingRes.data);+ const otterRow = firstRow(otterRes.data);+ const evaluationsRow = firstRow(evaluationsRes.data);+ const opengroundRow = firstRow(opengroundRes.data);++ const llm = Number(llmRow.total_usage_cost) || 0;+ const previousLlm = Number(llmRow.previous_total_usage_cost) || 0;+ const codingAgents = Number(codingRow.total_cost) || 0;+ const previousCodingAgents = Number(codingRow.previous_total_cost) || 0;+ const otter = Number(otterRow.total_cost) || 0;+ const previousOtter = Number(otterRow.previous_total_cost) || 0;+ const evaluations = Number(evaluationsRow.total_cost) || 0;+ const previousEvaluations = Number(evaluationsRow.previous_total_cost) || 0;+ const openground = Number(opengroundRow.total_cost) || 0;+ const previousOpenground = Number(opengroundRow.previous_total_cost) || 0;++ // Never surface partial bucket errors on the top-level payload.+ // useFetchWrapper treats any `err` as a total failure and clears data,+ // which zeroed every Costs summary StatCard when one bucket query failed.+ return {+ data: [+ {+ total_platform_cost:+ llm + codingAgents + otter + evaluations + openground,+ previous_total_platform_cost:+ previousLlm ++ previousCodingAgents ++ previousOtter ++ previousEvaluations ++ previousOpenground,+ llm_cost: llm,+ previous_llm_cost: previousLlm,+ coding_agents_cost: codingAgents,+ previous_coding_agents_cost: previousCodingAgents,+ otter_cost: otter,+ previous_otter_cost: previousOtter,+ evaluations_cost: evaluations,+ previous_evaluations_cost: previousEvaluations,+ openground_cost: openground,+ previous_openground_cost: previousOpenground,+ },+ ],+ };+}diff --git a/src/client/src/lib/platform/evaluation/analytics.ts b/src/client/src/lib/platform/evaluation/analytics.tsnew file mode 100644index 00000000..a3fb35b9--- /dev/null+++ b/src/client/src/lib/platform/evaluation/analytics.ts@@ -0,0 +1,488 @@+import { dataCollector, MetricParams } from "../common";+import { OPENLIT_EVALUATION_TABLE_NAME } from "./table-details";+import {+ dateTruncGroupingLogic,+ getFilterPreviousParams,+} from "@/helpers/server/platform";+import { EVALUATION_SOURCE } from "@/constants/evaluation-sources";+import asaw from "@/utils/asaw";+import { getEvaluationConfig } from "./config";+import { EVALUATION_TYPES } from "@/constants/evaluation-types";+import { getEvaluationStoredNameVariants } from "@/helpers/client/evaluation-type";+import type {+ EvaluationAnalyticsByTypeRow,+ EvaluationAnalyticsResponse,+ EvaluationAnalyticsSummary,+ EvaluationAnalyticsTimeseriesPoint,+} from "@/types/evaluation";++export type {+ EvaluationAnalyticsByTypeRow,+ EvaluationAnalyticsResponse,+ EvaluationAnalyticsSummary,+ EvaluationAnalyticsTimeseriesPoint,+};++const EXCLUDED_SOURCES = `('${EVALUATION_SOURCE.MANUAL_FEEDBACK}', '${EVALUATION_SOURCE.AUTO_SKIPPED}')`;++function timeWhere(parameters: MetricParams) {+ return `+ created_at >= parseDateTimeBestEffort('${parameters.timeLimit.start}')+ AND created_at <= parseDateTimeBestEffort('${parameters.timeLimit.end}')+ AND meta['source'] NOT IN ${EXCLUDED_SOURCES}+ `;+}++function escapeSqlString(value: string): string {+ return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");+}++function evaluationNameWhere(variants: string[]): string {+ if (variants.length === 0) return "1 = 0";+ const list = variants+ .map((v) => `'${escapeSqlString(v)}'`)+ .join(", ");+ return `evaluationData.evaluation IN (${list})`;+}++/**+ * Name variants for configured evaluators only. Hub KPIs / charts must not+ * include random ClickHouse labels (TypeA, ad-hoc judge names, etc.).+ */+async function getConfiguredEvaluationNameVariants(): Promise<string[]> {+ const [err, config] = await asaw(getEvaluationConfig(undefined, true, false));+ const configured = (!err && Array.isArray(config?.evaluationTypes)+ ? config.evaluationTypes+ : []) as Array<{ id: string; label?: string }>;++ const types =+ configured.length > 0+ ? configured+ : EVALUATION_TYPES.map((t) => ({ id: t.id, label: t.label }));++ const variants = new Set<string>();+ for (const type of types) {+ if (!type?.id) continue;+ const label =+ type.label ||+ EVALUATION_TYPES.find((t) => t.id === type.id)?.label ||+ type.id;+ for (const name of getEvaluationStoredNameVariants(type.id, label)) {+ variants.add(name);+ }+ }+ return Array.from(variants);+}++async function getTypeCounts(): Promise<{ evaluations: number; active: number }> {+ const [err, config] = await asaw(getEvaluationConfig(undefined, true, false));+ if (err || !config?.id) {+ return { evaluations: EVALUATION_TYPES.length, active: 0 };+ }++ const types = config.evaluationTypes || [];+ const evaluations = types.length || EVALUATION_TYPES.length;+ const active = types.filter((t: { enabled?: boolean }) => t.enabled).length;+ return { evaluations, active };+}++export async function getEvaluationAnalyticsSummary(+ params: MetricParams+): Promise<{+ err?: unknown;+ data?: EvaluationAnalyticsSummary;+}> {+ const current = params;+ const previous = getFilterPreviousParams(current);+ const variants = await getConfiguredEvaluationNameVariants();+ const nameWhere = evaluationNameWhere(variants);++ const periodQuery = (parameters: MetricParams) => `+ SELECT+ (+ SELECT countDistinct(span_id)+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ ) AS tracesEvaluated,+ (+ SELECT countDistinct(id)+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ ) AS executions,+ (+ SELECT countDistinct(if(meta['source'] = '${EVALUATION_SOURCE.AUTO}', id, NULL))+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ ) AS autoExecutions,+ (+ SELECT sum(runCost)+ FROM (+ SELECT+ id,+ any(toFloat64OrZero(meta['cost'])) AS runCost+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ GROUP BY id+ )+ ) AS totalCost,+ (+ SELECT+ if(+ count() = 0,+ 0,+ (countIf(evaluationData.verdict != 'yes') * 100.0) / count()+ )+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ ) AS avgPassRate,+ (+ SELECT countIf(evaluationData.verdict = 'yes')+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ ) AS failedScores+ `;++ const [{ data: currentData, err: currentErr }, { data: previousData, err: previousErr }] =+ await Promise.all([+ dataCollector({ query: periodQuery(current) }),+ dataCollector({ query: periodQuery(previous) }),+ ]);++ if (currentErr) return { err: currentErr };+ if (previousErr) return { err: previousErr };++ const row = (currentData as any[])?.[0] || {};+ const prev = (previousData as any[])?.[0] || {};+ const typeCounts = await getTypeCounts();++ return {+ data: {+ tracesEvaluated: Number(row.tracesEvaluated) || 0,+ executions: Number(row.executions) || 0,+ autoExecutions: Number(row.autoExecutions) || 0,+ totalCost: Number(row.totalCost) || 0,+ avgPassRate: Number(row.avgPassRate) || 0,+ failedScores: Number(row.failedScores) || 0,+ previous_tracesEvaluated: Number(prev.tracesEvaluated) || 0,+ previous_executions: Number(prev.executions) || 0,+ previous_autoExecutions: Number(prev.autoExecutions) || 0,+ previous_totalCost: Number(prev.totalCost) || 0,+ previous_avgPassRate: Number(prev.avgPassRate) || 0,+ previous_failedScores: Number(prev.failedScores) || 0,+ evaluations: typeCounts.evaluations,+ active: typeCounts.active,+ },+ };+}++export async function getEvaluationAnalyticsTimeseries(+ params: MetricParams+): Promise<{ err?: unknown; data?: EvaluationAnalyticsTimeseriesPoint[] }> {+ const { start, end } = params.timeLimit;+ const dateTrunc = dateTruncGroupingLogic(end as Date, start as Date);+ const variants = await getConfiguredEvaluationNameVariants();+ const nameWhere = evaluationNameWhere(variants);++ const query = `+ SELECT+ formatDateTime(DATE_TRUNC('${dateTrunc}', created_at), '%Y/%m/%d %H:%i') AS timestamp,+ countDistinct(id) AS executions,+ if(+ count() = 0,+ 0,+ (countIf(evaluationData.verdict != 'yes') * 100.0) / count()+ ) AS passRate+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(params)}+ AND ${nameWhere}+ GROUP BY timestamp+ ORDER BY timestamp ASC+ `;++ const { data, err } = await dataCollector({ query });+ if (err) return { err };++ return {+ data: ((data as any[]) || []).map((row) => ({+ timestamp: String(row.timestamp || ""),+ executions: Number(row.executions) || 0,+ passRate: Number(row.passRate) || 0,+ })),+ };+}++export async function getEvaluationAnalyticsByType(+ params: MetricParams+): Promise<{ err?: unknown; data?: EvaluationAnalyticsByTypeRow[] }> {+ const current = params;+ const previous = getFilterPreviousParams(current);++ const typeQuery = (parameters: MetricParams) => `+ SELECT+ evaluationData.evaluation AS evaluation,+ countDistinct(id) AS executions,+ if(+ count() = 0,+ 0,+ (countIf(evaluationData.verdict != 'yes') * 100.0) / count()+ ) AS passRate,+ countIf(evaluationData.verdict = 'yes') AS failedScores+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ GROUP BY evaluation+ `;++ const [{ data: currentData, err: currentErr }, { data: previousData, err: previousErr }] =+ await Promise.all([+ dataCollector({ query: typeQuery(current) }),+ dataCollector({ query: typeQuery(previous) }),+ ]);++ if (currentErr) return { err: currentErr };+ if (previousErr) return { err: previousErr };++ const previousMap = new Map(+ ((previousData as any[]) || []).map((row) => [+ String(row.evaluation || ""),+ Number(row.passRate) || 0,+ ])+ );++ return {+ data: ((currentData as any[]) || [])+ .map((row) => {+ const evaluation = String(row.evaluation || "");+ return {+ evaluation,+ executions: Number(row.executions) || 0,+ passRate: Number(row.passRate) || 0,+ previousPassRate: previousMap.get(evaluation) || 0,+ failedScores: Number(row.failedScores) || 0,+ };+ })+ .sort((a, b) => b.executions - a.executions),+ };+}++export async function getEvaluationAnalytics(+ params: MetricParams+): Promise<EvaluationAnalyticsResponse> {+ const [configErr, config] = await asaw(+ getEvaluationConfig(undefined, true, false)+ );++ if (configErr || !config?.id) {+ return { configured: false };+ }++ const [summaryRes, timeseriesRes, byTypeRes] = await Promise.all([+ getEvaluationAnalyticsSummary(params),+ getEvaluationAnalyticsTimeseries(params),+ getEvaluationAnalyticsByType(params),+ ]);++ return {+ configured: true,+ data: [+ {+ evaluations: summaryRes.data?.evaluations ?? 0,+ previous_evaluations: summaryRes.data?.evaluations ?? 0,+ active: summaryRes.data?.active ?? 0,+ previous_active: summaryRes.data?.active ?? 0,+ traces_evaluated: summaryRes.data?.tracesEvaluated ?? 0,+ previous_traces_evaluated:+ summaryRes.data?.previous_tracesEvaluated ?? 0,+ executions: summaryRes.data?.executions ?? 0,+ previous_executions: summaryRes.data?.previous_executions ?? 0,+ auto_executions: summaryRes.data?.autoExecutions ?? 0,+ previous_auto_executions:+ summaryRes.data?.previous_autoExecutions ?? 0,+ total_cost: summaryRes.data?.totalCost ?? 0,+ previous_total_cost: summaryRes.data?.previous_totalCost ?? 0,+ avg_pass_rate: summaryRes.data?.avgPassRate ?? 0,+ previous_avg_pass_rate: summaryRes.data?.previous_avgPassRate ?? 0,+ failed_scores: summaryRes.data?.failedScores ?? 0,+ previous_failed_scores: summaryRes.data?.previous_failedScores ?? 0,+ },+ ],+ summary: summaryRes.data,+ timeseries: timeseriesRes.data || [],+ byType: byTypeRes.data || [],+ };+}++export async function getEvaluationEvaluatorAnalytics(+ params: MetricParams,+ evaluatorId: string+): Promise<import("@/types/evaluation").EvaluationEvaluatorAnalyticsResponse> {+ const [configErr, config] = await asaw(+ getEvaluationConfig(undefined, true, false)+ );++ if (configErr || !config?.id) {+ return { configured: false, found: false };+ }++ const types = (config.evaluationTypes || []) as Array<{+ id: string;+ label?: string;+ enabled?: boolean;+ description?: string;+ isCustom?: boolean;+ }>;+ const typeConfig = types.find((t) => t.id === evaluatorId);+ const builtIn = EVALUATION_TYPES.find((t) => t.id === evaluatorId);++ if (!typeConfig && !builtIn) {+ return { configured: true, found: false };+ }++ const label = typeConfig?.label || builtIn?.label || evaluatorId;+ const variants = getEvaluationStoredNameVariants(evaluatorId, label);+ const nameWhere = evaluationNameWhere(variants);+ const current = params;+ const previous = getFilterPreviousParams(current);+ const { start, end } = params.timeLimit;+ const dateTrunc = dateTruncGroupingLogic(end as Date, start as Date);++ const summaryQuery = (parameters: MetricParams) => `+ SELECT+ countDistinct(id) AS executions,+ sum(toFloat64OrZero(meta['cost'])) AS totalCost,+ if(+ count() = 0,+ 0,+ (countIf(evaluationData.verdict != 'yes') * 100.0) / count()+ ) AS avgPassRate,+ countIf(evaluationData.verdict = 'yes') AS failedScores,+ countDistinct(span_id) AS tracesEvaluated+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(parameters)}+ AND ${nameWhere}+ `;++ const timeseriesQuery = `+ SELECT+ formatDateTime(DATE_TRUNC('${dateTrunc}', created_at), '%Y/%m/%d %H:%i') AS timestamp,+ countDistinct(id) AS executions,+ if(+ count() = 0,+ 0,+ (countIf(evaluationData.verdict != 'yes') * 100.0) / count()+ ) AS passRate+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(params)}+ AND ${nameWhere}+ GROUP BY timestamp+ ORDER BY timestamp ASC+ `;++ const recentQuery = `+ SELECT+ id,+ span_id AS spanId,+ created_at AS createdAt,+ evaluationData.verdict AS verdict,+ toFloat64OrZero(toString(scores[evaluationData.evaluation])) AS score,+ evaluationData.classification AS classification,+ evaluationData.explanation AS explanation,+ meta['source'] AS source,+ toFloat64OrZero(meta['cost']) AS cost+ FROM ${OPENLIT_EVALUATION_TABLE_NAME}+ ARRAY JOIN evaluationData+ WHERE ${timeWhere(params)}+ AND ${nameWhere}+ ORDER BY created_at DESC+ LIMIT 25+ `;++ const [+ { data: currentData, err: currentErr },+ { data: previousData, err: previousErr },+ { data: timeseriesData, err: timeseriesErr },+ { data: recentData, err: recentErr },+ ] = await Promise.all([+ dataCollector({ query: summaryQuery(current) }),+ dataCollector({ query: summaryQuery(previous) }),+ dataCollector({ query: timeseriesQuery }),+ dataCollector({ query: recentQuery }),+ ]);++ if (currentErr || previousErr || timeseriesErr || recentErr) {+ return {+ configured: true,+ found: true,+ evaluator: {+ id: evaluatorId,+ label,+ enabled: typeConfig?.enabled ?? builtIn?.enabledByDefault ?? false,+ description: typeConfig?.description || builtIn?.description,+ isCustom: typeConfig?.isCustom,+ },+ data: [],+ timeseries: [],+ recentResults: [],+ };+ }++ const row = (currentData as any[])?.[0] || {};+ const prev = (previousData as any[])?.[0] || {};++ return {+ configured: true,+ found: true,+ evaluator: {+ id: evaluatorId,+ label,+ enabled: typeConfig?.enabled ?? builtIn?.enabledByDefault ?? false,+ description: typeConfig?.description || builtIn?.description,+ isCustom: typeConfig?.isCustom,+ },+ data: [+ {+ executions: Number(row.executions) || 0,+ previous_executions: Number(prev.executions) || 0,+ avg_pass_rate: Number(row.avgPassRate) || 0,+ previous_avg_pass_rate: Number(prev.avgPassRate) || 0,+ failed_scores: Number(row.failedScores) || 0,+ previous_failed_scores: Number(prev.failedScores) || 0,+ total_cost: Number(row.totalCost) || 0,+ previous_total_cost: Number(prev.totalCost) || 0,+ traces_evaluated: Number(row.tracesEvaluated) || 0,+ previous_traces_evaluated: Number(prev.tracesEvaluated) || 0,+ },+ ],+ timeseries: ((timeseriesData as any[]) || []).map((point) => ({+ timestamp: String(point.timestamp || ""),+ executions: Number(point.executions) || 0,+ passRate: Number(point.passRate) || 0,+ })),+ recentResults: ((recentData as any[]) || []).map((r) => ({+ id: String(r.id || ""),+ spanId: String(r.spanId || ""),+ createdAt: String(r.createdAt || ""),+ verdict: String(r.verdict || ""),+ score: Number(r.score) || 0,+ classification: String(r.classification || ""),+ explanation: String(r.explanation || ""),+ source: r.source ? String(r.source) : undefined,+ cost: Number(r.cost) || 0,+ })),+ };+}diff --git a/src/client/src/lib/platform/evaluation/index.ts b/src/client/src/lib/platform/evaluation/index.tsindex 88d5ed4c..71290c5a 100644--- a/src/client/src/lib/platform/evaluation/index.ts+++ b/src/client/src/lib/platform/evaluation/index.ts@@ -44,32 +44,64 @@ import {import { CronType, CronRunStatus, CronLogData } from "@/types/cron";import { differenceInSeconds } from "date-fns";import { getFilterPreviousParams } from "@/helpers/server/platform";--/** Approximate cost per 1K tokens (USD) for common models. Fallback for evaluation cost. */-const EVAL_COST_PER_1K: Record<string, { prompt: number; completion: number }> = {- "gpt-4o": { prompt: 0.0025, completion: 0.01 },- "gpt-4o-mini": { prompt: 0.00015, completion: 0.0006 },- "gpt-4": { prompt: 0.03, completion: 0.06 },- "gpt-4-turbo": { prompt: 0.01, completion: 0.03 },- "gpt-3.5-turbo": { prompt: 0.0005, completion: 0.0015 },- "claude-3-5-sonnet": { prompt: 0.003, completion: 0.015 },- "claude-3-haiku": { prompt: 0.00025, completion: 0.00125 },- "claude-3-opus": { prompt: 0.015, completion: 0.075 },-};--function estimateEvaluationCost(+import { ProviderRegistry } from "@/lib/platform/providers/provider-registry";+import {+ resolveEvaluationType,+ type EvaluationTypeRef,+} from "@/helpers/client/evaluation-type";++/**+ * Cost from Manage Models (`openlit_provider_models`) — same source as+ * Pricing / Openground. Returns 0 if the model is not configured.+ */+async function estimateEvaluationCost(+ provider: string,model: string,promptTokens: number,- completionTokens: number-): number {- const modelKey = Object.keys(EVAL_COST_PER_1K).find(- (k) => model.toLowerCase().includes(k) || k.includes(model.toLowerCase())- );- const rates = modelKey ? EVAL_COST_PER_1K[modelKey] : { prompt: 0.001, completion: 0.002 };- return (- (promptTokens / 1000) * rates.prompt +- (completionTokens / 1000) * rates.completion- );+ completionTokens: number,+ databaseConfigId: string+): Promise<number> {+ if (!provider || !model || !databaseConfigId) return 0;+ if (promptTokens === 0 && completionTokens === 0) return 0;++ try {+ const modelMeta = await ProviderRegistry.getModel(+ provider,+ model,+ databaseConfigId+ );+ if (!modelMeta) {+ consoleLog(+ `Evaluation cost: model '${model}' not found under provider '${provider}' in Manage Models`+ );+ return 0;+ }++ return (+ (promptTokens / 1_000_000) * modelMeta.inputPricePerMToken ++ (completionTokens / 1_000_000) * modelMeta.outputPricePerMToken+ );+ } catch (e) {+ consoleLog("Evaluation cost lookup failed:", e);+ return 0;+ }+}++/**+ * Remap judge output names to configured type labels when possible so+ * ClickHouse stores canonical names (Hallucination) instead of variants+ * or inventons that still resolve (e.g. "Hallucination evaluation context").+ * Unmapped labels (TypeA) are left as-is and filtered from analytics.+ */+function normalizeEvaluationResults(+ evaluation: Evaluation[],+ customTypes: EvaluationTypeRef[] = []+): Evaluation[] {+ return evaluation.map((item) => {+ const resolved = resolveEvaluationType(item.evaluation, customTypes);+ if (!resolved) return item;+ return { ...item, evaluation: resolved.label };+ });}export async function getEvaluationSummaryForSpanId(spanId: string, dbConfigId?: string) {@@ -278,10 +310,12 @@ async function storeEvaluation(spanId: string,evaluation: Evaluation[],meta: any,- dbConfigId?: string+ dbConfigId?: string,+ customTypes: EvaluationTypeRef[] = []) {const now = new Date();const createdAt = toClickHouseDateTime(now);+ const normalized = normalizeEvaluationResults(evaluation, customTypes);// Ensure meta values are strings (Map(LowCardinality(String), String))const metaStrings =meta && typeof meta === "object"@@ -298,13 +332,13 @@ async function storeEvaluation(span_id: spanId,created_at: createdAt,meta: metaStrings,- "evaluationData.evaluation": evaluation.map((e) => e.evaluation),- "evaluationData.classification": evaluation.map(+ "evaluationData.evaluation": normalized.map((e) => e.evaluation),+ "evaluationData.classification": normalized.map((e) => e.classification),- "evaluationData.explanation": evaluation.map((e) => e.explanation),- "evaluationData.verdict": evaluation.map((e) => e.verdict),- scores: evaluation.reduce((acc: Record<string, number>, e) => {+ "evaluationData.explanation": normalized.map((e) => e.explanation),+ "evaluationData.verdict": normalized.map((e) => e.verdict),+ scores: normalized.reduce((acc: Record<string, number>, e) => {acc[e.evaluation] = e.score;return acc;}, {}),@@ -538,10 +572,12 @@ async function getEvaluationConfigForTrace(if (data.usage) {metaBase.promptTokens = String(data.usage.promptTokens);metaBase.completionTokens = String(data.usage.completionTokens);- const cost = estimateEvaluationCost(+ const cost = await estimateEvaluationCost(+ evaluationConfig.provider,evaluationConfig.model,data.usage.promptTokens,- data.usage.completionTokens+ data.usage.completionTokens,+ dbConfig);if (cost > 0) metaBase.cost = cost.toFixed(8);}@@ -550,7 +586,8 @@ async function getEvaluationConfigForTrace(trace.SpanId,data.result || [],metaBase,- dbConfig+ dbConfig,+ evaluationTypes.map((t) => ({ id: t.id, label: (t as any).label || t.id })));if (storeResult.err) {consoleLog(storeResult.err);@@ -933,21 +970,28 @@ export async function runOfflineEvaluation(if (data.usage) {metaBase.promptTokens = String(data.usage.promptTokens);metaBase.completionTokens = String(data.usage.completionTokens);- cost = estimateEvaluationCost(+ cost = await estimateEvaluationCost(+ evaluationConfig.provider,evaluationConfig.model,data.usage.promptTokens,- data.usage.completionTokens+ data.usage.completionTokens,+ databaseConfigId);if (cost > 0) metaBase.cost = cost.toFixed(8);}if (storeResults) {const spanId = `offline_${crypto.randomUUID()}`;+ const customTypes = allTypes.map((t) => ({+ id: t.id,+ label: (t as any).label || t.id,+ }));const storeResult = await storeEvaluation(spanId,data.result || [],metaBase,- databaseConfigId+ databaseConfigId,+ customTypes);if (storeResult?.err) {consoleLog("Failed to store offline eval results:", storeResult.err);diff --git a/src/client/src/lib/platform/evaluation/run-evaluation.ts b/src/client/src/lib/platform/evaluation/run-evaluation.tsindex 1a12ca6d..c929ce90 100644--- a/src/client/src/lib/platform/evaluation/run-evaluation.ts+++ b/src/client/src/lib/platform/evaluation/run-evaluation.ts@@ -44,8 +44,8 @@ Return JSON in format:{"success": true,"result": [- {"score": 0.8, "evaluation": "TypeA", "classification": "specific_issue", "explanation": "Score 0.8 because the response contradicts the provided context on key facts", "verdict": "yes"},- {"score": 0, "evaluation": "TypeB", "classification": "none", "explanation": "Score 0 because no issues were found — the response aligns with the provided context", "verdict": "no"}+ {"score": 0.8, "evaluation": "<TypeNameFromContextHeader>", "classification": "factual_inaccuracy", "explanation": "Score 0.8 because the response contradicts the provided context on key facts", "verdict": "yes"},+ {"score": 0, "evaluation": "<TypeNameFromContextHeader>", "classification": "none", "explanation": "Score 0 because no issues were found — the response aligns with the provided context", "verdict": "no"}]}diff --git a/src/client/src/lib/platform/llm/cost.ts b/src/client/src/lib/platform/llm/cost.tsindex 5489d832..90cb193c 100644--- a/src/client/src/lib/platform/llm/cost.ts+++ b/src/client/src/lib/platform/llm/cost.ts@@ -1,9 +1,26 @@import {+ dateTruncGroupingLogic,getFilterPreviousParams,getFilterWhereCondition,} from "@/helpers/server/platform";import { MetricParams, dataCollector, OTEL_TRACES_TABLE_NAME } from "../common";-import { getTraceMappingKeyFullPath } from "@/helpers/server/trace";+import {+ getTraceMappingKeyFullPath,+ getTraceMappingKeyFullPaths,+} from "@/helpers/server/trace";++function getProviderKeyPath() {+ const paths = (getTraceMappingKeyFullPaths("provider") as string[]).map(+ (path) => `SpanAttributes['${path}']`+ );+ return {+ paths,+ keyPath: paths.reduce(+ (expression, path) =>+ `if(notEmpty(${expression}), ${expression}, ${path})`+ ),+ };+}export async function getTotalCost(params: MetricParams) {const keyPath = `SpanAttributes['${getTraceMappingKeyFullPath("cost")}']`;@@ -71,20 +88,30 @@ export async function getAverageCost(params: MetricParams) {}export async function getCostByApplication(params: MetricParams) {- const keyPathApplicationName = `ResourceAttributes['${getTraceMappingKeyFullPath(- "applicationName"- )}']`;+ // Prefer OTel ServiceName / service.name (what instrumented apps emit).+ // Legacy spans may still carry gen_ai.application_name. The previous+ // path wrapped the SpanAttributes key in ResourceAttributes[...], which+ // never matched and left Cost by application empty.+ const applicationPaths = [+ "ServiceName",+ "ResourceAttributes['service.name']",+ "SpanAttributes['gen_ai.application_name']",+ ];+ const keyPathApplicationName = `coalesce(${applicationPaths+ .map((path) => `nullIf(${path}, '')`)+ .join(", ")})`;const keyPathCost = `SpanAttributes['${getTraceMappingKeyFullPath("cost")}']`;const query = `SELECT- DISTINCT ${keyPathApplicationName} As applicationName,- SUM(toFloat64OrZero(${keyPathCost})) AS cost+ ${keyPathApplicationName} AS applicationName,+ SUM(toFloat64OrZero(${keyPathCost})) AS costFROM ${OTEL_TRACES_TABLE_NAME}WHERE ${getFilterWhereCondition({...params,- notEmpty: [{ key: keyPathApplicationName }, { key: keyPathCost }],+ notOrEmpty: applicationPaths.map((key) => ({ key })),+ notEmpty: [{ key: keyPathCost }],operationType: "llm",}, true)}- GROUP BY applicationName;`;+ GROUP BY applicationName`;return dataCollector({ query });}@@ -107,3 +134,66 @@ export async function getCostByEnvironment(params: MetricParams) {return dataCollector({ query });}++export async function getCostByProvider(params: MetricParams) {+ const { paths, keyPath } = getProviderKeyPath();+ const keyPathCost = `SpanAttributes['${getTraceMappingKeyFullPath("cost")}']`;+ const query = `+ SELECT+ ${keyPath} AS provider,+ SUM(toFloat64OrZero(${keyPathCost})) AS cost+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE ${getFilterWhereCondition({+ ...params,+ notOrEmpty: paths.map((key) => ({ key })),+ notEmpty: [{ key: keyPathCost }],+ operationType: "llm",+ }, true)}+ GROUP BY provider+ ORDER BY cost DESC;+ `;++ return dataCollector({ query });+}++export async function getCostByModel(params: MetricParams) {+ const keyPathModel = `SpanAttributes['${getTraceMappingKeyFullPath("model")}']`;+ const keyPathCost = `SpanAttributes['${getTraceMappingKeyFullPath("cost")}']`;+ const query = `+ SELECT+ ${keyPathModel} AS model,+ SUM(toFloat64OrZero(${keyPathCost})) AS cost+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE ${getFilterWhereCondition({+ ...params,+ notEmpty: [{ key: keyPathModel }, { key: keyPathCost }],+ operationType: "llm",+ }, true)}+ GROUP BY model+ ORDER BY cost DESC;+ `;++ return dataCollector({ query });+}++export async function getCostPerTime(params: MetricParams) {+ const { start, end } = params.timeLimit;+ const dateTrunc = dateTruncGroupingLogic(end as Date, start as Date);+ const keyPathCost = `SpanAttributes['${getTraceMappingKeyFullPath("cost")}']`;++ const query = `+ SELECT+ CAST(SUM(toFloat64OrZero(${keyPathCost})) AS FLOAT) AS cost,+ formatDateTime(DATE_TRUNC('${dateTrunc}', Timestamp), '%Y/%m/%d %R') AS request_time+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE ${getFilterWhereCondition({+ ...params,+ notEmpty: [{ key: keyPathCost }],+ operationType: "llm",+ }, true)}+ GROUP BY request_time+ ORDER BY request_time;+ `;++ return dataCollector({ query });+}diff --git a/src/client/src/lib/platform/llm/endpoint.ts b/src/client/src/lib/platform/llm/endpoint.tsindex 74eefeb8..2c7f1240 100644--- a/src/client/src/lib/platform/llm/endpoint.ts+++ b/src/client/src/lib/platform/llm/endpoint.ts@@ -1,9 +1,22 @@import { getFilterWhereCondition } from "@/helpers/server/platform";import { MetricParams, dataCollector, OTEL_TRACES_TABLE_NAME } from "../common";-import { getTraceMappingKeyFullPath } from "@/helpers/server/trace";+import { getTraceMappingKeyFullPaths } from "@/helpers/server/trace";++function getProviderKeyPath() {+ const paths = (getTraceMappingKeyFullPaths("provider") as string[]).map(+ (path) => `SpanAttributes['${path}']`+ );+ return {+ paths,+ keyPath: paths.reduce(+ (expression, path) =>+ `if(notEmpty(${expression}), ${expression}, ${path})`+ ),+ };+}export async function getResultGenerationByEndpoint(params: MetricParams) {- const keyPath = `SpanAttributes['${getTraceMappingKeyFullPath("provider")}']`;+ const { paths, keyPath } = getProviderKeyPath();const query = `SELECT${keyPath} AS provider,@@ -12,7 +25,7 @@ export async function getResultGenerationByEndpoint(params: MetricParams) {${OTEL_TRACES_TABLE_NAME}WHERE ${getFilterWhereCondition({...params,- notEmpty: [{ key: keyPath }],+ notOrEmpty: paths.map((key) => ({ key })),operationType: "llm",}, true)}GROUP BY provider;diff --git a/src/client/src/lib/platform/manage-dashboard/widget.ts b/src/client/src/lib/platform/manage-dashboard/widget.tsindex 71d3fd1a..c79dad4a 100644--- a/src/client/src/lib/platform/manage-dashboard/widget.ts+++ b/src/client/src/lib/platform/manage-dashboard/widget.ts@@ -8,10 +8,55 @@ import {sanitizeWidget,escapeSingleQuotes,} from "@/helpers/server/widget";-import mustache from "mustache";-import { jsonStringify } from "@/utils/json";+/**+ * Resolve a dotted path under the filter object (e.g. `timeLimit.start`).+ * Returns undefined when any segment is missing — callers treat that as "".+ */+function getFilterPathValue(filter: unknown, path: string): unknown {+ const parts = path.split(".");+ let current: unknown = filter;+ for (const part of parts) {+ if (current == null || typeof current !== "object") {+ return undefined;+ }+ current = (current as Record<string, unknown>)[part];+ }+ return current;+}++/**+ * Substitute only `{{filter.*}}` / `{{{filter.*}}}` placeholders.+ *+ * Intentionally does **not** use Mustache (or any template engine): the+ * query string is user-controlled on the widget preview path, and treating+ * it as an executable template is a code-injection sink (CodeQL+ * js/code-injection / alert #126). Path lookup + stringification is enough+ * for dashboard filter interpolation, and filter values are validated+ * separately before substitution.+ */+function renderFilterPlaceholders(+ template: string,+ filter: MetricParams+): string {+ return template.replace(+ /\{\{\{\s*filter\.([a-zA-Z0-9_.]+)\s*\}\}\}|\{\{\s*filter\.([a-zA-Z0-9_.]+)\s*\}\}/g,+ (+ _match,+ unescapedPath: string | undefined,+ escapedPath: string | undefined+ ) => {+ const path = unescapedPath || escapedPath;+ if (!path) return "";+ const value = getFilterPathValue(filter, path);+ if (value == null) return "";+ if (typeof value === "object") return JSON.stringify(value);+ return String(value);+ }+ );+}+export async function getWidgetById(id: string) {const query = `SELECT id, title, description, widget_type AS type, created_at AS createdAt, updated_at AS updatedAt, properties,@@ -185,12 +230,70 @@ function validateQuery(query: string): { valid: boolean; error?: string } {return validateSafeQueryContent(trimmed);}+/**+ * Blank out the contents of single-quoted string literals so keyword /+ * table / function scanning only inspects executable SQL. Attribute keys+ * and values such as `SpanAttributes['gen_ai.system']` or+ * `'openlit-cli'` are data, not SQL — without this a literal like+ * `gen_ai.system` would false-positive on the `SYSTEM` keyword blocklist+ * (word-bounded by `.` and `'`) and reject an otherwise safe SELECT.+ * Handles both backslash (`\'`) and doubled (`''`) quote escapes.+ *+ * Implemented as a linear scan (not a regex) to avoid ReDoS from+ * overlapping alternatives in patterns like `'(?:\\.|''|[^'])*'`.+ * Unclosed literals are left unchanged, matching the previous regex.+ */+function stripStringLiterals(value: string): string {+ let result = "";+ let i = 0;+ while (i < value.length) {+ if (value[i] !== "'") {+ result += value[i];+ i += 1;+ continue;+ }++ const start = i;+ i += 1; // skip opening quote+ let closed = false;+ while (i < value.length) {+ const ch = value[i];+ if (ch === "\\" && i + 1 < value.length) {+ i += 2;+ continue;+ }+ if (ch === "'") {+ if (i + 1 < value.length && value[i + 1] === "'") {+ i += 2; // doubled-quote escape+ continue;+ }+ i += 1; // closing quote+ closed = true;+ break;+ }+ i += 1;+ }++ if (closed) {+ result += "''";+ } else {+ // No closing quote — leave the remainder intact so keyword+ // scanners can still see executable SQL after the opener.+ result += value.slice(start);+ break;+ }+ }+ return result;+}+function validateSafeQueryContent(value: string): { valid: boolean; error?: string } {- if (/\bsystem\./i.test(value)) {+ const scannable = stripStringLiterals(value);++ if (/\bsystem\./i.test(scannable)) {return { valid: false, error: "Access to system tables is not allowed" };}- if (/\binformation_schema\./i.test(value)) {+ if (/\binformation_schema\./i.test(scannable)) {return {valid: false,error: "Access to information_schema tables is not allowed",@@ -199,13 +302,13 @@ function validateSafeQueryContent(value: string): { valid: boolean; error?: striconst dangerousFunctions =/\b(url|file|remote|mysql|jdbc|s3|hdfs|input|numbers_mt|generateRandom|clusterAllReplicas)\s*\(/i;- if (dangerousFunctions.test(value)) {+ if (dangerousFunctions.test(scannable)) {return { valid: false, error: "Query contains disallowed functions" };}const dangerousKeywords =/\b(DROP|ALTER|TRUNCATE|INSERT|UPDATE|DELETE|CREATE|GRANT|REVOKE|INTO\s+OUTFILE|ATTACH|DETACH|RENAME|OPTIMIZE|SYSTEM)\b/i;- if (dangerousKeywords.test(value)) {+ if (dangerousKeywords.test(scannable)) {return { valid: false, error: "Query contains disallowed operations" };}@@ -259,7 +362,7 @@ export async function runWidgetQuery(return { err: filterValidation.error || "Invalid filter" };}- const exactQuery = mustache.render(query, { filter });+ const exactQuery = renderFilterPlaceholders(query, filter);const validation = validateQuery(exactQuery);if (!validation.valid) {diff --git a/src/client/src/lib/platform/openground/cost-analytics.ts b/src/client/src/lib/platform/openground/cost-analytics.tsnew file mode 100644index 00000000..870d6da7--- /dev/null+++ b/src/client/src/lib/platform/openground/cost-analytics.ts@@ -0,0 +1,108 @@+import { dataCollector, MetricParams } from "@/lib/platform/common";+import {+ OPENLIT_OPENGROUND_TABLE_NAME,+ OPENLIT_OPENGROUND_PROVIDERS_TABLE_NAME,+} from "@/lib/platform/openground/table-details";+import { getDBConfigByUser } from "@/lib/db-config";+import asaw from "@/utils/asaw";+import Sanitizer from "@/utils/sanitizer";+import { getFilterPreviousParams } from "@/helpers/server/platform";++function timeWhere(params: MetricParams, alias = "o") {+ return `+ ${alias}.created_at >= parseDateTimeBestEffort('${params.timeLimit.start}')+ AND ${alias}.created_at <= parseDateTimeBestEffort('${params.timeLimit.end}')+ `;+}++async function getActiveDatabaseConfigId(): Promise<string | null> {+ const [err, dbConfig] = await asaw(getDBConfigByUser(true));+ if (err || !dbConfig?.id) return null;+ return dbConfig.id as string;+}++export async function getOpengroundTotalCost(params: MetricParams) {+ const databaseConfigId = await getActiveDatabaseConfigId();+ if (!databaseConfigId) {+ return { data: [{ total_cost: 0, previous_total_cost: 0 }] };+ }++ const sanitizedId = Sanitizer.sanitizeValue(databaseConfigId);+ const current = params;+ const previous = getFilterPreviousParams(current);+ // Shared join key (same pattern as LLM getTotalCost) so the INNER JOIN+ // still returns a row when current vs previous window starts differ.+ const joinKey = Sanitizer.sanitizeValue(String(params.timeLimit.start));++ // p.cost is Float64 — toFloat64OrZero only accepts String and fails the query.+ const periodQuery = (periodParams: MetricParams) => `+ SELECT+ sum(ifNull(p.cost, 0)) AS total_cost,+ '${joinKey}' AS start_date+ FROM ${OPENLIT_OPENGROUND_PROVIDERS_TABLE_NAME} AS p+ INNER JOIN ${OPENLIT_OPENGROUND_TABLE_NAME} AS o+ ON p.openground_id = o.id+ WHERE o.database_config_id = '${sanitizedId}'+ AND ${timeWhere(periodParams, "o")}+ `;++ const query = `+ SELECT+ CAST(current_data.total_cost AS FLOAT) AS total_cost,+ CAST(previous_day.total_cost AS FLOAT) AS previous_total_cost+ FROM (${periodQuery(current)}) AS current_data+ JOIN (${periodQuery(previous)}) AS previous_day+ ON current_data.start_date = previous_day.start_date+ `;++ return dataCollector({ query }, "query", databaseConfigId);+}++export async function getOpengroundCostByProvider(params: MetricParams) {+ const databaseConfigId = await getActiveDatabaseConfigId();+ if (!databaseConfigId) {+ return { data: [] };+ }++ const sanitizedId = Sanitizer.sanitizeValue(databaseConfigId);+ const query = `+ SELECT+ p.provider AS provider,+ SUM(ifNull(p.cost, 0)) AS cost+ FROM ${OPENLIT_OPENGROUND_PROVIDERS_TABLE_NAME} AS p+ INNER JOIN ${OPENLIT_OPENGROUND_TABLE_NAME} AS o+ ON p.openground_id = o.id+ WHERE o.database_config_id = '${sanitizedId}'+ AND ${timeWhere(params, "o")}+ AND notEmpty(p.provider)+ GROUP BY provider+ ORDER BY cost DESC+ `;++ return dataCollector({ query }, "query", databaseConfigId);+}++export async function getOpengroundCostAnalytics(params: MetricParams) {+ const [totalRes, byProviderRes] = await Promise.all([+ getOpengroundTotalCost(params),+ getOpengroundCostByProvider(params),+ ]);++ const totalRow = Array.isArray(totalRes.data)+ ? (totalRes.data[0] as {+ total_cost?: number;+ previous_total_cost?: number;+ })+ : undefined;++ return {+ data: [+ {+ total_cost: Number(totalRow?.total_cost) || 0,+ previous_total_cost: Number(totalRow?.previous_total_cost) || 0,+ },+ ],+ byProvider: byProviderRes.data || [],+ err: totalRes.err || byProviderRes.err,+ };+}diff --git a/src/client/src/lib/platform/pricing/cron-analytics.ts b/src/client/src/lib/platform/pricing/cron-analytics.tsnew file mode 100644index 00000000..9b714c5b--- /dev/null+++ b/src/client/src/lib/platform/pricing/cron-analytics.ts@@ -0,0 +1,142 @@+import { dataCollector, MetricParams } from "@/lib/platform/common";+import { OPENLIT_CRON_LOG_TABLE_NAME } from "@/lib/platform/cron-log/table-details";+import { CronType } from "@/types/cron";+import Sanitizer from "@/utils/sanitizer";+import { getFilterPreviousParams } from "@/helpers/server/platform";++export type PricingCronRunRow = {+ startedAt: string;+ finishedAt: string;+ duration: number;+ runStatus: string;+ totalSpans: number;+ totalUpdated: number;+ totalFailed: number;+ totalSkipped: number;+};++export type PricingCronAnalyticsSummary = {+ totalRuns: number;+ previousTotalRuns: number;+ totalUpdated: number;+ previousTotalUpdated: number;+ totalSpans: number;+ previousTotalSpans: number;+ successfulRuns: number;+ previousSuccessfulRuns: number;+};++function timeWhere(params: MetricParams) {+ const start = Sanitizer.sanitizeValue(String(params.timeLimit.start));+ const end = Sanitizer.sanitizeValue(String(params.timeLimit.end));+ return `+ started_at >= parseDateTimeBestEffort('${start}')+ AND started_at <= parseDateTimeBestEffort('${end}')+ AND cron_type = '${CronType.SPAN_PRICING}'+ `;+}++function toNumber(value: unknown): number {+ const n = Number(value);+ return Number.isFinite(n) ? n : 0;+}++async function getSummaryForPeriod(+ params: MetricParams+): Promise<PricingCronAnalyticsSummary & { err?: unknown }> {+ const query = `+ SELECT+ count() AS totalRuns,+ countIf(run_status = 'SUCCESS' OR run_status = 'PARTIAL_SUCCESS') AS successfulRuns,+ sum(toFloat64OrZero(meta['totalUpdated'])) AS totalUpdated,+ sum(toFloat64OrZero(meta['totalSpans'])) AS totalSpans+ FROM ${OPENLIT_CRON_LOG_TABLE_NAME}+ WHERE ${timeWhere(params)}+ `;++ const { data, err } = await dataCollector({ query });+ if (err) {+ return {+ err,+ totalRuns: 0,+ previousTotalRuns: 0,+ totalUpdated: 0,+ previousTotalUpdated: 0,+ totalSpans: 0,+ previousTotalSpans: 0,+ successfulRuns: 0,+ previousSuccessfulRuns: 0,+ };+ }++ const row = Array.isArray(data) ? data[0] : undefined;+ return {+ totalRuns: toNumber(row?.totalRuns),+ previousTotalRuns: 0,+ totalUpdated: toNumber(row?.totalUpdated),+ previousTotalUpdated: 0,+ totalSpans: toNumber(row?.totalSpans),+ previousTotalSpans: 0,+ successfulRuns: toNumber(row?.successfulRuns),+ previousSuccessfulRuns: 0,+ };+}++export async function getPricingCronAnalytics(+ params: MetricParams,+ options?: { limit?: number }+) {+ const limit = Math.min(Math.max(options?.limit ?? 25, 1), 100);+ const previousParams = getFilterPreviousParams(params);++ const [current, previous, runsRes] = await Promise.all([+ getSummaryForPeriod(params),+ getSummaryForPeriod(previousParams),+ dataCollector({+ query: `+ SELECT+ started_at AS startedAt,+ finished_at AS finishedAt,+ duration,+ run_status AS runStatus,+ toFloat64OrZero(meta['totalSpans']) AS totalSpans,+ toFloat64OrZero(meta['totalUpdated']) AS totalUpdated,+ toFloat64OrZero(meta['totalFailed']) AS totalFailed,+ toFloat64OrZero(meta['totalSkipped']) AS totalSkipped+ FROM ${OPENLIT_CRON_LOG_TABLE_NAME}+ WHERE ${timeWhere(params)}+ AND toFloat64OrZero(meta['totalUpdated']) > 0+ ORDER BY started_at DESC+ LIMIT ${limit}+ `,+ }),+ ]);++ const runs = ((runsRes.data as PricingCronRunRow[]) || []).map((row) => ({+ startedAt: String(row.startedAt ?? ""),+ finishedAt: String(row.finishedAt ?? ""),+ duration: toNumber(row.duration),+ runStatus: String(row.runStatus ?? ""),+ totalSpans: toNumber(row.totalSpans),+ totalUpdated: toNumber(row.totalUpdated),+ totalFailed: toNumber(row.totalFailed),+ totalSkipped: toNumber(row.totalSkipped),+ }));++ return {+ data: [+ {+ total_runs: current.totalRuns,+ previous_total_runs: previous.totalRuns,+ successful_runs: current.successfulRuns,+ previous_successful_runs: previous.successfulRuns,+ total_updated: current.totalUpdated,+ previous_total_updated: previous.totalUpdated,+ total_spans: current.totalSpans,+ previous_total_spans: previous.totalSpans,+ },+ ],+ runs,+ err: current.err || previous.err || runsRes.err,+ };+}diff --git a/src/client/src/lib/platform/pricing/guidance.ts b/src/client/src/lib/platform/pricing/guidance.tsnew file mode 100644index 00000000..f131adf4--- /dev/null+++ b/src/client/src/lib/platform/pricing/guidance.ts@@ -0,0 +1,43 @@+import { dataCollector, MetricParams, OTEL_TRACES_TABLE_NAME } from "@/lib/platform/common";+import { getFilterWhereCondition } from "@/helpers/server/platform";+import { getTraceMappingKeyFullPath } from "@/helpers/server/trace";+import { getPricingConfig } from "@/lib/platform/pricing/config";++export type CostPricingGuidance = {+ autoEnabled: boolean;+ missingCostSpans: number;+ showBackfillBanner: boolean;+};++export async function getCostPricingGuidance(+ params: MetricParams+): Promise<CostPricingGuidance> {+ const config = await getPricingConfig();+ const autoEnabled = !!config?.auto;++ const costKeyPath = `SpanAttributes['${getTraceMappingKeyFullPath("cost")}']`;+ const query = `+ SELECT+ CAST(count() AS INTEGER) AS missing_cost_spans+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE ${getFilterWhereCondition(+ {+ ...params,+ operationType: "llm",+ },+ true+ )}+ AND (${costKeyPath} = '' OR toFloat64OrZero(${costKeyPath}) = 0)+ `;++ const { data, err } = await dataCollector({ query });+ const row = Array.isArray(data) ? data[0] : undefined;+ const missingCostSpans =+ err || !row ? 0 : Number(row.missing_cost_spans) || 0;++ return {+ autoEnabled,+ missingCostSpans,+ showBackfillBanner: !autoEnabled && missingCostSpans > 0,+ };+}diff --git a/src/client/src/lib/platform/pricing/index.ts b/src/client/src/lib/platform/pricing/index.tsindex 0cb24861..fb29e7f1 100644--- a/src/client/src/lib/platform/pricing/index.ts+++ b/src/client/src/lib/platform/pricing/index.ts@@ -19,9 +19,11 @@ import asaw from "@/utils/asaw";const COST_KEY = getTraceMappingKeyFullPath("cost") as string; // gen_ai.usage.costconst MODEL_KEY = getTraceMappingKeyFullPath("model") as string; // gen_ai.request.model-const PROVIDER_KEY = getTraceMappingKeyFullPath("provider") as string; // gen_ai.system+const PROVIDER_KEYS = getTraceMappingKeyFullPaths("provider") as string[]; // gen_ai.provider.name, gen_ai.systemconst TYPE_KEY = getTraceMappingKeyFullPath("type") as string; // gen_ai.operation.name-const PROMPT_TOKENS_KEYS = getTraceMappingKeyFullPaths("promptTokens") as string[];+const PROMPT_TOKENS_KEYS = getTraceMappingKeyFullPaths(+ "promptTokens"+) as string[];const COMPLETION_TOKENS_KEYS = getTraceMappingKeyFullPaths("completionTokens") as string[];@@ -36,6 +38,14 @@ function getAttr(trace: TraceRow, key: string): string {return (trace.SpanAttributes || {})[key] ?? "";}+function getFirstAttr(trace: TraceRow, keys: string[]): string {+ for (const key of keys) {+ const value = getAttr(trace, key);+ if (value) return value;+ }+ return "";+}+function getNumericAttr(trace: TraceRow, keys: string[]): number {const attributes = trace.SpanAttributes || {};@@ -63,7 +73,7 @@ async function computeCostForTrace(trace: TraceRow,databaseConfigId: string): Promise<{ cost: number | null; reason?: string }> {- const provider = getAttr(trace, PROVIDER_KEY);+ const provider = getFirstAttr(trace, PROVIDER_KEYS);const model = getAttr(trace, MODEL_KEY);const promptTokens = getNumericAttr(trace, PROMPT_TOKENS_KEYS);const completionTokens = getNumericAttr(trace, COMPLETION_TOKENS_KEYS);diff --git a/src/client/src/lib/platform/request/index.ts b/src/client/src/lib/platform/request/index.tsindex 5b85ad59..7d69c05a 100644--- a/src/client/src/lib/platform/request/index.ts+++ b/src/client/src/lib/platform/request/index.ts@@ -1,6 +1,7 @@import { MetricParams, dataCollector, OTEL_TRACES_TABLE_NAME } from "../common";import {getTraceMappingKeyFullPath,+ getTraceMappingKeyFullPaths,buildHierarchy,} from "@/helpers/server/trace";import { SYNTHETIC_SPAN_ID_PREFIX } from "@/helpers/client/trace";@@ -13,7 +14,12 @@ import {const PREDEFINED_GROUP_BY: Record<string, string> = {model: `SpanAttributes['gen_ai.request.model']`,- provider: `SpanAttributes['gen_ai.system']`,+ // OTel GenAI renamed gen_ai.system -> gen_ai.provider.name (1.30+). The+ // current SDK emits provider.name. This must stay a single-attribute+ // expression so the drill-down customFilter in grouped-table.tsx+ // (SpanAttributes['gen_ai.provider.name'] = <value>) matches the exact+ // same rows this GROUP BY produces.+ provider: `SpanAttributes['gen_ai.provider.name']`,spanName: `SpanName`,applicationName: `ResourceAttributes['service.name']`,};@@ -156,11 +162,19 @@ export async function getRequestsConfig(params: MetricParams) {// of that vendor's telemetry from any view — the user asked// for vendor visibility without a new control. The WHERE// side of this fold lives in helpers/server/platform.ts.+ // LLM provider lives on gen_ai.provider.name (OTel 1.30+) with a+ // gen_ai.system fallback for legacy spans; fold both plus the+ // vector-DB (db.system) and coding-agent (coding_agent.client)+ // namespaces so the "Provider" dropdown is a universal traffic-shaper.+ const llmProviderArrays = (getTraceMappingKeyFullPaths("provider") as string[])+ .map(+ (path) =>+ `arrayFilter(x -> x != '', groupArray(DISTINCT SpanAttributes['${path}']))`+ )+ .join(",\n\t\t\t");select.push(`arrayConcat(- arrayFilter(x -> x != '', groupArray(DISTINCT SpanAttributes['${getTraceMappingKeyFullPath(- "provider"- )}'])),+ ${llmProviderArrays},arrayFilter(x -> x != '', groupArray(DISTINCT SpanAttributes['${getTraceMappingKeyFullPath("system")}'])),@@ -198,6 +212,10 @@ export async function getRequestsConfig(params: MetricParams) {`arrayFilter(x -> x != '', ARRAY_AGG(DISTINCT SpanName)) AS spanNames`);+ select.push(+ `arrayFilter(x -> x != '', ARRAY_AGG(DISTINCT ServiceName)) AS services`+ );+select.push(// OTel-standard environment is `ResourceAttributes['deployment.environment']`.// The legacy `getTraceMappingKeyFullPath("environment")` returns a@@ -335,8 +353,16 @@ export async function getHeirarchyViaSpanId(spanId: string, dbConfigId?: string))} = '${safeTraceId}' OR SpanAttributes['coding_agent.session.id'] = '${escapeClickHouseString(codingSessionId)}' OR ResourceAttributes['coding_agent.agent.parent_id'] = '${escapeClickHouseString(codingSessionId)}' OR SpanAttributes['coding_agent.agent.parent_id'] = '${escapeClickHouseString(codingSessionId)}')`: `${getTraceMappingKeyFullPath("id")} = '${safeTraceId}'`;- const allSpansQuery = `- SELECT+ // Span columns shared by the hierarchy SELECT. Coding-agent sessions+ // can exceed a few thousand spans (tool noise); a naive+ // `ORDER BY Timestamp ASC LIMIT 5000` kept the *oldest* slice and+ // dropped recent user prompts / assistant replies from Chat view.+ // For coding sessions we:+ // 1. take the newest window (DESC + limit),+ // 2. UNION any prompt/response-bearing turns so Chat never loses+ // conversation turns that fell outside that window,+ // 3. re-sort ascending in JS before building the tree.+ const spanSelectColumns = `${getTraceMappingKeyFullPath("id")},${getTraceMappingKeyFullPath("parentSpanId")},${getTraceMappingKeyFullPath("spanId")},@@ -354,18 +380,54 @@ export async function getHeirarchyViaSpanId(spanId: string, dbConfigId?: string)SpanAttributes,ResourceAttributes,Events,- Links+ Links`;++ const allSpansQuery = codingSessionId+ ? `+ SELECT * FROM (+ SELECT ${spanSelectColumns}+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE ${filterClause}+ ORDER BY Timestamp DESC+ LIMIT 8000+ )+ UNION DISTINCT+ SELECT * FROM (+ SELECT ${spanSelectColumns}+ FROM ${OTEL_TRACES_TABLE_NAME}+ WHERE ${filterClause}+ AND (+ notEmpty(SpanAttributes['gen_ai.input.messages'])+ OR notEmpty(SpanAttributes['gen_ai.output.messages'])+ OR SpanAttributes['coding_agent.llm.turn.kind'] IN ('user_prompt', 'assistant_only')+ )+ ORDER BY Timestamp DESC+ LIMIT 2000+ )+ `+ : `+ SELECT ${spanSelectColumns}FROM ${OTEL_TRACES_TABLE_NAME}WHERE ${filterClause}ORDER BY Timestamp ASCLIMIT 5000`;- const { data: allSpans, err: allSpansErr } = await dataCollector(+ const { data: allSpansRaw, err: allSpansErr } = await dataCollector({ query: allSpansQuery },"query",dbConfigId);+ const allSpans = Array.isArray(allSpansRaw)+ ? codingSessionId+ ? [...allSpansRaw].sort((a: any, b: any) => {+ const ta = a?.Timestamp ? new Date(a.Timestamp).getTime() : 0;+ const tb = b?.Timestamp ? new Date(b.Timestamp).getTime() : 0;+ return ta - tb;+ })+ : allSpansRaw+ : allSpansRaw;+if (allSpansErr || !Array.isArray(allSpans) || allSpans.length === 0) {return { err: "Failed to fetch trace spans", record: {} };}diff --git a/src/client/src/middleware/check-auth.ts b/src/client/src/middleware/check-auth.tsindex 30d049c8..e81cd191 100644--- a/src/client/src/middleware/check-auth.ts+++ b/src/client/src/middleware/check-auth.ts@@ -34,7 +34,10 @@ function isValidCronJobRequest(request: NextRequest) {}const rateLimitWindows = new Map<string, { count: number; resetAt: number }>();-const SENSITIVE_API_RATE_LIMIT = 120;+// UI pages (telemetry/agents/dashboards) fire multiple parallel reads per+// navigation. 120/min was too low and 429'd normal tab switching; 10x keeps+// a abuse ceiling without breaking self-hosted browsing + auto-refresh.+const SENSITIVE_API_RATE_LIMIT = 1200;const SENSITIVE_API_RATE_LIMIT_WINDOW_MS = 60_000;const SENSITIVE_API_RATE_LIMIT_CLEANUP_MS = 60_000;const RATE_LIMITED_API_PREFIXES = [diff --git a/src/client/src/types/evaluation.ts b/src/client/src/types/evaluation.tsindex 2b0501a3..5ce6bfd3 100644--- a/src/client/src/types/evaluation.ts+++ b/src/client/src/types/evaluation.ts@@ -59,3 +59,70 @@ export interface AutoEvaluationConfig {}export type EvaluationConfigWithSecret = EvaluationConfig & { secret: Secret };++export type EvaluationAnalyticsSummary = {+ tracesEvaluated: number;+ executions: number;+ autoExecutions: number;+ totalCost: number;+ avgPassRate: number;+ failedScores: number;+ previous_tracesEvaluated: number;+ previous_executions: number;+ previous_autoExecutions: number;+ previous_totalCost: number;+ previous_avgPassRate: number;+ previous_failedScores: number;+ evaluations: number;+ active: number;+};++export type EvaluationAnalyticsTimeseriesPoint = {+ timestamp: string;+ executions: number;+ passRate: number;+};++export type EvaluationAnalyticsByTypeRow = {+ evaluation: string;+ executions: number;+ passRate: number;+ previousPassRate: number;+ failedScores: number;+};++export type EvaluationAnalyticsRecentResult = {+ id: string;+ spanId: string;+ createdAt: string;+ verdict: string;+ score: number;+ classification: string;+ explanation: string;+ source?: string;+ cost?: number;+};++export type EvaluationEvaluatorAnalyticsResponse = {+ configured: boolean;+ found: boolean;+ evaluator?: {+ id: string;+ label: string;+ enabled: boolean;+ description?: string;+ isCustom?: boolean;+ };+ data?: Array<Record<string, number>>;+ timeseries?: EvaluationAnalyticsTimeseriesPoint[];+ recentResults?: EvaluationAnalyticsRecentResult[];+};++export type EvaluationAnalyticsResponse = {+ configured: boolean;+ data?: Array<Record<string, number>>;+ summary?: EvaluationAnalyticsSummary;+ timeseries?: EvaluationAnalyticsTimeseriesPoint[];+ byType?: EvaluationAnalyticsByTypeRow[];+};+diff --git a/src/client/src/types/platform.ts b/src/client/src/types/platform.tsindex 062f10d0..d44a6bbf 100644--- a/src/client/src/types/platform.ts+++ b/src/client/src/types/platform.ts@@ -29,6 +29,8 @@ export type FilterWhereConditionType = {traceTypes: string[];applicationNames: string[];serviceNames: string[];+ /** UI ComboDropdown key — same semantics as serviceNames for ServiceName WHERE. */+ services: string[];spanNames: string[];environments: string[];customFilters: { attributeType: string; key: string; value: string }[];diff --git a/src/client/src/utils/breadcrumbs.ts b/src/client/src/utils/breadcrumbs.tsindex af20bbba..437c5d9a 100644--- a/src/client/src/utils/breadcrumbs.ts+++ b/src/client/src/utils/breadcrumbs.ts@@ -77,6 +77,27 @@ export const ROUTE_CONFIGS: RouteConfig[] = [getBreadcrumbs: () => [],},+ // Costs+ {+ regex: /^\/costs$/,+ getTitle: () => getMessage().COSTS_TITLE,+ getBreadcrumbs: () => [],+ },+ {+ regex: /^\/pricing$/,+ getTitle: () => getMessage().COSTS_TAB_CONFIGURATION,+ getBreadcrumbs: () => [+ { title: getMessage().COSTS_TITLE, href: "/costs" },+ ],+ },+ {+ regex: /^\/manage-models$/,+ getTitle: () => getMessage().OPENGROUND_MANAGE_MODELS,+ getBreadcrumbs: () => [+ { title: getMessage().COSTS_TITLE, href: "/costs" },+ ],+ },+// Dashboard routes{regex: /^\/dashboard$/,@@ -259,11 +280,37 @@ export const ROUTE_CONFIGS: RouteConfig[] = [getBreadcrumbs: () => [],},+ {+ regex: /^\/evaluations\/evaluators\/[^/]+$/,+ getTitle: () => "Evaluator",+ getBreadcrumbs: () => [+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" },+ { title: "Evaluator", href: "/evaluations" },+ ],+ },+ {+ regex: /^\/evaluations\/types\/[^/]+$/,+ getTitle: () => "Evaluation Type",+ getBreadcrumbs: () => [+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" },+ { title: "Evaluators", href: "/evaluations?tab=evaluators" },+ { title: "Evaluation Type", href: "/evaluations/types" },+ ],+ },+ {+ regex: /^\/evaluations\/types\/new$/,+ getTitle: () => "Create Evaluation Type",+ getBreadcrumbs: () => [+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" },+ { title: "Evaluators", href: "/evaluations?tab=evaluators" },+ { title: "Create", href: "/evaluations/types/new" },+ ],+ },{regex: /^\/evaluations\/settings$/,getTitle: () => "Evaluation Settings",getBreadcrumbs: () => [- { title: "Evaluations", href: "/evaluations" },+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" },{ title: "Settings", href: "/evaluations/settings" }],},@@ -271,7 +318,7 @@ export const ROUTE_CONFIGS: RouteConfig[] = [regex: /^\/evaluations\/types$/,getTitle: () => "Evaluation Types",getBreadcrumbs: () => [- { title: "Evaluations", href: "/evaluations" },+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" },{ title: "Evaluation Types", href: "/evaluations/types" }],},@@ -279,15 +326,15 @@ export const ROUTE_CONFIGS: RouteConfig[] = [regex: /^\/evaluations\/manual$/,getTitle: () => "Manual Marking",getBreadcrumbs: () => [- { title: "Evaluations", href: "/evaluations" },+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" },{ title: "Manual Marking", href: "/evaluations/manual" }],},{regex: /^\/evaluations\/?$/,- getTitle: () => "Evaluations",+ getTitle: () => getMessage().FEATURE_EVALS,getBreadcrumbs: () => [- { title: "Evaluations", href: "/evaluations" }+ { title: getMessage().FEATURE_EVALS, href: "/evaluations" }],},@@ -298,7 +345,25 @@ export const ROUTE_CONFIGS: RouteConfig[] = [{ title: "Settings", href: "/settings" }],},-++ {+ regex: /^\/organisation\/?$/,+ getTitle: () => getMessage().ORGANISATION,+ getBreadcrumbs: () => [+ { title: "Settings", href: "/settings" },+ { title: getMessage().ORGANISATION, href: "/organisation" },+ ],+ },+ {+ regex: /^\/organisation\/project\/[^/]+$/,+ getTitle: () => "",+ getBreadcrumbs: () => [+ { title: "Settings", href: "/settings" },+ { title: getMessage().ORGANISATION, href: "/organisation" },+ { title: getMessage().PROJECTS, href: "/organisation?tab=projects" },+ ],+ },+{regex: /^\/settings\/api-keys$/,getTitle: () => "Api Keys",@@ -342,7 +407,7 @@ export const ROUTE_CONFIGS: RouteConfig[] = [// under `/agents/<key>` like the per-vendor detail page) because it// rolls up across vendors. Without an explicit route config the// breadcrumb generator falls through to the path-tail fallback,- // which yields "Home › [email protected]" — useless context for+ // which yields "Home › username — useless context for// the operator. We thread back through the Coding Agents tab on// the unified Agents hub so the user can backtrack out.{