Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
---
title: 'Installation'
description: 'Install the OpenTelemetry GPU Collector via Docker, binary, or from source'
icon: 'circle-down'
---
## Docker (recommended)
The easiest way to run the collector. The image is published to GitHub Container Registry and supports `linux/amd64` and `linux/arm64`.
```bash
docker pull ghcr.io/openlit/otel-gpu-collector:latest
```
For per-process GPU attribution (cmdline, PID, zombie/`process.state`, owner), run with host PID namespace access: Docker `--pid=host`, Compose `pid: host`, or Kubernetes `hostPID: true`. Device-level `hw.gpu.*` metrics work without it.
### Tags
| Tag | Description |
|---|---|
| `latest` | Most recent release |
| `1.2.3` | Specific version |
| `1.2` | Latest patch of a minor version |
---
## Pre-built binaries
Download a binary for your platform from the [GitHub Releases](https://github.com/openlit/openlit/releases) page. Binaries are available for:
| Platform | Architecture |
|---|---|
| Linux | amd64, arm64, armv7 |
| macOS | amd64 (Intel), arm64 (Apple Silicon) |
| Windows | amd64, arm64 |
```bash
# Example: Linux amd64
curl -L https://github.com/openlit/openlit/releases/latest/download/opentelemetry-gpu-collector-<version>-linux-amd64 \
-o opentelemetry-gpu-collector
chmod +x opentelemetry-gpu-collector
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector
```
<Note>
GPU device and process metrics work on **Linux and Windows** (NVIDIA via NVML; AMD/Intel via sysfs on Linux or DXGI+PDH on Windows). eBPF CUDA tracing and occupancy are **Linux only**. On macOS the binary runs with host and process metrics only.
</Note>
Verify the SHA256 checksum from the `SHA256SUMS.txt` file in the release:
```bash
sha256sum -c SHA256SUMS.txt --ignore-missing
```
---
## Build from source
Requirements: Go 1.21+, CGO enabled (required for NVML on Linux).
```bash
git clone https://github.com/openlit/openlit.git
cd openlit/opentelemetry-gpu-collector
make build
./opentelemetry-gpu-collector
```
For eBPF CUDA tracing support, also run:
```bash
make setup-bpf # installs bpftool, generates vmlinux.h
make generate # runs bpf2go code generation
make build
```
---
## Kubernetes DaemonSet
Run one collector per GPU node. Use the OpenTelemetry-recommended pattern: downward API → `K8S_NODE_NAME` → `OTEL_RESOURCE_ATTRIBUTES` with `host.name` and `k8s.node.name`. On GKE, AKS, and EKS the collector also auto-detects `k8s.cluster.name`, `cloud.provider`, and `host.type` (instance type) from the Kubernetes Node object and/or cloud metadata.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: otel-gpu-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: otel-gpu-collector-node-get
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: otel-gpu-collector-node-get
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: otel-gpu-collector-node-get
subjects:
- kind: ServiceAccount
name: otel-gpu-collector
namespace: default # change to your namespace
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: otel-gpu-collector
spec:
selector:
matchLabels:
app: otel-gpu-collector
template:
metadata:
labels:
app: otel-gpu-collector
spec:
serviceAccountName: otel-gpu-collector
hostPID: true
containers:
- name: otel-gpu-collector
image: ghcr.io/openlit/otel-gpu-collector:latest
env:
- name: K8S_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: OTEL_SERVICE_NAME
value: otel-gpu-collector
- name: OTEL_RESOURCE_ATTRIBUTES
value: "host.name=$(K8S_NODE_NAME),k8s.node.name=$(K8S_NODE_NAME)"
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://openlit:4318"
# Optional for on-prem / self-managed clusters:
# - name: K8S_CLUSTER_NAME
# value: my-cluster
# eBPF CUDA tracing is on by default on Linux; set false to disable:
# - name: OTEL_GPU_EBPF_ENABLED
# value: "false"
volumeMounts:
- name: pod-resources
mountPath: /var/lib/kubelet/pod-resources
readOnly: true
# AMD/Intel DRM:
# - name: dri
# mountPath: /dev/dri
securityContext:
capabilities:
add: ["SYS_ADMIN"] # or privileged / CAP_BPF+CAP_PERFMON for eBPF
volumes:
- name: pod-resources
hostPath:
path: /var/lib/kubelet/pod-resources
# - name: dri
# hostPath:
# path: /dev/dri
```
<Note>
`hostPID: true` (same role as Docker `--pid=host`) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under `/proc`. Without it, device-level `hw.gpu.*` metrics still work. `nodes/get` enables OpenCost-style `host.type` / `cloud.provider` discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1). `pods/list` is what `OPENLIT_K8S_POD_LOOKUP` uses to turn a process's cgroup pod UID into `k8s.pod.name`. The list call is field-selected to this node; the permission is cluster-scoped.
</Note>
See [Configuration](/latest/gpu-collector/configuration) for identity env vars and detection order.
---
## Upgrade
### Docker
```bash
docker pull ghcr.io/openlit/otel-gpu-collector:latest
docker stop otel-gpu-collector
docker rm otel-gpu-collector
# re-run with same flags
```
### Binary
Download the new binary from the [Releases](https://github.com/openlit/openlit/releases) page, replace the existing file, and restart the process.
---
## Uninstall
### Docker
```bash
docker stop otel-gpu-collector
docker rm otel-gpu-collector
docker rmi ghcr.io/openlit/otel-gpu-collector:latest
```
### Binary
```bash
rm /usr/local/bin/opentelemetry-gpu-collector
```
---
## Troubleshooting
<AccordionGroup>
<Accordion title="No GPU metrics - collector starts but reports no hw.gpu.* metrics">
- Confirm the host has a supported GPU: `lspci | grep -E 'VGA|3D|Display'`
- For NVIDIA: verify `libnvidia-ml.so` is present: `ldconfig -p | grep nvidia-ml`
- For Docker: ensure `--gpus all` (NVIDIA) or `--device /dev/dri` (AMD/Intel) is passed
- Check logs: `docker logs otel-gpu-collector` for `"discovered GPU"` entries
</Accordion>
<Accordion title="eBPF tracing not working">
- On Linux it is enabled by default; confirm it is not disabled via `OTEL_GPU_EBPF_ENABLED=false`
- Check kernel version: `uname -r` (requires 5.8+)
- The process needs `CAP_BPF` and `CAP_PERFMON`, or run as root
- For Docker: `--cap-add CAP_BPF --cap-add CAP_PERFMON`, `--pid=host`, and `--ulimit memlock=-1:-1` (BPF maps need locked memory; no CUDA mount needed)
- For Kubernetes: `hostPID: true` plus BPF capabilities (or `SYS_ADMIN`); raise memlock if map create returns EPERM
- If no CUDA process is running yet, the collector rescans `/proc` every 30s
- Verify a workload has loaded CUDA: `grep -E 'libcudart|libcuda.so' /proc/*/maps 2>/dev/null | head`
</Accordion>
<Accordion title="Connection refused / no data reaching backend">
- Verify `OTEL_EXPORTER_OTLP_ENDPOINT` is reachable from the container: `curl http://<endpoint>/health`
- For Docker networking: use the host IP or service name, not `localhost`
- Check if gRPC vs HTTP/protobuf matches the backend: set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` for HTTP backends (port 4318)
</Accordion>
<Accordion title="Intel GPU not detected">
- Verify the i915 or Xe driver is loaded: `lsmod | grep -E 'i915|xe'`
- Check DRM entries exist: `ls /sys/class/drm/`
- Requires Linux kernel 5.10+ for sysfs metric exposure
- Fan speed requires kernel 6.16+
</Accordion>
</AccordionGroup>