Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

7.9 KB
Raw Blame
---
title: 'Installation'
description: 'Install the OpenTelemetry GPU Collector via Docker, binary, or from source'
icon: 'circle-down'
---

## Docker (recommended)

The easiest way to run the collector. The image is published to GitHub Container Registry and supports `linux/amd64` and `linux/arm64`.

```bash
docker pull ghcr.io/openlit/otel-gpu-collector:latest
```

For per-process GPU attribution (cmdline, PID, zombie/`process.state`, owner), run with host PID namespace access: Docker `--pid=host`, Compose `pid: host`, or Kubernetes `hostPID: true`. Device-level `hw.gpu.*` metrics work without it.

### Tags

| Tag | Description |
|---|---|
| `latest` | Most recent release |
| `1.2.3` | Specific version |
| `1.2` | Latest patch of a minor version |

---

## Pre-built binaries

Download a binary for your platform from the [GitHub Releases](https://github.com/openlit/openlit/releases) page. Binaries are available for:

| Platform | Architecture |
|---|---|
| Linux | amd64, arm64, armv7 |
| macOS | amd64 (Intel), arm64 (Apple Silicon) |
| Windows | amd64, arm64 |

```bash
# Example: Linux amd64
curl -L https://github.com/openlit/openlit/releases/latest/download/opentelemetry-gpu-collector-<version>-linux-amd64 \
    -o opentelemetry-gpu-collector
chmod +x opentelemetry-gpu-collector

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector
```

<Note>
GPU device and process metrics work on **Linux and Windows** (NVIDIA via NVML; AMD/Intel via sysfs on Linux or DXGI+PDH on Windows). eBPF CUDA tracing and occupancy are **Linux only**. On macOS the binary runs with host and process metrics only.
</Note>

Verify the SHA256 checksum from the `SHA256SUMS.txt` file in the release:

```bash
sha256sum -c SHA256SUMS.txt --ignore-missing
```

---

## Build from source

Requirements: Go 1.21+, CGO enabled (required for NVML on Linux).

```bash
git clone https://github.com/openlit/openlit.git
cd openlit/opentelemetry-gpu-collector
make build
./opentelemetry-gpu-collector
```

For eBPF CUDA tracing support, also run:

```bash
make setup-bpf   # installs bpftool, generates vmlinux.h
make generate    # runs bpf2go code generation
make build
```

---

## Kubernetes DaemonSet

Run one collector per GPU node. Use the OpenTelemetry-recommended pattern: downward API → `K8S_NODE_NAME` → `OTEL_RESOURCE_ATTRIBUTES` with `host.name` and `k8s.node.name`. On GKE, AKS, and EKS the collector also auto-detects `k8s.cluster.name`, `cloud.provider`, and `host.type` (instance type) from the Kubernetes Node object and/or cloud metadata.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-gpu-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-gpu-collector-node-get
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-gpu-collector-node-get
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-gpu-collector-node-get
subjects:
  - kind: ServiceAccount
    name: otel-gpu-collector
    namespace: default # change to your namespace
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-gpu-collector
spec:
  selector:
    matchLabels:
      app: otel-gpu-collector
  template:
    metadata:
      labels:
        app: otel-gpu-collector
    spec:
      serviceAccountName: otel-gpu-collector
      hostPID: true
      containers:
        - name: otel-gpu-collector
          image: ghcr.io/openlit/otel-gpu-collector:latest
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: OTEL_SERVICE_NAME
              value: otel-gpu-collector
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "host.name=$(K8S_NODE_NAME),k8s.node.name=$(K8S_NODE_NAME)"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://openlit:4318"
            # Optional for on-prem / self-managed clusters:
            # - name: K8S_CLUSTER_NAME
            #   value: my-cluster
            # eBPF CUDA tracing is on by default on Linux; set false to disable:
            # - name: OTEL_GPU_EBPF_ENABLED
            #   value: "false"
          volumeMounts:
            - name: pod-resources
              mountPath: /var/lib/kubelet/pod-resources
              readOnly: true
            # AMD/Intel DRM:
            # - name: dri
            #   mountPath: /dev/dri
          securityContext:
            capabilities:
              add: ["SYS_ADMIN"] # or privileged / CAP_BPF+CAP_PERFMON for eBPF
      volumes:
        - name: pod-resources
          hostPath:
            path: /var/lib/kubelet/pod-resources
        # - name: dri
        #   hostPath:
        #     path: /dev/dri
```

<Note>
`hostPID: true` (same role as Docker `--pid=host`) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under `/proc`. Without it, device-level `hw.gpu.*` metrics still work. `nodes/get` enables OpenCost-style `host.type` / `cloud.provider` discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1). `pods/list` is what `OPENLIT_K8S_POD_LOOKUP` uses to turn a process's cgroup pod UID into `k8s.pod.name`. The list call is field-selected to this node; the permission is cluster-scoped.
</Note>

See [Configuration](/latest/gpu-collector/configuration) for identity env vars and detection order.

---

## Upgrade

### Docker

```bash
docker pull ghcr.io/openlit/otel-gpu-collector:latest
docker stop otel-gpu-collector
docker rm otel-gpu-collector
# re-run with same flags
```

### Binary

Download the new binary from the [Releases](https://github.com/openlit/openlit/releases) page, replace the existing file, and restart the process.

---

## Uninstall

### Docker

```bash
docker stop otel-gpu-collector
docker rm otel-gpu-collector
docker rmi ghcr.io/openlit/otel-gpu-collector:latest
```

### Binary

```bash
rm /usr/local/bin/opentelemetry-gpu-collector
```

---

## Troubleshooting

<AccordionGroup>
  <Accordion title="No GPU metrics - collector starts but reports no hw.gpu.* metrics">
    - Confirm the host has a supported GPU: `lspci | grep -E 'VGA|3D|Display'`
    - For NVIDIA: verify `libnvidia-ml.so` is present: `ldconfig -p | grep nvidia-ml`
    - For Docker: ensure `--gpus all` (NVIDIA) or `--device /dev/dri` (AMD/Intel) is passed
    - Check logs: `docker logs otel-gpu-collector` for `"discovered GPU"` entries
  </Accordion>
  <Accordion title="eBPF tracing not working">
    - On Linux it is enabled by default; confirm it is not disabled via `OTEL_GPU_EBPF_ENABLED=false`
    - Check kernel version: `uname -r` (requires 5.8+)
    - The process needs `CAP_BPF` and `CAP_PERFMON`, or run as root
    - For Docker: `--cap-add CAP_BPF --cap-add CAP_PERFMON`, `--pid=host`, and `--ulimit memlock=-1:-1` (BPF maps need locked memory; no CUDA mount needed)
    - For Kubernetes: `hostPID: true` plus BPF capabilities (or `SYS_ADMIN`); raise memlock if map create returns EPERM
    - If no CUDA process is running yet, the collector rescans `/proc` every 30s
    - Verify a workload has loaded CUDA: `grep -E 'libcudart|libcuda.so' /proc/*/maps 2>/dev/null | head`
  </Accordion>
  <Accordion title="Connection refused / no data reaching backend">
    - Verify `OTEL_EXPORTER_OTLP_ENDPOINT` is reachable from the container: `curl http://<endpoint>/health`
    - For Docker networking: use the host IP or service name, not `localhost`
    - Check if gRPC vs HTTP/protobuf matches the backend: set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` for HTTP backends (port 4318)
  </Accordion>
  <Accordion title="Intel GPU not detected">
    - Verify the i915 or Xe driver is loaded: `lsmod | grep -E 'i915|xe'`
    - Check DRM entries exist: `ls /sys/class/drm/`
    - Requires Linux kernel 5.10+ for sysfs metric exposure
    - Fan speed requires kernel 6.16+
  </Accordion>
</AccordionGroup>