Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.
TypeScript agent-harnessagent-harness-engineeringai-engineeringai-observabilityamd-gpuclaude-codeclickhousecodexcoding-agentcoding-agentscursorgenaigpu-monitoringllm-observabilityllmopsnvidia-smiobservabilityopen-sourceopenaiopentelemetry
fix(gpu-collector): attribute GPU processes to their own pod, not to the GPU owner (#1719)
Hoang Nguyen4605c7fe4c00
diff --git a/docs/latest/gpu-collector/configuration.mdx b/docs/latest/gpu-collector/configuration.mdxindex 7bda7fe6..5d3ddee3 100644--- a/docs/latest/gpu-collector/configuration.mdx+++ b/docs/latest/gpu-collector/configuration.mdx@@ -34,8 +34,8 @@ The collector is configured entirely via environment variables. All variables fo| `K8S_NODE_NAME` | | Kubernetes node name via downward API (`spec.nodeName`). Also accepts `OTEL_RESOURCE_ATTRIBUTES_NODE_NAME` (Operator) or legacy `NODE_NAME` || `K8S_CLUSTER_NAME` | | Explicit cluster name when cloud auto-detect fails (on-prem). Alias: `OPENLIT_K8S_CLUSTER_NAME`. Only applied in Kubernetes || `OPENLIT_K8S_NODE_LOOKUP` | `true` | When `false`, skip `GET /api/v1/nodes/$K8S_NODE_NAME` for instance-type / provider discovery |-| `OPENLIT_K8S_POD_RESOURCES` | `true` in K8s | Use kubelet PodResources socket; joins GPU UUID → `k8s.pod.name` / namespace / container |-| `OPENLIT_K8S_POD_LOOKUP` | `true` in K8s when `K8S_NODE_NAME` is set | List pods on this node via the Kubernetes API (needs `list` on pods) for UID/container-id joins |+| `OPENLIT_K8S_POD_RESOURCES` | `true` in K8s | Kubelet PodResources socket. Names a process only when it has no pod of its own, or the allocation is confirmed as its pod. See below |+| `OPENLIT_K8S_POD_LOOKUP` | `true` in K8s when `K8S_NODE_NAME` is set | List pods on this node (`list` on pods). Resolves a containerized process's cgroup UID to its own `k8s.pod.name` and namespace || `POD_RESOURCES_SOCKET` | OS default | Override kubelet PodResources socket / named pipe path || `OPENLIT_CLOUD_DETECT` | `true` | When `false`, skip AWS/GCP/Azure IMDS probes (recommended on bare metal to avoid link-local timeouts) || `OTEL_GPU_DCGM_ENABLED` | mode-dependent (`true` for `serving`+) | Optional NVIDIA DCGM profiling (soft-skipped without libdcgm / when DCGM cannot init) |@@ -64,6 +64,12 @@ The collector is configured entirely via environment variables. All variables fo| `OTEL_HOST_NIC_ENABLED` | mode-dependent (`true` for `training`+) | Per-NIC `hw.network.*` (ethtool/sysfs). When enabled, `system.network.*` is skipped to avoid duplicates || `OTEL_HOST_RDMA_ENABLED` | mode-dependent (`true` for `training`+) | RDMA device counters |+<Note>+**Pod name on `process.gpu.*`.** `OPENLIT_K8S_POD_LOOKUP` joins the process cgroup UID to that pod's name and namespace. It needs `list` on pods (see the DaemonSet role below). The list call is field-selected to this node; the permission itself is cluster-scoped.++Without that list, kubelet PodResources supplies a name for a host process (the owner of the GPU being scraped) or when a container's `/proc/<pid>/root/etc/hostname` confirms an allocation. The kubelet writes the pod name there. `setHostnameAsFQDN` counts only when the owner's namespace is in the name (`<pod>.<subdomain>.<namespace>.svc.<cluster>`). `hostNetwork` sees the node hostname, and `spec.hostname` sees the override, so those stay UID-only unless that hostname still names the owner. Two pods with the same name in different namespaces are left UID-only. Confirmation looks at every GPU allocation on the node, so the name does not depend on which GPU is scraped first. `k8s.pod.uid` stays the cgroup value either way.+</Note>+## Host, Kubernetes, and cloud identityResource attributes follow [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/) for host, K8s, and cloud:@@ -113,7 +119,7 @@ env:If you only set `K8S_NODE_NAME` (without packing it into `OTEL_RESOURCE_ATTRIBUTES`), the collector still maps it to `host.name` and `k8s.node.name` automatically.-For K8s node label / providerID discovery, grant the DaemonSet ServiceAccount:+For node label / providerID discovery and for resolving `k8s.pod.name`, grant the DaemonSet ServiceAccount:```yamlapiVersion: rbac.authorization.k8s.io/v1@@ -124,6 +130,9 @@ rules:- apiGroups: [""]resources: ["nodes"]verbs: ["get"]+ - apiGroups: [""]+ resources: ["pods"]+ verbs: ["list"]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingdiff --git a/docs/latest/gpu-collector/installation.mdx b/docs/latest/gpu-collector/installation.mdxindex c2ef891f..d8885659 100644--- a/docs/latest/gpu-collector/installation.mdx+++ b/docs/latest/gpu-collector/installation.mdx@@ -95,6 +95,9 @@ rules:- apiGroups: [""]resources: ["nodes"]verbs: ["get"]+ - apiGroups: [""]+ resources: ["pods"]+ verbs: ["list"]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBinding@@ -164,7 +167,7 @@ spec:```<Note>-`hostPID: true` (same role as Docker `--pid=host`) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under `/proc`. Without it, device-level `hw.gpu.*` metrics still work. The `nodes/get` ClusterRole enables OpenCost-style `host.type` / `cloud.provider` discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1).+`hostPID: true` (same role as Docker `--pid=host`) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under `/proc`. Without it, device-level `hw.gpu.*` metrics still work. `nodes/get` enables OpenCost-style `host.type` / `cloud.provider` discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1). `pods/list` is what `OPENLIT_K8S_POD_LOOKUP` uses to turn a process's cgroup pod UID into `k8s.pod.name`. The list call is field-selected to this node; the permission is cluster-scoped.</Note>See [Configuration](/latest/gpu-collector/configuration) for identity env vars and detection order.diff --git a/docs/latest/gpu-collector/metrics.mdx b/docs/latest/gpu-collector/metrics.mdxindex 9501f487..a4326e57 100644--- a/docs/latest/gpu-collector/metrics.mdx+++ b/docs/latest/gpu-collector/metrics.mdx@@ -151,11 +151,13 @@ These attributes support views like zombie-process triage: filter `process.state| `hw.id`, `hw.name`, `hw.vendor`, `gpu.index`, `gpu.pci_address` | Same device attrs as `hw.gpu.*` || `hw.gpu.task` | On utilization only: `general`, `encoder`, `decoder` || `k8s.pod.uid` | From cgroup path when present |-| `k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name` | Via kubelet PodResources (GPU device join) and/or pod API lookup |+| `k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name` | The process's own pod (cgroup container id / pod UID joined with the pod API). PodResources is a fallback: per device for host processes, and for a containerized process only when an allocation is confirmed as that pod || `container.id` | Runtime container ID when resolvable |<Note>`process.start_time` is not exported as a metric attribute (cardinality). Use `process.uptime` for lifetime. On MIG devices, NVML often provides **memory only** (no per-process SM util). Process util may not sum to device util (time-slicing / sampling windows differ). Use `process.state == zombie` to find defunct processes still holding GPU contexts.++PodResources carries no pod UID. A containerized process is confirmed by its own `/etc/hostname` (the kubelet writes the pod name; with `setHostnameAsFQDN`, the owner's namespace must appear in it) matched against allocations on the node, not against whichever GPU is scraped first. `hostNetwork` uses the node hostname. `spec.hostname` uses the override. Those stay UID-only unless the hostname still names the owner. Two pods with the same name in different namespaces stay UID-only unless `OPENLIT_K8S_POD_LOOKUP` can list pods. A process on GPUs allocated to other pods keeps its own pod name. `k8s.pod.uid` is the cgroup value.</Note>---diff --git a/opentelemetry-gpu-collector/internal/workload/enricher.go b/opentelemetry-gpu-collector/internal/workload/enricher.goindex bffd94b5..3e80378c 100644--- a/opentelemetry-gpu-collector/internal/workload/enricher.go+++ b/opentelemetry-gpu-collector/internal/workload/enricher.go@@ -13,20 +13,29 @@ import (// Enricher resolves PID → pod identity using cgroup + optional PodResources + API.// Outside Kubernetes (or when both enrichment flags are off) Resolve is a free no-op.type Enricher struct {- logger *slog.Logger- cfg *config.Config+ logger *slog.Logger+ cfg *config.Configenabled bool- mu sync.RWMutex- cache map[int32]PodInfo- negCache map[int32]time.Time // PID → when negative miss was cached- cacheAt time.Time- ttl time.Duration- byUID map[string]PodInfo- byName map[string]PodInfo- byContID map[string]PodInfo- byPID map[int32]PodInfo- byDevice map[string]PodInfo // GPU UUID / device id → pod+ // resolveCgroup reads the process's own pod UID / container id (ResolvePod); injectable for tests.+ resolveCgroup func(pid int32) (PodInfo, bool)+ // resolveHostname reads the process's own /etc/hostname (ProcHostname); injectable for tests.+ resolveHostname func(pid int32) string++ mu sync.RWMutex+ own map[int32]PodInfo // PID → identity from the process itself; valid on every device+ // cache / negCache hold device-dependent results (GPU-owner fallback, partial+ // identities, misses), keyed per (pid, device).+ cache map[cacheKey]PodInfo+ negCache map[cacheKey]time.Time // when the negative miss was cached+ hostnames map[int32]string // PID → container hostname memo ("" when unreadable)+ cacheAt time.Time+ ttl time.Duration+ byUID map[string]PodInfo+ byName map[string]PodInfo+ byContID map[string]PodInfo+ byPID map[int32]PodInfo+ byDevice map[string]PodInfo // GPU UUID / device id → podrefreshMu sync.MutexstopCh chan struct{}@@ -40,6 +49,14 @@ type Enricher struct {connectInterval time.Duration}+// cacheKey scopes a resolution that depended on the GPU it was asked for: a process can+// hold contexts on GPUs allocated to other pods, so whether the PodResources owner of a+// GPU applies (or whether there is one at all) differs per device.+type cacheKey struct {+ pid int32+ deviceID string+}+// NewEnricher builds a workload enricher. Safe on non-K8s (no-op enrichment).func NewEnricher(cfg *config.Config, logger *slog.Logger) *Enricher {if cfg == nil {@@ -51,8 +68,12 @@ func NewEnricher(cfg *config.Config, logger *slog.Logger) *Enricher {logger: logger,cfg: cfg,enabled: enabled,- cache: make(map[int32]PodInfo),- negCache: make(map[int32]time.Time),+ resolveCgroup: ResolvePod,+ resolveHostname: ProcHostname,+ own: make(map[int32]PodInfo),+ cache: make(map[cacheKey]PodInfo),+ negCache: make(map[cacheKey]time.Time),+ hostnames: make(map[int32]string),byUID: make(map[string]PodInfo),byName: make(map[string]PodInfo),byContID: make(map[string]PodInfo),@@ -121,17 +142,36 @@ func (e *Enricher) ensureClients() {// Resolve returns the best-known pod identity for pid.// deviceID is an optional GPU UUID used to join PodResources device allocations.+//+// Identity comes from the process itself first (cgroup → container id / pod UID →+// kubelet PodResources and pod API maps). The pod that PodResources allocated a GPU+// to is only a fallback when the process has no pod name of its own: a process may+// hold a context on a GPU allocated to a different pod (privileged pods,+// NVIDIA_VISIBLE_DEVICES=all, host processes), and the GPU owner is not that process's+// pod. A host process takes the owner of the device being scraped. A process with a+// cgroup pod UID takes an owner only when that owner is confirmed as the same pod,+// by UID or (PodResources carries no UID) by the process hostname matched against+// every allocation on the node. The metrics callback emits GPUs in index order, so+// confirmation cannot depend on the device passed in. An unconfirmed or ambiguous+// owner leaves the process UID-only. Identity resolved from or confirmed against the+// process itself is cached per pid; device-dependent fallback, partial identity, and+// misses are cached per (pid, device).func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {if e == nil || !e.enabled || pid <= 0 {return PodInfo{}, false}+ key := cacheKey{pid: pid, deviceID: deviceID}e.mu.RLock()- if info, ok := e.cache[pid]; ok {+ if info, ok := e.own[pid]; ok {e.mu.RUnlock()- return info, info.PodUID != "" || info.PodName != ""+ return info, true}- if at, ok := e.negCache[pid]; ok && time.Since(at) < e.ttl {+ if info, ok := e.cache[key]; ok {+ e.mu.RUnlock()+ return info, true+ }+ if at, ok := e.negCache[key]; ok && time.Since(at) < e.ttl {e.mu.RUnlock()return PodInfo{}, false}@@ -141,8 +181,12 @@ func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {byDev := e.byDevicee.mu.RUnlock()+ resolveCgroup := e.resolveCgroup+ if resolveCgroup == nil {+ resolveCgroup = ResolvePod+ }info := byPID- if cg, ok := ResolvePod(pid); ok {+ if cg, ok := resolveCgroup(pid); ok {if info.PodUID == "" {info.PodUID = cg.PodUID}@@ -150,13 +194,7 @@ func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {info.ContainerID = cg.ContainerID}}- // Device allocation from PodResources (works without Pod UID).- if deviceID != "" {- if p, ok := byDev[deviceID]; ok {- mergePod(&info, p)- }- }- // Prefer container ID for multi-container pods.+ // 1. The process's own pod: container id (multi-container pods) then pod UID.if info.ContainerID != "" {if p, ok := byCont[info.ContainerID]; ok {mergePod(&info, p)@@ -167,19 +205,116 @@ func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {mergePod(&info, p)}}+ ownIdentity := info.PodName != ""+ // 2. Fallback to a PodResources GPU owner.+ // Host process: the owner of this device only, cached per device.+ // Cgroup UID and no name yet: one confirmation against every allocation.+ // That result is the process's own pod, so the first GPU in a scrape already+ // carries it. The hostname file is read only when some owner has no UID.+ if !ownIdentity && info.PodUID == "" && deviceID != "" {+ if p, ok := byDev[deviceID]; ok {+ mergePod(&info, p)+ }+ } else if !ownIdentity && info.PodUID != "" {+ if p, ok := e.confirmProcessOwner(byDev, info.PodUID, pid); ok {+ mergePod(&info, p)+ ownIdentity = true+ }+ }hit := info.PodUID != "" || info.PodName != ""e.mu.Lock()- if hit {- e.cache[pid] = info- delete(e.negCache, pid)- } else {- e.negCache[pid] = time.Now()+ switch {+ case ownIdentity:+ e.own[pid] = info+ case hit:+ e.cache[key] = info+ delete(e.negCache, key)+ default:+ e.negCache[key] = time.Now()}e.mu.Unlock()return info, hit}+// confirmProcessOwner reports the single PodResources pod that is this process.+// A UID match wins and does not consult the hostname. Otherwise the process hostname+// is matched against owners that carry no UID. Two different name/namespace pairs+// are ambiguous: a same-named pod in another namespace must not be attached.+func (e *Enricher) confirmProcessOwner(byDev map[string]PodInfo, podUID string, pid int32) (PodInfo, bool) {+ if p, ok, ambiguous := collectOwners(byDev, func(p PodInfo) bool {+ return podUID != "" && p.PodUID != "" && p.PodUID == podUID+ }); ambiguous {+ return PodInfo{}, false+ } else if ok {+ return p, true+ }+ if !ownerNeedsHostname(byDev) {+ return PodInfo{}, false+ }+ hostname := e.hostnameOf(pid)+ p, ok, ambiguous := collectOwners(byDev, func(p PodInfo) bool {+ return p.PodUID == "" && hostnameMatchesPod(hostname, p.PodName, p.Namespace)+ })+ if ambiguous || !ok {+ return PodInfo{}, false+ }+ return p, true+}++// collectOwners returns the one pod wanted by match. A second pod with a different+// name or namespace is ambiguous. Several allocations of the same pod keep the pod+// and drop ContainerName when the containers disagree.+func collectOwners(byDev map[string]PodInfo, match func(PodInfo) bool) (info PodInfo, ok bool, ambiguous bool) {+ for _, p := range byDev {+ if p.PodName == "" || !match(p) {+ continue+ }+ if !ok {+ info, ok = p, true+ continue+ }+ if p.Namespace != info.Namespace || p.PodName != info.PodName {+ return PodInfo{}, false, true+ }+ if p.ContainerName != info.ContainerName {+ info.ContainerName = ""+ }+ }+ return info, ok, false+}++func ownerNeedsHostname(byDev map[string]PodInfo) bool {+ for _, p := range byDev {+ if p.PodUID == "" && p.PodName != "" {+ return true+ }+ }+ return false+}++// hostnameOf returns the process's container hostname, read once per pid per refresh.+func (e *Enricher) hostnameOf(pid int32) string {+ e.mu.RLock()+ h, ok := e.hostnames[pid]+ e.mu.RUnlock()+ if ok {+ return h+ }+ read := e.resolveHostname+ if read == nil {+ read = ProcHostname+ }+ h = read(pid)+ e.mu.Lock()+ if e.hostnames == nil {+ e.hostnames = make(map[int32]string)+ }+ e.hostnames[pid] = h+ e.mu.Unlock()+ return h+}+func mergePod(dst *PodInfo, src PodInfo) {if dst.PodUID == "" {dst.PodUID = src.PodUID@@ -246,9 +381,11 @@ func (e *Enricher) refreshMaps() {e.mu.Lock()defer e.mu.Unlock()+ e.hostnames = make(map[int32]string) // PIDs may be reused; re-read next cycleif indexed {- e.cache = make(map[int32]PodInfo)- e.negCache = make(map[int32]time.Time)+ e.own = make(map[int32]PodInfo)+ e.cache = make(map[cacheKey]PodInfo)+ e.negCache = make(map[cacheKey]time.Time)e.byUID = byUIDe.byName = byNamee.byContID = byContIDdiff --git a/opentelemetry-gpu-collector/internal/workload/enricher_test.go b/opentelemetry-gpu-collector/internal/workload/enricher_test.goindex c1e6d29d..141458ed 100644--- a/opentelemetry-gpu-collector/internal/workload/enricher_test.go+++ b/opentelemetry-gpu-collector/internal/workload/enricher_test.go@@ -1,6 +1,9 @@package workload-import "testing"+import (+ "testing"+ "time"+)func TestIndexPodSeparatesContainerFromUID(t *testing.T) {byUID := map[string]PodInfo{}@@ -16,7 +19,7 @@ func TestIndexPodSeparatesContainerFromUID(t *testing.T) {indexPod(byUID, byName, byCont, byPID, byDev, PodInfo{PodUID: "uid-1", PodName: "p", Namespace: "ns",ContainerName: "app", ContainerID: "bbb",- DeviceIDs: []string{"GPU-1"},+ DeviceIDs: []string{"GPU-1"},})if byUID["uid-1"].ContainerName != "" || byUID["uid-1"].ContainerID != "" {@@ -62,3 +65,352 @@ func TestMergePodPrefersExistingContainer(t *testing.T) {t.Fatalf("merge missed pod fields: %+v", info)}}++func newTestEnricher(cgroup func(int32) (PodInfo, bool)) *Enricher {+ return &Enricher{+ enabled: true,+ resolveCgroup: cgroup,+ resolveHostname: func(int32) string { return "" }, // unreadable unless a test sets it+ own: make(map[int32]PodInfo),+ cache: make(map[cacheKey]PodInfo),+ negCache: make(map[cacheKey]time.Time),+ hostnames: make(map[int32]string),+ byUID: make(map[string]PodInfo),+ byName: make(map[string]PodInfo),+ byContID: make(map[string]PodInfo),+ byPID: make(map[int32]PodInfo),+ byDevice: make(map[string]PodInfo),+ ttl: 15 * time.Second,+ }+}++var (+ gemma = PodInfo{PodUID: "uid-gemma", PodName: "gemma-worker-0", Namespace: "llms", ContainerName: "main"}+ qwen = PodInfo{PodUID: "uid-qwen", PodName: "qwen-worker-0", Namespace: "llms-dev", ContainerName: "main"}+ lmcache = PodInfo{PodUID: "uid-lmcache", PodName: "lmcache-server", Namespace: "cache", ContainerName: "main"}+)++// A privileged pod (NVIDIA_VISIBLE_DEVICES=all) holding contexts on every GPU of the+// node must keep its own identity on GPUs allocated to other pods.+func TestResolvePrefersCgroupIdentityOverDeviceOwner(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-lmcache", ContainerID: "cid-lmcache"}, true+ })+ e.byUID["uid-lmcache"] = lmcache+ e.byDevice["GPU-3"] = gemma+ e.byDevice["GPU-0"] = qwen++ got, ok := e.Resolve(42, "GPU-3")+ if !ok || got.PodName != "lmcache-server" || got.Namespace != "cache" || got.PodUID != "uid-lmcache" {+ t.Fatalf("expected cgroup pod, got %+v", got)+ }+ // Cached per pid: same answer on another GPU owned by another pod.+ got, _ = e.Resolve(42, "GPU-0")+ if got.PodName != "lmcache-server" {+ t.Fatalf("cached identity changed with device: %+v", got)+ }+}++func TestResolveContainerIDBeatsDeviceOwner(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-qwen", ContainerID: "cid-qwen"}, true+ })+ e.byContID["cid-qwen"] = PodInfo{PodUID: "uid-qwen", PodName: "qwen-worker-0", Namespace: "llms-dev", ContainerName: "main", ContainerID: "cid-qwen"}+ e.byDevice["GPU-3"] = gemma++ got, _ := e.Resolve(7, "GPU-3")+ if got.PodName != "qwen-worker-0" || got.ContainerName != "main" || got.ContainerID != "cid-qwen" {+ t.Fatalf("expected container-id join, got %+v", got)+ }+}++// PodResources only (no pod API): the process has a cgroup pod UID but no name for it,+// and the GPU owner carries no UID. With no hostname to confirm the owner (unreadable),+// or a hostname naming another pod (lmcache on a GPU allocated to gemma: the #1718+// mislabel), the owner's name must not be attached next to a UID that may belong to+// another pod. Identity stays UID-only, on every device.+func TestResolveUnconfirmedDeviceOwnerNotAppliedToCgroupProcess(t *testing.T) {+ for name, hostname := range map[string]string{"unreadable": "", "other pod": "lmcache-server"} {+ t.Run(name, func(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-lmcache"}, true+ })+ e.resolveHostname = func(int32) string { return hostname }+ owner := func(p PodInfo) PodInfo { p.PodUID = ""; return p } // PodResources carries no UID+ e.byDevice["GPU-3"] = owner(gemma)+ e.byDevice["GPU-0"] = owner(qwen)++ for _, dev := range []string{"GPU-3", "GPU-0", "GPU-3"} {+ got, ok := e.Resolve(9, dev)+ if !ok || got.PodUID != "uid-lmcache" {+ t.Fatalf("%s: expected cgroup uid, got %+v ok=%v", dev, got, ok)+ }+ if got.PodName != "" || got.Namespace != "" || got.ContainerName != "" {+ t.Fatalf("%s: borrowed unconfirmed owner labels: %+v", dev, got)+ }+ }+ })+ }+}++// PodResources only, common case: a pod's own process on the GPU allocated to it. The+// owner carries no UID, but the container hostname is the pod name, which confirms it.+// The confirmed identity is the process's own: cached per pid, hostname read once, and+// kept on a GPU allocated to another pod.+func TestResolveDeviceOwnerConfirmedByHostname(t *testing.T) {+ for name, hostname := range map[string]string{+ "pod name": "gemma-worker-0",+ "fqdn": "gemma-worker-0.gemma.llms.svc.cluster.local",+ } {+ t.Run(name, func(t *testing.T) {+ reads := 0+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-gemma", ContainerID: "cid-gemma"}, true+ })+ e.resolveHostname = func(int32) string { reads++; return hostname }+ owner := func(p PodInfo) PodInfo { p.PodUID = ""; return p }+ e.byDevice["GPU-3"] = owner(gemma)+ e.byDevice["GPU-0"] = owner(qwen)++ got, ok := e.Resolve(21, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.ContainerName != "main" {+ t.Fatalf("expected hostname-confirmed owner, got %+v", got)+ }+ if got.PodUID != "uid-gemma" || got.ContainerID != "cid-gemma" {+ t.Fatalf("cgroup identity lost: %+v", got)+ }+ got, _ = e.Resolve(21, "GPU-0")+ if got.PodName != "gemma-worker-0" {+ t.Fatalf("confirmed identity must follow the process to other GPUs, got %+v", got)+ }+ if reads != 1 {+ t.Fatalf("hostname should be read once per pid, got %d", reads)+ }+ })+ }+}++// Confirmation searches every allocation. The GPU passed to Resolve is often a device+// this process does not own, and the metrics callback emits that device first.+func TestResolveHostnameConfirmIgnoresScrapeOrder(t *testing.T) {+ reads := 0+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-qwen"}, true+ })+ e.resolveHostname = func(int32) string { reads++; return "qwen-worker-0" }+ owner := func(p PodInfo) PodInfo { p.PodUID = ""; return p }+ e.byDevice["GPU-3"] = owner(gemma)+ e.byDevice["GPU-0"] = owner(qwen)++ got, ok := e.Resolve(22, "GPU-9")+ if !ok || got.PodName != "qwen-worker-0" || got.Namespace != "llms-dev" || got.PodUID != "uid-qwen" {+ t.Fatalf("name must be present on a GPU this pod does not own, got %+v ok=%v", got, ok)+ }+ got, _ = e.Resolve(22, "GPU-3")+ if got.PodName != "qwen-worker-0" {+ t.Fatalf("confirmed identity must follow the process, got %+v", got)+ }+ if reads != 1 {+ t.Fatalf("hostname should be read once per pid, got %d", reads)+ }+}++// Same pod name in two namespaces is not proof. Leave the process UID-only.+func TestResolveAmbiguousHostnameStaysUIDOnly(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-worker"}, true+ })+ e.resolveHostname = func(int32) string { return "worker" }+ owner := func(ns string) PodInfo {+ return PodInfo{PodName: "worker", Namespace: ns, ContainerName: "main"}+ }+ e.byDevice["GPU-0"] = owner("a")+ e.byDevice["GPU-1"] = owner("b")++ for _, dev := range []string{"GPU-0", "GPU-1"} {+ got, ok := e.Resolve(24, dev)+ if !ok || got.PodUID != "uid-worker" || got.PodName != "" || got.Namespace != "" {+ t.Fatalf("%s: ambiguous hostname must stay uid-only, got %+v", dev, got)+ }+ }+}++// setHostnameAsFQDN includes the namespace, so the other same-named pod is not a candidate.+func TestResolveFQDNHostnameSelectsNamespace(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-gemma"}, true+ })+ e.resolveHostname = func(int32) string { return "gemma-worker-0.gemma.llms.svc.cluster.local" }+ strip := func(p PodInfo, ns string) PodInfo {+ p.PodUID = ""+ p.Namespace = ns+ return p+ }+ e.byDevice["GPU-3"] = strip(gemma, "other")+ e.byDevice["GPU-0"] = strip(gemma, "llms")++ got, ok := e.Resolve(25, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.ContainerName != "main" {+ t.Fatalf("expected the namespace named by the FQDN, got %+v ok=%v", got, ok)+ }+}++// Two containers of the confirmed pod must not contribute an arbitrary container name.+func TestResolveConfirmedPodDropsDisagreeingContainerName(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-gemma"}, true+ })+ e.resolveHostname = func(int32) string { return "gemma-worker-0" }+ e.byDevice["GPU-0"] = PodInfo{PodName: "gemma-worker-0", Namespace: "llms", ContainerName: "main"}+ e.byDevice["GPU-1"] = PodInfo{PodName: "gemma-worker-0", Namespace: "llms", ContainerName: "sidecar"}++ got, ok := e.Resolve(26, "GPU-1")+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.ContainerName != "" {+ t.Fatalf("expected pod without a guessed container, got %+v ok=%v", got, ok)+ }+}++// Hostname confirmation is only for processes with a cgroup pod UID: a host process+// (node hostname) keeps the per-device owner fallback.+func TestResolveHostProcessIgnoresHostname(t *testing.T) {+ reads := 0+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })+ e.resolveHostname = func(int32) string { reads++; return "gpu-node-7" }+ e.byDevice["GPU-3"] = gemma++ got, ok := e.Resolve(23, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" {+ t.Fatalf("host process should still fall back to the GPU owner, got %+v", got)+ }+ if reads != 0 {+ t.Fatalf("host process must not read a container hostname, got %d reads", reads)+ }+}++// The GPU owner is applied to a process with a cgroup pod UID only when the owner's UID+// confirms it is the same pod.+func TestResolveDeviceOwnerConfirmedByUID(t *testing.T) {+ reads := 0+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-gemma"}, true+ })+ e.resolveHostname = func(int32) string { reads++; return "gemma-worker-0" }+ e.byDevice["GPU-3"] = gemma++ got, ok := e.Resolve(10, "GPU-5")+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.PodUID != "uid-gemma" {+ t.Fatalf("expected confirmed owner, got %+v", got)+ }+ if reads != 0 {+ t.Fatalf("a UID match must not read the container hostname, got %d", reads)+ }+}++// A process without pod identity of its own (host process) falls back to the GPU owner,+// but that is a guess per device: seen on two GPUs it is not pinned to the owner of the+// first one.+func TestResolveDeviceOwnerFallbackIsPerDevice(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })+ e.byDevice["GPU-3"] = gemma+ e.byDevice["GPU-0"] = qwen++ got, ok := e.Resolve(9, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" {+ t.Fatalf("expected device-owner fallback, got %+v", got)+ }+ got, _ = e.Resolve(9, "GPU-0")+ if got.PodName != "qwen-worker-0" || got.Namespace != "llms-dev" {+ t.Fatalf("fallback must be per device, got %+v", got)+ }+ got, _ = e.Resolve(9, "GPU-3")+ if got.PodName != "gemma-worker-0" {+ t.Fatalf("per-device fallback not reproducible, got %+v", got)+ }+}++// A miss on a GPU without an owner must not be cached per PID: the same process on a+// GPU that has an owner still reaches the fallback.+func TestResolveMissOnOneDeviceDoesNotHideOwnerOnAnother(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })+ e.byDevice["GPU-3"] = gemma++ if got, ok := e.Resolve(13, "GPU-5"); ok {+ t.Fatalf("no owner for GPU-5: expected miss, got %+v", got)+ }+ got, ok := e.Resolve(13, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" {+ t.Fatalf("negative cache leaked across devices, got %+v ok=%v", got, ok)+ }+ if got, ok := e.Resolve(13, "GPU-5"); ok {+ t.Fatalf("GPU-5 still has no owner: expected miss, got %+v", got)+ }+}++// A UID-only result cached for a device that had no confirming owner must not stick+// to the pid. A later device whose owner confirms the UID still supplies the name,+// including on the device that was cached first.+func TestResolvePartialIdentityDoesNotHideOwnerOnAnotherDevice(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-gemma"}, true+ })++ got, ok := e.Resolve(14, "GPU-5")+ if !ok || got.PodUID != "uid-gemma" || got.PodName != "" {+ t.Fatalf("expected uid-only partial on GPU-5, got %+v ok=%v", got, ok)+ }+ e.byDevice["GPU-3"] = gemma+ got, ok = e.Resolve(14, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" {+ t.Fatalf("partial identity cached per pid hid the owner, got %+v", got)+ }+ got, _ = e.Resolve(14, "GPU-5")+ if got.PodName != "gemma-worker-0" {+ t.Fatalf("confirmed identity must replace the earlier uid-only series, got %+v", got)+ }+}++// Identity resolved from the process itself is cached once and reused on every device.+func TestResolveOwnIdentityCachedAcrossDevices(t *testing.T) {+ calls := 0+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ calls+++ return PodInfo{PodUID: "uid-lmcache"}, true+ })+ e.byUID["uid-lmcache"] = lmcache+ e.byDevice["GPU-3"] = gemma++ for _, dev := range []string{"GPU-3", "GPU-0", "GPU-1"} {+ got, _ := e.Resolve(15, dev)+ if got.PodName != "lmcache-server" {+ t.Fatalf("%s: expected own pod, got %+v", dev, got)+ }+ }+ if calls != 1 {+ t.Fatalf("own identity should be resolved once per pid, cgroup read %d times", calls)+ }+}++func TestResolveNoDeviceOwnerWhenUIDDiffers(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) {+ return PodInfo{PodUID: "uid-lmcache"}, true+ })+ e.byDevice["GPU-3"] = gemma // owner UID known (pod API) and different++ got, ok := e.Resolve(11, "GPU-3")+ if !ok || got.PodUID != "uid-lmcache" || got.PodName != "" {+ t.Fatalf("must not borrow the owner's name, got %+v", got)+ }+}++func TestResolveHostProcessUsesDeviceOwner(t *testing.T) {+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })+ e.byDevice["GPU-3"] = gemma++ got, ok := e.Resolve(13, "GPU-3")+ if !ok || got.PodName != "gemma-worker-0" {+ t.Fatalf("process without cgroup identity should fall back to the GPU owner, got %+v", got)+ }+ if _, ok := e.Resolve(13, "GPU-5"); ok {+ t.Fatal("no owner for GPU-5: expected miss")+ }+}diff --git a/opentelemetry-gpu-collector/internal/workload/hostname.go b/opentelemetry-gpu-collector/internal/workload/hostname.gonew file mode 100644index 00000000..7809f3bd--- /dev/null+++ b/opentelemetry-gpu-collector/internal/workload/hostname.go@@ -0,0 +1,62 @@+package workload++import (+ "io"+ "strconv"+ "strings"+)++// maxHostnameBytes bounds the read of a container's /etc/hostname (a pod name is ≤ 253 bytes).+const maxHostnameBytes = 4096++// ProcHostname returns the hostname of the process's own mount namespace, read from+// /proc/<pid>/root/etc/hostname. In Kubernetes the kubelet writes the pod name there+// (unless spec.hostname / setHostnameAsFQDN / hostNetwork change it), which lets the+// enricher confirm a kubelet PodResources owner, who carries no pod UID, against the+// process itself. Empty when unreadable (no hostPID, no ptrace access, non-Linux).+func ProcHostname(pid int32) string {+ if pid <= 0 {+ return ""+ }+ return readHostnameFile("/proc/" + strconv.FormatInt(int64(pid), 10) + "/root/etc/hostname")+}++func readHostnameFile(path string) string {+ // openHostname refuses symlinks, directories, and FIFOs. A container can+ // replace /etc/hostname with a FIFO; a blocking open here stalls the single+ // metrics callback that walks every GPU.+ f, err := openHostname(path)+ if err != nil {+ return ""+ }+ defer f.Close()+ data, err := io.ReadAll(io.LimitReader(f, maxHostnameBytes))+ if err != nil {+ return ""+ }+ s := string(data)+ if i := strings.IndexByte(s, '\n'); i >= 0 {+ s = s[:i]+ }+ return strings.TrimSpace(s)+}++// hostnameMatchesPod reports whether a container hostname names podName in namespace.+// The kubelet default writes the pod name. setHostnameAsFQDN writes+// <pod>.<subdomain>.<namespace>.svc.<cluster>; a dotted name counts only when+// namespace appears in that position, so pod "api" in namespace dev does not+// confirm hostname "api.web.prod.svc.cluster.local".+func hostnameMatchesPod(hostname, podName, namespace string) bool {+ if hostname == "" || podName == "" {+ return false+ }+ if hostname == podName {+ return true+ }+ if namespace == "" || !strings.HasPrefix(hostname, podName+".") {+ return false+ }+ rest := hostname[len(podName)+1:]+ needle := namespace + ".svc."+ return strings.HasPrefix(rest, needle) || strings.Contains(rest, "."+needle)+}diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_linux.go b/opentelemetry-gpu-collector/internal/workload/hostname_linux.gonew file mode 100644index 00000000..d7047881--- /dev/null+++ b/opentelemetry-gpu-collector/internal/workload/hostname_linux.go@@ -0,0 +1,29 @@+//go:build linux++package workload++import (+ "fmt"+ "os"++ "golang.org/x/sys/unix"+)++// openHostname opens path without following a final symlink and without blocking+// on a FIFO. /proc/<pid>/root is an intermediate symlink and is still followed.+func openHostname(path string) (*os.File, error) {+ fd, err := unix.Open(path, unix.O_RDONLY|unix.O_NONBLOCK|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)+ if err != nil {+ return nil, err+ }+ var st unix.Stat_t+ if err := unix.Fstat(fd, &st); err != nil {+ unix.Close(fd)+ return nil, err+ }+ if st.Mode&unix.S_IFMT != unix.S_IFREG {+ unix.Close(fd)+ return nil, fmt.Errorf("hostname is not a regular file")+ }+ return os.NewFile(uintptr(fd), path), nil+}diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_linux_test.go b/opentelemetry-gpu-collector/internal/workload/hostname_linux_test.gonew file mode 100644index 00000000..99bef08a--- /dev/null+++ b/opentelemetry-gpu-collector/internal/workload/hostname_linux_test.go@@ -0,0 +1,39 @@+//go:build linux++package workload++import (+ "os"+ "path/filepath"+ "testing"+ "time"++ "golang.org/x/sys/unix"+)++func TestReadHostnameFileRejectsFifoAndSymlink(t *testing.T) {+ dir := t.TempDir()+ fifo := filepath.Join(dir, "hostname-fifo")+ if err := unix.Mkfifo(fifo, 0o644); err != nil {+ t.Fatal(err)+ }+ start := time.Now()+ if got := readHostnameFile(fifo); got != "" {+ t.Fatalf("fifo read = %q", got)+ }+ if elapsed := time.Since(start); elapsed > time.Second {+ t.Fatalf("fifo open blocked for %s", elapsed)+ }++ real := filepath.Join(dir, "real")+ if err := os.WriteFile(real, []byte("gemma-worker-0\n"), 0o644); err != nil {+ t.Fatal(err)+ }+ link := filepath.Join(dir, "hostname-link")+ if err := os.Symlink(real, link); err != nil {+ t.Fatal(err)+ }+ if got := readHostnameFile(link); got != "" {+ t.Fatalf("symlink read = %q", got)+ }+}diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_other.go b/opentelemetry-gpu-collector/internal/workload/hostname_other.gonew file mode 100644index 00000000..21adec3d--- /dev/null+++ b/opentelemetry-gpu-collector/internal/workload/hostname_other.go@@ -0,0 +1,9 @@+//go:build !linux++package workload++import "os"++func openHostname(path string) (*os.File, error) {+ return os.Open(path)+}diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_test.go b/opentelemetry-gpu-collector/internal/workload/hostname_test.gonew file mode 100644index 00000000..376d1196--- /dev/null+++ b/opentelemetry-gpu-collector/internal/workload/hostname_test.go@@ -0,0 +1,48 @@+package workload++import (+ "os"+ "path/filepath"+ "testing"+)++func TestHostnameMatchesPod(t *testing.T) {+ cases := []struct {+ hostname, pod, namespace string+ want bool+ }{+ {"gemma-worker-0", "gemma-worker-0", "llms", true},+ {"gemma-worker-0.gemma.llms.svc.cluster.local", "gemma-worker-0", "llms", true}, // setHostnameAsFQDN+ {"gemma-worker-0.gemma.llms.svc.cluster.local", "gemma-worker-0", "other", false},+ {"api.web.prod.svc.cluster.local", "api", "prod", true},+ {"api.web.prod.svc.cluster.local", "api", "dev", false},+ {"gemma-worker-0.gemma.llms.svc.cluster.local", "gemma-worker-0", "", false},+ {"gemma-worker-01", "gemma-worker-0", "llms", false},+ {"gemma-worker-0", "gemma-worker-01", "llms", false},+ {"gpu-node-7", "gemma-worker-0", "llms", false}, // hostNetwork pod / host process+ {"", "gemma-worker-0", "llms", false},+ {"gemma-worker-0", "", "llms", false},+ }+ for _, c := range cases {+ if got := hostnameMatchesPod(c.hostname, c.pod, c.namespace); got != c.want {+ t.Errorf("hostnameMatchesPod(%q, %q, %q) = %v, want %v", c.hostname, c.pod, c.namespace, got, c.want)+ }+ }+}++func TestReadHostnameFile(t *testing.T) {+ dir := t.TempDir()+ p := filepath.Join(dir, "hostname")+ if err := os.WriteFile(p, []byte(" gemma-worker-0\n# ignored second line\n"), 0o644); err != nil {+ t.Fatal(err)+ }+ if got := readHostnameFile(p); got != "gemma-worker-0" {+ t.Fatalf("readHostnameFile = %q", got)+ }+ if got := readHostnameFile(filepath.Join(dir, "missing")); got != "" {+ t.Fatalf("missing file should be empty, got %q", got)+ }+ if got := ProcHostname(0); got != "" {+ t.Fatalf("pid 0 should be empty, got %q", got)+ }+}