Skip to content

Same agent Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

fix(gpu-collector): attribute GPU processes to their own pod, not to the GPU owner (#1719)

Hoang Nguyen4605c7fe4c00
diff --git a/docs/latest/gpu-collector/configuration.mdx b/docs/latest/gpu-collector/configuration.mdx
index 7bda7fe6..5d3ddee3 100644
--- a/docs/latest/gpu-collector/configuration.mdx
+++ b/docs/latest/gpu-collector/configuration.mdx
@@ -34,8 +34,8 @@ The collector is configured entirely via environment variables. All variables fo
| `K8S_NODE_NAME` | | Kubernetes node name via downward API (`spec.nodeName`). Also accepts `OTEL_RESOURCE_ATTRIBUTES_NODE_NAME` (Operator) or legacy `NODE_NAME` |
| `K8S_CLUSTER_NAME` | | Explicit cluster name when cloud auto-detect fails (on-prem). Alias: `OPENLIT_K8S_CLUSTER_NAME`. Only applied in Kubernetes |
| `OPENLIT_K8S_NODE_LOOKUP` | `true` | When `false`, skip `GET /api/v1/nodes/$K8S_NODE_NAME` for instance-type / provider discovery |
-| `OPENLIT_K8S_POD_RESOURCES` | `true` in K8s | Use kubelet PodResources socket; joins GPU UUID → `k8s.pod.name` / namespace / container |
-| `OPENLIT_K8S_POD_LOOKUP` | `true` in K8s when `K8S_NODE_NAME` is set | List pods on this node via the Kubernetes API (needs `list` on pods) for UID/container-id joins |
+| `OPENLIT_K8S_POD_RESOURCES` | `true` in K8s | Kubelet PodResources socket. Names a process only when it has no pod of its own, or the allocation is confirmed as its pod. See below |
+| `OPENLIT_K8S_POD_LOOKUP` | `true` in K8s when `K8S_NODE_NAME` is set | List pods on this node (`list` on pods). Resolves a containerized process's cgroup UID to its own `k8s.pod.name` and namespace |
| `POD_RESOURCES_SOCKET` | OS default | Override kubelet PodResources socket / named pipe path |
| `OPENLIT_CLOUD_DETECT` | `true` | When `false`, skip AWS/GCP/Azure IMDS probes (recommended on bare metal to avoid link-local timeouts) |
| `OTEL_GPU_DCGM_ENABLED` | mode-dependent (`true` for `serving`+) | Optional NVIDIA DCGM profiling (soft-skipped without libdcgm / when DCGM cannot init) |
@@ -64,6 +64,12 @@ The collector is configured entirely via environment variables. All variables fo
| `OTEL_HOST_NIC_ENABLED` | mode-dependent (`true` for `training`+) | Per-NIC `hw.network.*` (ethtool/sysfs). When enabled, `system.network.*` is skipped to avoid duplicates |
| `OTEL_HOST_RDMA_ENABLED` | mode-dependent (`true` for `training`+) | RDMA device counters |
+<Note>
+**Pod name on `process.gpu.*`.** `OPENLIT_K8S_POD_LOOKUP` joins the process cgroup UID to that pod's name and namespace. It needs `list` on pods (see the DaemonSet role below). The list call is field-selected to this node; the permission itself is cluster-scoped.
+
+Without that list, kubelet PodResources supplies a name for a host process (the owner of the GPU being scraped) or when a container's `/proc/<pid>/root/etc/hostname` confirms an allocation. The kubelet writes the pod name there. `setHostnameAsFQDN` counts only when the owner's namespace is in the name (`<pod>.<subdomain>.<namespace>.svc.<cluster>`). `hostNetwork` sees the node hostname, and `spec.hostname` sees the override, so those stay UID-only unless that hostname still names the owner. Two pods with the same name in different namespaces are left UID-only. Confirmation looks at every GPU allocation on the node, so the name does not depend on which GPU is scraped first. `k8s.pod.uid` stays the cgroup value either way.
+</Note>
+
## Host, Kubernetes, and cloud identity
Resource attributes follow [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/) for host, K8s, and cloud:
@@ -113,7 +119,7 @@ env:
If you only set `K8S_NODE_NAME` (without packing it into `OTEL_RESOURCE_ATTRIBUTES`), the collector still maps it to `host.name` and `k8s.node.name` automatically.
-For K8s node label / providerID discovery, grant the DaemonSet ServiceAccount:
+For node label / providerID discovery and for resolving `k8s.pod.name`, grant the DaemonSet ServiceAccount:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
@@ -124,6 +130,9 @@ rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
+ - apiGroups: [""]
+ resources: ["pods"]
+ verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
diff --git a/docs/latest/gpu-collector/installation.mdx b/docs/latest/gpu-collector/installation.mdx
index c2ef891f..d8885659 100644
--- a/docs/latest/gpu-collector/installation.mdx
+++ b/docs/latest/gpu-collector/installation.mdx
@@ -95,6 +95,9 @@ rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get"]
+ - apiGroups: [""]
+ resources: ["pods"]
+ verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
@@ -164,7 +167,7 @@ spec:
```
<Note>
-`hostPID: true` (same role as Docker `--pid=host`) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under `/proc`. Without it, device-level `hw.gpu.*` metrics still work. The `nodes/get` ClusterRole enables OpenCost-style `host.type` / `cloud.provider` discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1).
+`hostPID: true` (same role as Docker `--pid=host`) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under `/proc`. Without it, device-level `hw.gpu.*` metrics still work. `nodes/get` enables OpenCost-style `host.type` / `cloud.provider` discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1). `pods/list` is what `OPENLIT_K8S_POD_LOOKUP` uses to turn a process's cgroup pod UID into `k8s.pod.name`. The list call is field-selected to this node; the permission is cluster-scoped.
</Note>
See [Configuration](/latest/gpu-collector/configuration) for identity env vars and detection order.
diff --git a/docs/latest/gpu-collector/metrics.mdx b/docs/latest/gpu-collector/metrics.mdx
index 9501f487..a4326e57 100644
--- a/docs/latest/gpu-collector/metrics.mdx
+++ b/docs/latest/gpu-collector/metrics.mdx
@@ -151,11 +151,13 @@ These attributes support views like zombie-process triage: filter `process.state
| `hw.id`, `hw.name`, `hw.vendor`, `gpu.index`, `gpu.pci_address` | Same device attrs as `hw.gpu.*` |
| `hw.gpu.task` | On utilization only: `general`, `encoder`, `decoder` |
| `k8s.pod.uid` | From cgroup path when present |
-| `k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name` | Via kubelet PodResources (GPU device join) and/or pod API lookup |
+| `k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name` | The process's own pod (cgroup container id / pod UID joined with the pod API). PodResources is a fallback: per device for host processes, and for a containerized process only when an allocation is confirmed as that pod |
| `container.id` | Runtime container ID when resolvable |
<Note>
`process.start_time` is not exported as a metric attribute (cardinality). Use `process.uptime` for lifetime. On MIG devices, NVML often provides **memory only** (no per-process SM util). Process util may not sum to device util (time-slicing / sampling windows differ). Use `process.state == zombie` to find defunct processes still holding GPU contexts.
+
+PodResources carries no pod UID. A containerized process is confirmed by its own `/etc/hostname` (the kubelet writes the pod name; with `setHostnameAsFQDN`, the owner's namespace must appear in it) matched against allocations on the node, not against whichever GPU is scraped first. `hostNetwork` uses the node hostname. `spec.hostname` uses the override. Those stay UID-only unless the hostname still names the owner. Two pods with the same name in different namespaces stay UID-only unless `OPENLIT_K8S_POD_LOOKUP` can list pods. A process on GPUs allocated to other pods keeps its own pod name. `k8s.pod.uid` is the cgroup value.
</Note>
---
diff --git a/opentelemetry-gpu-collector/internal/workload/enricher.go b/opentelemetry-gpu-collector/internal/workload/enricher.go
index bffd94b5..3e80378c 100644
--- a/opentelemetry-gpu-collector/internal/workload/enricher.go
+++ b/opentelemetry-gpu-collector/internal/workload/enricher.go
@@ -13,20 +13,29 @@ import (
// Enricher resolves PID → pod identity using cgroup + optional PodResources + API.
// Outside Kubernetes (or when both enrichment flags are off) Resolve is a free no-op.
type Enricher struct {
- logger *slog.Logger
- cfg *config.Config
+ logger *slog.Logger
+ cfg *config.Config
enabled bool
- mu sync.RWMutex
- cache map[int32]PodInfo
- negCache map[int32]time.Time // PID → when negative miss was cached
- cacheAt time.Time
- ttl time.Duration
- byUID map[string]PodInfo
- byName map[string]PodInfo
- byContID map[string]PodInfo
- byPID map[int32]PodInfo
- byDevice map[string]PodInfo // GPU UUID / device id → pod
+ // resolveCgroup reads the process's own pod UID / container id (ResolvePod); injectable for tests.
+ resolveCgroup func(pid int32) (PodInfo, bool)
+ // resolveHostname reads the process's own /etc/hostname (ProcHostname); injectable for tests.
+ resolveHostname func(pid int32) string
+
+ mu sync.RWMutex
+ own map[int32]PodInfo // PID → identity from the process itself; valid on every device
+ // cache / negCache hold device-dependent results (GPU-owner fallback, partial
+ // identities, misses), keyed per (pid, device).
+ cache map[cacheKey]PodInfo
+ negCache map[cacheKey]time.Time // when the negative miss was cached
+ hostnames map[int32]string // PID → container hostname memo ("" when unreadable)
+ cacheAt time.Time
+ ttl time.Duration
+ byUID map[string]PodInfo
+ byName map[string]PodInfo
+ byContID map[string]PodInfo
+ byPID map[int32]PodInfo
+ byDevice map[string]PodInfo // GPU UUID / device id → pod
refreshMu sync.Mutex
stopCh chan struct{}
@@ -40,6 +49,14 @@ type Enricher struct {
connectInterval time.Duration
}
+// cacheKey scopes a resolution that depended on the GPU it was asked for: a process can
+// hold contexts on GPUs allocated to other pods, so whether the PodResources owner of a
+// GPU applies (or whether there is one at all) differs per device.
+type cacheKey struct {
+ pid int32
+ deviceID string
+}
+
// NewEnricher builds a workload enricher. Safe on non-K8s (no-op enrichment).
func NewEnricher(cfg *config.Config, logger *slog.Logger) *Enricher {
if cfg == nil {
@@ -51,8 +68,12 @@ func NewEnricher(cfg *config.Config, logger *slog.Logger) *Enricher {
logger: logger,
cfg: cfg,
enabled: enabled,
- cache: make(map[int32]PodInfo),
- negCache: make(map[int32]time.Time),
+ resolveCgroup: ResolvePod,
+ resolveHostname: ProcHostname,
+ own: make(map[int32]PodInfo),
+ cache: make(map[cacheKey]PodInfo),
+ negCache: make(map[cacheKey]time.Time),
+ hostnames: make(map[int32]string),
byUID: make(map[string]PodInfo),
byName: make(map[string]PodInfo),
byContID: make(map[string]PodInfo),
@@ -121,17 +142,36 @@ func (e *Enricher) ensureClients() {
// Resolve returns the best-known pod identity for pid.
// deviceID is an optional GPU UUID used to join PodResources device allocations.
+//
+// Identity comes from the process itself first (cgroup → container id / pod UID →
+// kubelet PodResources and pod API maps). The pod that PodResources allocated a GPU
+// to is only a fallback when the process has no pod name of its own: a process may
+// hold a context on a GPU allocated to a different pod (privileged pods,
+// NVIDIA_VISIBLE_DEVICES=all, host processes), and the GPU owner is not that process's
+// pod. A host process takes the owner of the device being scraped. A process with a
+// cgroup pod UID takes an owner only when that owner is confirmed as the same pod,
+// by UID or (PodResources carries no UID) by the process hostname matched against
+// every allocation on the node. The metrics callback emits GPUs in index order, so
+// confirmation cannot depend on the device passed in. An unconfirmed or ambiguous
+// owner leaves the process UID-only. Identity resolved from or confirmed against the
+// process itself is cached per pid; device-dependent fallback, partial identity, and
+// misses are cached per (pid, device).
func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {
if e == nil || !e.enabled || pid <= 0 {
return PodInfo{}, false
}
+ key := cacheKey{pid: pid, deviceID: deviceID}
e.mu.RLock()
- if info, ok := e.cache[pid]; ok {
+ if info, ok := e.own[pid]; ok {
e.mu.RUnlock()
- return info, info.PodUID != "" || info.PodName != ""
+ return info, true
}
- if at, ok := e.negCache[pid]; ok && time.Since(at) < e.ttl {
+ if info, ok := e.cache[key]; ok {
+ e.mu.RUnlock()
+ return info, true
+ }
+ if at, ok := e.negCache[key]; ok && time.Since(at) < e.ttl {
e.mu.RUnlock()
return PodInfo{}, false
}
@@ -141,8 +181,12 @@ func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {
byDev := e.byDevice
e.mu.RUnlock()
+ resolveCgroup := e.resolveCgroup
+ if resolveCgroup == nil {
+ resolveCgroup = ResolvePod
+ }
info := byPID
- if cg, ok := ResolvePod(pid); ok {
+ if cg, ok := resolveCgroup(pid); ok {
if info.PodUID == "" {
info.PodUID = cg.PodUID
}
@@ -150,13 +194,7 @@ func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {
info.ContainerID = cg.ContainerID
}
}
- // Device allocation from PodResources (works without Pod UID).
- if deviceID != "" {
- if p, ok := byDev[deviceID]; ok {
- mergePod(&info, p)
- }
- }
- // Prefer container ID for multi-container pods.
+ // 1. The process's own pod: container id (multi-container pods) then pod UID.
if info.ContainerID != "" {
if p, ok := byCont[info.ContainerID]; ok {
mergePod(&info, p)
@@ -167,19 +205,116 @@ func (e *Enricher) Resolve(pid int32, deviceID string) (PodInfo, bool) {
mergePod(&info, p)
}
}
+ ownIdentity := info.PodName != ""
+ // 2. Fallback to a PodResources GPU owner.
+ // Host process: the owner of this device only, cached per device.
+ // Cgroup UID and no name yet: one confirmation against every allocation.
+ // That result is the process's own pod, so the first GPU in a scrape already
+ // carries it. The hostname file is read only when some owner has no UID.
+ if !ownIdentity && info.PodUID == "" && deviceID != "" {
+ if p, ok := byDev[deviceID]; ok {
+ mergePod(&info, p)
+ }
+ } else if !ownIdentity && info.PodUID != "" {
+ if p, ok := e.confirmProcessOwner(byDev, info.PodUID, pid); ok {
+ mergePod(&info, p)
+ ownIdentity = true
+ }
+ }
hit := info.PodUID != "" || info.PodName != ""
e.mu.Lock()
- if hit {
- e.cache[pid] = info
- delete(e.negCache, pid)
- } else {
- e.negCache[pid] = time.Now()
+ switch {
+ case ownIdentity:
+ e.own[pid] = info
+ case hit:
+ e.cache[key] = info
+ delete(e.negCache, key)
+ default:
+ e.negCache[key] = time.Now()
}
e.mu.Unlock()
return info, hit
}
+// confirmProcessOwner reports the single PodResources pod that is this process.
+// A UID match wins and does not consult the hostname. Otherwise the process hostname
+// is matched against owners that carry no UID. Two different name/namespace pairs
+// are ambiguous: a same-named pod in another namespace must not be attached.
+func (e *Enricher) confirmProcessOwner(byDev map[string]PodInfo, podUID string, pid int32) (PodInfo, bool) {
+ if p, ok, ambiguous := collectOwners(byDev, func(p PodInfo) bool {
+ return podUID != "" && p.PodUID != "" && p.PodUID == podUID
+ }); ambiguous {
+ return PodInfo{}, false
+ } else if ok {
+ return p, true
+ }
+ if !ownerNeedsHostname(byDev) {
+ return PodInfo{}, false
+ }
+ hostname := e.hostnameOf(pid)
+ p, ok, ambiguous := collectOwners(byDev, func(p PodInfo) bool {
+ return p.PodUID == "" && hostnameMatchesPod(hostname, p.PodName, p.Namespace)
+ })
+ if ambiguous || !ok {
+ return PodInfo{}, false
+ }
+ return p, true
+}
+
+// collectOwners returns the one pod wanted by match. A second pod with a different
+// name or namespace is ambiguous. Several allocations of the same pod keep the pod
+// and drop ContainerName when the containers disagree.
+func collectOwners(byDev map[string]PodInfo, match func(PodInfo) bool) (info PodInfo, ok bool, ambiguous bool) {
+ for _, p := range byDev {
+ if p.PodName == "" || !match(p) {
+ continue
+ }
+ if !ok {
+ info, ok = p, true
+ continue
+ }
+ if p.Namespace != info.Namespace || p.PodName != info.PodName {
+ return PodInfo{}, false, true
+ }
+ if p.ContainerName != info.ContainerName {
+ info.ContainerName = ""
+ }
+ }
+ return info, ok, false
+}
+
+func ownerNeedsHostname(byDev map[string]PodInfo) bool {
+ for _, p := range byDev {
+ if p.PodUID == "" && p.PodName != "" {
+ return true
+ }
+ }
+ return false
+}
+
+// hostnameOf returns the process's container hostname, read once per pid per refresh.
+func (e *Enricher) hostnameOf(pid int32) string {
+ e.mu.RLock()
+ h, ok := e.hostnames[pid]
+ e.mu.RUnlock()
+ if ok {
+ return h
+ }
+ read := e.resolveHostname
+ if read == nil {
+ read = ProcHostname
+ }
+ h = read(pid)
+ e.mu.Lock()
+ if e.hostnames == nil {
+ e.hostnames = make(map[int32]string)
+ }
+ e.hostnames[pid] = h
+ e.mu.Unlock()
+ return h
+}
+
func mergePod(dst *PodInfo, src PodInfo) {
if dst.PodUID == "" {
dst.PodUID = src.PodUID
@@ -246,9 +381,11 @@ func (e *Enricher) refreshMaps() {
e.mu.Lock()
defer e.mu.Unlock()
+ e.hostnames = make(map[int32]string) // PIDs may be reused; re-read next cycle
if indexed {
- e.cache = make(map[int32]PodInfo)
- e.negCache = make(map[int32]time.Time)
+ e.own = make(map[int32]PodInfo)
+ e.cache = make(map[cacheKey]PodInfo)
+ e.negCache = make(map[cacheKey]time.Time)
e.byUID = byUID
e.byName = byName
e.byContID = byContID
diff --git a/opentelemetry-gpu-collector/internal/workload/enricher_test.go b/opentelemetry-gpu-collector/internal/workload/enricher_test.go
index c1e6d29d..141458ed 100644
--- a/opentelemetry-gpu-collector/internal/workload/enricher_test.go
+++ b/opentelemetry-gpu-collector/internal/workload/enricher_test.go
@@ -1,6 +1,9 @@
package workload
-import "testing"
+import (
+ "testing"
+ "time"
+)
func TestIndexPodSeparatesContainerFromUID(t *testing.T) {
byUID := map[string]PodInfo{}
@@ -16,7 +19,7 @@ func TestIndexPodSeparatesContainerFromUID(t *testing.T) {
indexPod(byUID, byName, byCont, byPID, byDev, PodInfo{
PodUID: "uid-1", PodName: "p", Namespace: "ns",
ContainerName: "app", ContainerID: "bbb",
- DeviceIDs: []string{"GPU-1"},
+ DeviceIDs: []string{"GPU-1"},
})
if byUID["uid-1"].ContainerName != "" || byUID["uid-1"].ContainerID != "" {
@@ -62,3 +65,352 @@ func TestMergePodPrefersExistingContainer(t *testing.T) {
t.Fatalf("merge missed pod fields: %+v", info)
}
}
+
+func newTestEnricher(cgroup func(int32) (PodInfo, bool)) *Enricher {
+ return &Enricher{
+ enabled: true,
+ resolveCgroup: cgroup,
+ resolveHostname: func(int32) string { return "" }, // unreadable unless a test sets it
+ own: make(map[int32]PodInfo),
+ cache: make(map[cacheKey]PodInfo),
+ negCache: make(map[cacheKey]time.Time),
+ hostnames: make(map[int32]string),
+ byUID: make(map[string]PodInfo),
+ byName: make(map[string]PodInfo),
+ byContID: make(map[string]PodInfo),
+ byPID: make(map[int32]PodInfo),
+ byDevice: make(map[string]PodInfo),
+ ttl: 15 * time.Second,
+ }
+}
+
+var (
+ gemma = PodInfo{PodUID: "uid-gemma", PodName: "gemma-worker-0", Namespace: "llms", ContainerName: "main"}
+ qwen = PodInfo{PodUID: "uid-qwen", PodName: "qwen-worker-0", Namespace: "llms-dev", ContainerName: "main"}
+ lmcache = PodInfo{PodUID: "uid-lmcache", PodName: "lmcache-server", Namespace: "cache", ContainerName: "main"}
+)
+
+// A privileged pod (NVIDIA_VISIBLE_DEVICES=all) holding contexts on every GPU of the
+// node must keep its own identity on GPUs allocated to other pods.
+func TestResolvePrefersCgroupIdentityOverDeviceOwner(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-lmcache", ContainerID: "cid-lmcache"}, true
+ })
+ e.byUID["uid-lmcache"] = lmcache
+ e.byDevice["GPU-3"] = gemma
+ e.byDevice["GPU-0"] = qwen
+
+ got, ok := e.Resolve(42, "GPU-3")
+ if !ok || got.PodName != "lmcache-server" || got.Namespace != "cache" || got.PodUID != "uid-lmcache" {
+ t.Fatalf("expected cgroup pod, got %+v", got)
+ }
+ // Cached per pid: same answer on another GPU owned by another pod.
+ got, _ = e.Resolve(42, "GPU-0")
+ if got.PodName != "lmcache-server" {
+ t.Fatalf("cached identity changed with device: %+v", got)
+ }
+}
+
+func TestResolveContainerIDBeatsDeviceOwner(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-qwen", ContainerID: "cid-qwen"}, true
+ })
+ e.byContID["cid-qwen"] = PodInfo{PodUID: "uid-qwen", PodName: "qwen-worker-0", Namespace: "llms-dev", ContainerName: "main", ContainerID: "cid-qwen"}
+ e.byDevice["GPU-3"] = gemma
+
+ got, _ := e.Resolve(7, "GPU-3")
+ if got.PodName != "qwen-worker-0" || got.ContainerName != "main" || got.ContainerID != "cid-qwen" {
+ t.Fatalf("expected container-id join, got %+v", got)
+ }
+}
+
+// PodResources only (no pod API): the process has a cgroup pod UID but no name for it,
+// and the GPU owner carries no UID. With no hostname to confirm the owner (unreadable),
+// or a hostname naming another pod (lmcache on a GPU allocated to gemma: the #1718
+// mislabel), the owner's name must not be attached next to a UID that may belong to
+// another pod. Identity stays UID-only, on every device.
+func TestResolveUnconfirmedDeviceOwnerNotAppliedToCgroupProcess(t *testing.T) {
+ for name, hostname := range map[string]string{"unreadable": "", "other pod": "lmcache-server"} {
+ t.Run(name, func(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-lmcache"}, true
+ })
+ e.resolveHostname = func(int32) string { return hostname }
+ owner := func(p PodInfo) PodInfo { p.PodUID = ""; return p } // PodResources carries no UID
+ e.byDevice["GPU-3"] = owner(gemma)
+ e.byDevice["GPU-0"] = owner(qwen)
+
+ for _, dev := range []string{"GPU-3", "GPU-0", "GPU-3"} {
+ got, ok := e.Resolve(9, dev)
+ if !ok || got.PodUID != "uid-lmcache" {
+ t.Fatalf("%s: expected cgroup uid, got %+v ok=%v", dev, got, ok)
+ }
+ if got.PodName != "" || got.Namespace != "" || got.ContainerName != "" {
+ t.Fatalf("%s: borrowed unconfirmed owner labels: %+v", dev, got)
+ }
+ }
+ })
+ }
+}
+
+// PodResources only, common case: a pod's own process on the GPU allocated to it. The
+// owner carries no UID, but the container hostname is the pod name, which confirms it.
+// The confirmed identity is the process's own: cached per pid, hostname read once, and
+// kept on a GPU allocated to another pod.
+func TestResolveDeviceOwnerConfirmedByHostname(t *testing.T) {
+ for name, hostname := range map[string]string{
+ "pod name": "gemma-worker-0",
+ "fqdn": "gemma-worker-0.gemma.llms.svc.cluster.local",
+ } {
+ t.Run(name, func(t *testing.T) {
+ reads := 0
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-gemma", ContainerID: "cid-gemma"}, true
+ })
+ e.resolveHostname = func(int32) string { reads++; return hostname }
+ owner := func(p PodInfo) PodInfo { p.PodUID = ""; return p }
+ e.byDevice["GPU-3"] = owner(gemma)
+ e.byDevice["GPU-0"] = owner(qwen)
+
+ got, ok := e.Resolve(21, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.ContainerName != "main" {
+ t.Fatalf("expected hostname-confirmed owner, got %+v", got)
+ }
+ if got.PodUID != "uid-gemma" || got.ContainerID != "cid-gemma" {
+ t.Fatalf("cgroup identity lost: %+v", got)
+ }
+ got, _ = e.Resolve(21, "GPU-0")
+ if got.PodName != "gemma-worker-0" {
+ t.Fatalf("confirmed identity must follow the process to other GPUs, got %+v", got)
+ }
+ if reads != 1 {
+ t.Fatalf("hostname should be read once per pid, got %d", reads)
+ }
+ })
+ }
+}
+
+// Confirmation searches every allocation. The GPU passed to Resolve is often a device
+// this process does not own, and the metrics callback emits that device first.
+func TestResolveHostnameConfirmIgnoresScrapeOrder(t *testing.T) {
+ reads := 0
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-qwen"}, true
+ })
+ e.resolveHostname = func(int32) string { reads++; return "qwen-worker-0" }
+ owner := func(p PodInfo) PodInfo { p.PodUID = ""; return p }
+ e.byDevice["GPU-3"] = owner(gemma)
+ e.byDevice["GPU-0"] = owner(qwen)
+
+ got, ok := e.Resolve(22, "GPU-9")
+ if !ok || got.PodName != "qwen-worker-0" || got.Namespace != "llms-dev" || got.PodUID != "uid-qwen" {
+ t.Fatalf("name must be present on a GPU this pod does not own, got %+v ok=%v", got, ok)
+ }
+ got, _ = e.Resolve(22, "GPU-3")
+ if got.PodName != "qwen-worker-0" {
+ t.Fatalf("confirmed identity must follow the process, got %+v", got)
+ }
+ if reads != 1 {
+ t.Fatalf("hostname should be read once per pid, got %d", reads)
+ }
+}
+
+// Same pod name in two namespaces is not proof. Leave the process UID-only.
+func TestResolveAmbiguousHostnameStaysUIDOnly(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-worker"}, true
+ })
+ e.resolveHostname = func(int32) string { return "worker" }
+ owner := func(ns string) PodInfo {
+ return PodInfo{PodName: "worker", Namespace: ns, ContainerName: "main"}
+ }
+ e.byDevice["GPU-0"] = owner("a")
+ e.byDevice["GPU-1"] = owner("b")
+
+ for _, dev := range []string{"GPU-0", "GPU-1"} {
+ got, ok := e.Resolve(24, dev)
+ if !ok || got.PodUID != "uid-worker" || got.PodName != "" || got.Namespace != "" {
+ t.Fatalf("%s: ambiguous hostname must stay uid-only, got %+v", dev, got)
+ }
+ }
+}
+
+// setHostnameAsFQDN includes the namespace, so the other same-named pod is not a candidate.
+func TestResolveFQDNHostnameSelectsNamespace(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-gemma"}, true
+ })
+ e.resolveHostname = func(int32) string { return "gemma-worker-0.gemma.llms.svc.cluster.local" }
+ strip := func(p PodInfo, ns string) PodInfo {
+ p.PodUID = ""
+ p.Namespace = ns
+ return p
+ }
+ e.byDevice["GPU-3"] = strip(gemma, "other")
+ e.byDevice["GPU-0"] = strip(gemma, "llms")
+
+ got, ok := e.Resolve(25, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.ContainerName != "main" {
+ t.Fatalf("expected the namespace named by the FQDN, got %+v ok=%v", got, ok)
+ }
+}
+
+// Two containers of the confirmed pod must not contribute an arbitrary container name.
+func TestResolveConfirmedPodDropsDisagreeingContainerName(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-gemma"}, true
+ })
+ e.resolveHostname = func(int32) string { return "gemma-worker-0" }
+ e.byDevice["GPU-0"] = PodInfo{PodName: "gemma-worker-0", Namespace: "llms", ContainerName: "main"}
+ e.byDevice["GPU-1"] = PodInfo{PodName: "gemma-worker-0", Namespace: "llms", ContainerName: "sidecar"}
+
+ got, ok := e.Resolve(26, "GPU-1")
+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.ContainerName != "" {
+ t.Fatalf("expected pod without a guessed container, got %+v ok=%v", got, ok)
+ }
+}
+
+// Hostname confirmation is only for processes with a cgroup pod UID: a host process
+// (node hostname) keeps the per-device owner fallback.
+func TestResolveHostProcessIgnoresHostname(t *testing.T) {
+ reads := 0
+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })
+ e.resolveHostname = func(int32) string { reads++; return "gpu-node-7" }
+ e.byDevice["GPU-3"] = gemma
+
+ got, ok := e.Resolve(23, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" {
+ t.Fatalf("host process should still fall back to the GPU owner, got %+v", got)
+ }
+ if reads != 0 {
+ t.Fatalf("host process must not read a container hostname, got %d reads", reads)
+ }
+}
+
+// The GPU owner is applied to a process with a cgroup pod UID only when the owner's UID
+// confirms it is the same pod.
+func TestResolveDeviceOwnerConfirmedByUID(t *testing.T) {
+ reads := 0
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-gemma"}, true
+ })
+ e.resolveHostname = func(int32) string { reads++; return "gemma-worker-0" }
+ e.byDevice["GPU-3"] = gemma
+
+ got, ok := e.Resolve(10, "GPU-5")
+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" || got.PodUID != "uid-gemma" {
+ t.Fatalf("expected confirmed owner, got %+v", got)
+ }
+ if reads != 0 {
+ t.Fatalf("a UID match must not read the container hostname, got %d", reads)
+ }
+}
+
+// A process without pod identity of its own (host process) falls back to the GPU owner,
+// but that is a guess per device: seen on two GPUs it is not pinned to the owner of the
+// first one.
+func TestResolveDeviceOwnerFallbackIsPerDevice(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })
+ e.byDevice["GPU-3"] = gemma
+ e.byDevice["GPU-0"] = qwen
+
+ got, ok := e.Resolve(9, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" || got.Namespace != "llms" {
+ t.Fatalf("expected device-owner fallback, got %+v", got)
+ }
+ got, _ = e.Resolve(9, "GPU-0")
+ if got.PodName != "qwen-worker-0" || got.Namespace != "llms-dev" {
+ t.Fatalf("fallback must be per device, got %+v", got)
+ }
+ got, _ = e.Resolve(9, "GPU-3")
+ if got.PodName != "gemma-worker-0" {
+ t.Fatalf("per-device fallback not reproducible, got %+v", got)
+ }
+}
+
+// A miss on a GPU without an owner must not be cached per PID: the same process on a
+// GPU that has an owner still reaches the fallback.
+func TestResolveMissOnOneDeviceDoesNotHideOwnerOnAnother(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })
+ e.byDevice["GPU-3"] = gemma
+
+ if got, ok := e.Resolve(13, "GPU-5"); ok {
+ t.Fatalf("no owner for GPU-5: expected miss, got %+v", got)
+ }
+ got, ok := e.Resolve(13, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" {
+ t.Fatalf("negative cache leaked across devices, got %+v ok=%v", got, ok)
+ }
+ if got, ok := e.Resolve(13, "GPU-5"); ok {
+ t.Fatalf("GPU-5 still has no owner: expected miss, got %+v", got)
+ }
+}
+
+// A UID-only result cached for a device that had no confirming owner must not stick
+// to the pid. A later device whose owner confirms the UID still supplies the name,
+// including on the device that was cached first.
+func TestResolvePartialIdentityDoesNotHideOwnerOnAnotherDevice(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-gemma"}, true
+ })
+
+ got, ok := e.Resolve(14, "GPU-5")
+ if !ok || got.PodUID != "uid-gemma" || got.PodName != "" {
+ t.Fatalf("expected uid-only partial on GPU-5, got %+v ok=%v", got, ok)
+ }
+ e.byDevice["GPU-3"] = gemma
+ got, ok = e.Resolve(14, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" {
+ t.Fatalf("partial identity cached per pid hid the owner, got %+v", got)
+ }
+ got, _ = e.Resolve(14, "GPU-5")
+ if got.PodName != "gemma-worker-0" {
+ t.Fatalf("confirmed identity must replace the earlier uid-only series, got %+v", got)
+ }
+}
+
+// Identity resolved from the process itself is cached once and reused on every device.
+func TestResolveOwnIdentityCachedAcrossDevices(t *testing.T) {
+ calls := 0
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ calls++
+ return PodInfo{PodUID: "uid-lmcache"}, true
+ })
+ e.byUID["uid-lmcache"] = lmcache
+ e.byDevice["GPU-3"] = gemma
+
+ for _, dev := range []string{"GPU-3", "GPU-0", "GPU-1"} {
+ got, _ := e.Resolve(15, dev)
+ if got.PodName != "lmcache-server" {
+ t.Fatalf("%s: expected own pod, got %+v", dev, got)
+ }
+ }
+ if calls != 1 {
+ t.Fatalf("own identity should be resolved once per pid, cgroup read %d times", calls)
+ }
+}
+
+func TestResolveNoDeviceOwnerWhenUIDDiffers(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) {
+ return PodInfo{PodUID: "uid-lmcache"}, true
+ })
+ e.byDevice["GPU-3"] = gemma // owner UID known (pod API) and different
+
+ got, ok := e.Resolve(11, "GPU-3")
+ if !ok || got.PodUID != "uid-lmcache" || got.PodName != "" {
+ t.Fatalf("must not borrow the owner's name, got %+v", got)
+ }
+}
+
+func TestResolveHostProcessUsesDeviceOwner(t *testing.T) {
+ e := newTestEnricher(func(int32) (PodInfo, bool) { return PodInfo{}, false })
+ e.byDevice["GPU-3"] = gemma
+
+ got, ok := e.Resolve(13, "GPU-3")
+ if !ok || got.PodName != "gemma-worker-0" {
+ t.Fatalf("process without cgroup identity should fall back to the GPU owner, got %+v", got)
+ }
+ if _, ok := e.Resolve(13, "GPU-5"); ok {
+ t.Fatal("no owner for GPU-5: expected miss")
+ }
+}
diff --git a/opentelemetry-gpu-collector/internal/workload/hostname.go b/opentelemetry-gpu-collector/internal/workload/hostname.go
new file mode 100644
index 00000000..7809f3bd
--- /dev/null
+++ b/opentelemetry-gpu-collector/internal/workload/hostname.go
@@ -0,0 +1,62 @@
+package workload
+
+import (
+ "io"
+ "strconv"
+ "strings"
+)
+
+// maxHostnameBytes bounds the read of a container's /etc/hostname (a pod name is ≤ 253 bytes).
+const maxHostnameBytes = 4096
+
+// ProcHostname returns the hostname of the process's own mount namespace, read from
+// /proc/<pid>/root/etc/hostname. In Kubernetes the kubelet writes the pod name there
+// (unless spec.hostname / setHostnameAsFQDN / hostNetwork change it), which lets the
+// enricher confirm a kubelet PodResources owner, who carries no pod UID, against the
+// process itself. Empty when unreadable (no hostPID, no ptrace access, non-Linux).
+func ProcHostname(pid int32) string {
+ if pid <= 0 {
+ return ""
+ }
+ return readHostnameFile("/proc/" + strconv.FormatInt(int64(pid), 10) + "/root/etc/hostname")
+}
+
+func readHostnameFile(path string) string {
+ // openHostname refuses symlinks, directories, and FIFOs. A container can
+ // replace /etc/hostname with a FIFO; a blocking open here stalls the single
+ // metrics callback that walks every GPU.
+ f, err := openHostname(path)
+ if err != nil {
+ return ""
+ }
+ defer f.Close()
+ data, err := io.ReadAll(io.LimitReader(f, maxHostnameBytes))
+ if err != nil {
+ return ""
+ }
+ s := string(data)
+ if i := strings.IndexByte(s, '\n'); i >= 0 {
+ s = s[:i]
+ }
+ return strings.TrimSpace(s)
+}
+
+// hostnameMatchesPod reports whether a container hostname names podName in namespace.
+// The kubelet default writes the pod name. setHostnameAsFQDN writes
+// <pod>.<subdomain>.<namespace>.svc.<cluster>; a dotted name counts only when
+// namespace appears in that position, so pod "api" in namespace dev does not
+// confirm hostname "api.web.prod.svc.cluster.local".
+func hostnameMatchesPod(hostname, podName, namespace string) bool {
+ if hostname == "" || podName == "" {
+ return false
+ }
+ if hostname == podName {
+ return true
+ }
+ if namespace == "" || !strings.HasPrefix(hostname, podName+".") {
+ return false
+ }
+ rest := hostname[len(podName)+1:]
+ needle := namespace + ".svc."
+ return strings.HasPrefix(rest, needle) || strings.Contains(rest, "."+needle)
+}
diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_linux.go b/opentelemetry-gpu-collector/internal/workload/hostname_linux.go
new file mode 100644
index 00000000..d7047881
--- /dev/null
+++ b/opentelemetry-gpu-collector/internal/workload/hostname_linux.go
@@ -0,0 +1,29 @@
+//go:build linux
+
+package workload
+
+import (
+ "fmt"
+ "os"
+
+ "golang.org/x/sys/unix"
+)
+
+// openHostname opens path without following a final symlink and without blocking
+// on a FIFO. /proc/<pid>/root is an intermediate symlink and is still followed.
+func openHostname(path string) (*os.File, error) {
+ fd, err := unix.Open(path, unix.O_RDONLY|unix.O_NONBLOCK|unix.O_CLOEXEC|unix.O_NOFOLLOW, 0)
+ if err != nil {
+ return nil, err
+ }
+ var st unix.Stat_t
+ if err := unix.Fstat(fd, &st); err != nil {
+ unix.Close(fd)
+ return nil, err
+ }
+ if st.Mode&unix.S_IFMT != unix.S_IFREG {
+ unix.Close(fd)
+ return nil, fmt.Errorf("hostname is not a regular file")
+ }
+ return os.NewFile(uintptr(fd), path), nil
+}
diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_linux_test.go b/opentelemetry-gpu-collector/internal/workload/hostname_linux_test.go
new file mode 100644
index 00000000..99bef08a
--- /dev/null
+++ b/opentelemetry-gpu-collector/internal/workload/hostname_linux_test.go
@@ -0,0 +1,39 @@
+//go:build linux
+
+package workload
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "golang.org/x/sys/unix"
+)
+
+func TestReadHostnameFileRejectsFifoAndSymlink(t *testing.T) {
+ dir := t.TempDir()
+ fifo := filepath.Join(dir, "hostname-fifo")
+ if err := unix.Mkfifo(fifo, 0o644); err != nil {
+ t.Fatal(err)
+ }
+ start := time.Now()
+ if got := readHostnameFile(fifo); got != "" {
+ t.Fatalf("fifo read = %q", got)
+ }
+ if elapsed := time.Since(start); elapsed > time.Second {
+ t.Fatalf("fifo open blocked for %s", elapsed)
+ }
+
+ real := filepath.Join(dir, "real")
+ if err := os.WriteFile(real, []byte("gemma-worker-0\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ link := filepath.Join(dir, "hostname-link")
+ if err := os.Symlink(real, link); err != nil {
+ t.Fatal(err)
+ }
+ if got := readHostnameFile(link); got != "" {
+ t.Fatalf("symlink read = %q", got)
+ }
+}
diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_other.go b/opentelemetry-gpu-collector/internal/workload/hostname_other.go
new file mode 100644
index 00000000..21adec3d
--- /dev/null
+++ b/opentelemetry-gpu-collector/internal/workload/hostname_other.go
@@ -0,0 +1,9 @@
+//go:build !linux
+
+package workload
+
+import "os"
+
+func openHostname(path string) (*os.File, error) {
+ return os.Open(path)
+}
diff --git a/opentelemetry-gpu-collector/internal/workload/hostname_test.go b/opentelemetry-gpu-collector/internal/workload/hostname_test.go
new file mode 100644
index 00000000..376d1196
--- /dev/null
+++ b/opentelemetry-gpu-collector/internal/workload/hostname_test.go
@@ -0,0 +1,48 @@
+package workload
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+)
+
+func TestHostnameMatchesPod(t *testing.T) {
+ cases := []struct {
+ hostname, pod, namespace string
+ want bool
+ }{
+ {"gemma-worker-0", "gemma-worker-0", "llms", true},
+ {"gemma-worker-0.gemma.llms.svc.cluster.local", "gemma-worker-0", "llms", true}, // setHostnameAsFQDN
+ {"gemma-worker-0.gemma.llms.svc.cluster.local", "gemma-worker-0", "other", false},
+ {"api.web.prod.svc.cluster.local", "api", "prod", true},
+ {"api.web.prod.svc.cluster.local", "api", "dev", false},
+ {"gemma-worker-0.gemma.llms.svc.cluster.local", "gemma-worker-0", "", false},
+ {"gemma-worker-01", "gemma-worker-0", "llms", false},
+ {"gemma-worker-0", "gemma-worker-01", "llms", false},
+ {"gpu-node-7", "gemma-worker-0", "llms", false}, // hostNetwork pod / host process
+ {"", "gemma-worker-0", "llms", false},
+ {"gemma-worker-0", "", "llms", false},
+ }
+ for _, c := range cases {
+ if got := hostnameMatchesPod(c.hostname, c.pod, c.namespace); got != c.want {
+ t.Errorf("hostnameMatchesPod(%q, %q, %q) = %v, want %v", c.hostname, c.pod, c.namespace, got, c.want)
+ }
+ }
+}
+
+func TestReadHostnameFile(t *testing.T) {
+ dir := t.TempDir()
+ p := filepath.Join(dir, "hostname")
+ if err := os.WriteFile(p, []byte(" gemma-worker-0\n# ignored second line\n"), 0o644); err != nil {
+ t.Fatal(err)
+ }
+ if got := readHostnameFile(p); got != "gemma-worker-0" {
+ t.Fatalf("readHostnameFile = %q", got)
+ }
+ if got := readHostnameFile(filepath.Join(dir, "missing")); got != "" {
+ t.Fatalf("missing file should be empty, got %q", got)
+ }
+ if got := ProcHostname(0); got != "" {
+ t.Fatalf("pid 0 should be empty, got %q", got)
+ }
+}