Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Add AMD GPU support on native Linux via amdgpu sysfs and DRM fdinfo

Pinta365571f3d0a14e0
diff --git a/AGENTS.md b/AGENTS.md
index e471859..e1f252b 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -47,9 +47,9 @@ these rules:
```
cli.py ──► Collector.sample() (collector.py)
├─ adapters dxgi.py (Windows) / linux.enumerate_adapters / macos.enumerate_adapters
- ├─ counters pdh.py GpuCounters (Windows PDH) / linux.py LinuxCounters (NVML) /
+ ├─ counters pdh.py GpuCounters (Windows PDH) / linux.py LinuxCounters (NVML, + drm.py for amdgpu) /
│ macos.py MacCounters (IOKit Metal clients + footprint poller)
- ├─ device metrics nvml.py (macOS: MacCounters.device from IOKit)
+ ├─ device metrics nvml.py; drm.py read_device (AMD, Linux) (macOS: MacCounters.device from IOKit)
├─ WSL host wslhost.py: HostStream (powershell PDH via interop), ContextTracker
├─ processes procinfo.py: runtime DLLs, mapped files, cmdline, classify()
├─ model naming stores.py (Ollama, LM Studio, Hugging Face), gguf.py (header, kind, sizes)
@@ -60,7 +60,9 @@ cli.py diff ──► diff.py: compares two saved --json snapshots (no collectio
```
The backend is chosen at import time in `collector.py`: `BACKEND` is `wddm` on Windows, `nvml` on
-native Linux, `wsl` under WSL, and `metal` on macOS. One-shot output (`--once`, `--json`) uses `scan_budget=None` to
+native Linux, `wsl` under WSL, and `metal` on macOS. On native Linux, `Collector.backend` becomes `drm`
+when the selected GPU is an amdgpu card (device readings from sysfs, processes from DRM fdinfo). One-shot
+output (`--once`, `--json`) uses `scan_budget=None` to
identify everything immediately; the TUI spreads scans across refreshes.
## Where to change things
@@ -124,6 +126,7 @@ run builds artifacts without publishing.
## Known limits
Validated live: Windows 11 with an RTX 4080 SUPER (WDDM, plus WSL2), a Linux container with an
-RTX 3060 Ti, and macOS 26.3.1 on an M4 Mac mini (Metal; own-user processes, not run as root).
-Not yet validated on hardware: bare-metal Linux, multi-GPU systems, and GPU-backed WinML on
+RTX 3060 Ti, macOS 26.3.1 on an M4 Mac mini (Metal; own-user processes, not run as root), and
+bare-metal Linux with an amdgpu Radeon RX 9070-series card plus a Radeon 610M (DRM path).
+Not yet validated on hardware: bare-metal Linux with NVIDIA, multi-GPU systems, and GPU-backed WinML on
Copilot+ PCs. Planned work is listed under **Next** in the README.
diff --git a/README.md b/README.md
index bc61a3b..cbc21f7 100644
--- a/README.md
+++ b/README.md
@@ -2,8 +2,8 @@
**An `htop` for what's inside your GPU, not just how full it is.**
-Windows + NVIDIA/AMD, with NVIDIA support on Linux and WSL (see [Linux and WSL](#linux-and-wsl)) and
-macOS on Apple Silicon (see [macOS](#macos-apple-silicon)). A terminal UI
+Windows + NVIDIA/AMD, with Linux and WSL support (NVIDIA everywhere, AMD on native Linux; see
+[Linux and WSL](#linux-and-wsl)) and macOS on Apple Silicon (see [macOS](#macos-apple-silicon)). A terminal UI
that connects per-process GPU memory and engine activity with loaded runtimes, model files, and
LLM buffer breakdowns.
@@ -97,7 +97,8 @@ skips the consistency check, so do not use it after changing dependencies.
Tagged releases include portable Linux builds: `gputaskman-VERSION-linux-x64.tar.gz` and
`gputaskman-VERSION-linux-arm64.tar.gz`. They need glibc 2.35 or newer (Ubuntu 22.04, Debian 12,
-or newer) and an NVIDIA driver, and work natively, in containers, and in WSL:
+or newer) and an NVIDIA driver (or, on native Linux, an AMD GPU on the `amdgpu` driver), and work
+natively, in containers, and in WSL:
```bash
tar -xzf gputaskman-VERSION-linux-x64.tar.gz
@@ -150,6 +151,49 @@ An Ollama `qwen3:0.6b` runner was detected and named from Ollama's store with `a
evidence, its weights / KV-cache / compute split shown, and the TUI rendered. Bare-metal Linux
(same PID namespace, no matching needed) is covered by tests but has not been run on hardware.
+**AMD GPUs on native Linux** (the `amdgpu` driver) need no ROCm, AMD SMI, or extra package: the
+kernel already exposes what's needed.
+
+| | Source |
+|---|---|
+| Device VRAM, GTT, load, temperature, power, clocks, fan, PCIe link | `/sys/class/drm/card*/device` and its `hwmon` |
+| Per-process VRAM and GTT | `drm-resident-vram` / `drm-resident-gtt` (older kernels: `drm-memory-*`) and `drm-shared-*` in `/proc/<pid>/fdinfo/<fd>` |
+| Per-process engine load (gfx, compute, video) | `drm-engine-*` busy time, sampled twice; `n/a` for ROCm/HIP processes (see below) |
+
+GTT is GPU-accessible system RAM, so it fills the SPILL column (and `shared (in system RAM)` for
+the device), like WDDM's non-local usage. On an APU (an integrated GPU, detected by amdgpu not
+reporting a VRAM vendor, a heuristic validated on one APU) GTT is the GPU's normal memory and "VRAM"
+is only a RAM carve-out, so GTT is still shown but never flagged as spill, and an LLM's memory is its
+VRAM plus GTT. The SDMA engine (`dma`) fills the COPY column. Paged stays blank: there is no commit
+counter. A DRM client
+shared between processes (fd passing, fork) is counted once, for the lowest PID. A *buffer* shared
+between clients (a window a compositor imports, for example) is resident in each of them, and fdinfo
+does not say which one owns it. VRAM therefore shows each process's private bytes (resident minus
+shared), which add up without double counting; shared buffers stay in *unattributed* and are listed
+per process, owner unknown, in the inspector and JSON (`vram_shared_bytes`); shared GTT is left out
+of SPILL the same way. Older kernels whose amdgpu fdinfo has no `drm-shared-*` keys can't make that
+split, so per-process VRAM shows `n/a` there instead of double counting. amdgpu has no device-wide
+video engine counter, so the device's encode/decode load stays N/A; per-process video load is
+measured. On VCN 4 and later (RDNA3 and newer, including the RX 9070) decode and encode share one
+ring, so it is reported as `video` rather than split. Only processes you can inspect are listed; run with `sudo` to include other users'. `--gpu 1`
+selects the next card, such as an APU's integrated GPU next to a discrete one. A ROCm/HIP process
+(tagged `HIP`) has its VRAM in fdinfo like any other. Its GPU% is not: HIP submits compute through user-mode queues on `/dev/kfd`, which the kernel
+does not time per process, so a process holding `/dev/kfd` shows engines as `n/a` (`null` in JSON)
+rather than a measured-looking 0. The device's `AMD gfx` load does include that work.
+This path is native-Linux only; under WSL the GPU is `/dev/dxg`, with no DRM driver involved.
+
+Live check, bare-metal Arch-based Linux (kernel 7.2.15, `amdgpu`), Radeon RX 9070-series (Navi 48)
+plus a Radeon 610M iGPU: both cards were found, named, and read; VAAPI HEVC encode and decode with `ffmpeg`
+each showed as a `video` row at ~95-100% video load (on the RX 9070 both were charged to amdgpu's
+unified `enc` ring, hence `video`); desktop processes' private VRAM summed
+to less than the card's used VRAM, with the compositor's imported windows listed as shared.
+llama.cpp's ROCm server image (`ghcr.io/ggml-org/llama.cpp:server-rocm`, Qwen3-4B-Instruct-2507
+Q4_K_M, `-ngl 99 -c 8192`, next to a running game) showed as an `LLM` row with `llama.cpp+ggml+HIP`
+and 3.95 GB of private VRAM, matching the 2.49 GB of weights plus 1.21 GB of f16 KV cache plus compute
+buffers, and the 4.11 GB KFD reports for the process in `/sys/class/kfd/kfd/proc/<pid>/vram_*`.
+While it generated at 103 tok/s the device read 99% gfx and 339 W, but fdinfo had no engine lines
+for it, which is why ROCm rows show engines as `n/a`. PyTorch-ROCm has not been run on this path.
+
**WSL** is different, because Windows owns the GPU. Linux cannot see how much VRAM its own
processes use: `nvidia-smi` lists no processes, NVML's per-process samples carry *Windows* PIDs
(never looked up in Linux's `/proc`), and Windows charges all Linux GPU memory to the VM worker
@@ -501,6 +545,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:
| GGUF header, model-store manifests | ms, cached | Once per file / every 60 s |
| Local API probes | ms, local only | llama-server `/v1/models` only when its model path changes |
| `psutil.open_files()` | 0.6-2.5 s per process | Never |
+| AMD on Linux: DRM fdinfo scan (`readlink` every fd, read fdinfo of `/dev/dri` ones) | ~9 ms for ~8,000 fds (Linux desktop) | Every refresh, only when an amdgpu card is shown |
| macOS: `ioreg` device stats + Metal clients | ~10 ms | Every refresh |
| macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above |
| macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) |
@@ -627,6 +672,7 @@ downloads are bundled in this repository.
|---|---|
| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration |
| `linux.py` | Linux backend: NVML adapters and per-process usage, container PID matching, WSL detection, `/dev/dxg` holders |
+| `drm.py` | AMD on native Linux: amdgpu cards and device readings from sysfs, per-process VRAM/GTT/engine time from DRM fdinfo |
| `wslhost.py` | WSL: interop check, Windows host counter stream, adapter pairing, context matching |
| `macos.py` | macOS backend: IOKit device stats and per-process GPU time, `footprint` GPU memory poller, libproc memory maps |
| `nvml.py` | NVIDIA device readings |
@@ -644,7 +690,7 @@ downloads are bundled in this repository.
- Model identity for non-GGUF runtimes (ONNX Runtime, PyTorch, TensorRT) beyond installed candidates.
- Explicit provenance and availability for every counter and estimate.
- Reliable multi-GPU matching and per-device model placement.
-- Bare-metal Linux validation on hardware; AMD and Intel GPUs on Linux through DRM fdinfo.
+- Bare-metal Linux validation on NVIDIA hardware; Intel GPUs (`i915`/`xe`) through the same DRM fdinfo path as AMD.
- macOS: a portable build, Ollama/llama.cpp/MLX validation, and compressed GPU memory (footprint's
`swapped`) as the paged column.
- WSL: per-process GPU load for Linux rows, and Ollama's systemd journal as a log source.
diff --git a/src/gputaskman/app.py b/src/gputaskman/app.py
index 74cade3..8f934ae 100644
--- a/src/gputaskman/app.py
+++ b/src/gputaskman/app.py
@@ -20,7 +20,8 @@ from textual.worker import get_current_worker
from .collector import BACKEND, Collector, ProcRow, Snapshot
from .llm import gb
from .nvml import NvDevice, format_metric
-from .report import METAL_NOTE, WSL_EMPTY, has_engine_counters, has_wddm_counters, inferred_note, vram_label, wsl_note
+from .report import (DRM_UNSPLIT_NOTE, METAL_NOTE, WSL_EMPTY, has_engine_counters, has_spill_counters, has_wddm_counters,
+ inferred_note, vram_label, wsl_note)
KIND_STYLE = {
"LLM": "bold magenta", "decision": "bold orchid", "decision?": "orchid",
@@ -59,8 +60,8 @@ HELP_MD = """\
| Layer | Source | Works for |
|---|---|---|
-| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI | availability varies by GPU and driver |
-| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl) | workloads visible to WDDM |
+| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI; amdgpu sysfs on Linux | availability varies by GPU and driver |
+| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl); DRM fdinfo for amdgpu on Linux | workloads visible to WDDM or DRM |
| **Which runtime** (CUDA, HIP, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything |
| **Which model file** | memory-mapped files + command line | llama.cpp, stable-diffusion.cpp, LM Studio, Ollama |
| **Model name & kind** (LLM, decision, embedding, speech) | GGUF header, Ollama/LM Studio/Hugging Face stores, local server APIs | GGUF family |
@@ -78,7 +79,11 @@ GGUF kind is missing. It does not confirm the kind, GPU residency, or execution.
candidates remain unproven; engine activity is measured per process, not per model.
**Linux / WSL:** on native Linux, per-process VRAM and SM/encoder/decoder load come from NVML
-(no spill or commit counters exist there, so those columns stay blank). Under **WSL**, Windows
+(no spill or commit counters exist there, so those columns stay blank). **AMD GPUs on Linux**
+(`amdgpu`) use sysfs for the device and DRM fdinfo (`/proc/<pid>/fdinfo`) per process: VRAM,
+GTT (GPU memory in system RAM, shown as SPILL), and gfx/compute/video engine time (`n/a` for ROCm/HIP
+processes: their compute runs on `/dev/kfd` queues the kernel doesn't time per process). Only processes
+you can inspect are listed; run with `sudo` to see other users'. Under **WSL**, Windows
charges all Linux GPU memory to the VM (`vmwp.exe`), so VRAM and load are device-wide only; rows
are Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up share. When WSL
interop is enabled, the Windows host's counters are read through `powershell.exe`: **WSL VM**
@@ -472,8 +477,9 @@ class GpuTaskManager(App):
wddm = has_wddm_counters(s, r)
engines = has_engine_counters(s, r)
split = engines and s.backend != "metal" # Metal reports one GPU time, no per-engine split
- spill = Text(gb(r.gpu.shared) if wddm and r.gpu.shared >= 1 << 20 else "", justify="right",
- style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")
+ spill = Text(gb(r.gpu.shared) if has_spill_counters(s, r) and r.gpu.shared >= 1 << 20 else "", justify="right",
+ style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") and s.shared_is_spill
+ else "yellow")
paged = Text(gb(r.gpu.paged_out) if wddm and r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")
if wddm and "shared" not in r.gpu.available:
spill = Text("n/a", justify="right", style="grey62")
@@ -488,7 +494,7 @@ class GpuTaskManager(App):
pct(r.gpu.gpu_pct if engines else None),
pct(e.get("3d", 0) + e.get("compute", 0) if split else None),
pct(e.get("copy", 0) if split else None),
- pct(e.get("decode", 0) + e.get("encode", 0) if split else None),
+ pct(e.get("decode", 0) + e.get("encode", 0) + e.get("video", 0) if split else None),
Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),
key=r.key,
)
@@ -515,7 +521,8 @@ class GpuTaskManager(App):
wddm = has_wddm_counters(s, r)
engines = " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle"
hdr = Text.assemble(("VRAM ", "bold"), (vram_label(r.gpu), "bold"), " spill(RAM) ",
- (gb(r.gpu.shared) if wddm and "shared" in r.gpu.available else "n/a", "bold red" if r.gpu.shared > 256 << 20 else ""),
+ (gb(r.gpu.shared) if has_spill_counters(s, r) else "n/a",
+ "bold red" if r.gpu.shared > 256 << 20 and s.shared_is_spill else ""),
" paged(est) ", gb(r.gpu.paged_out) if wddm and r.gpu.paged_available else "n/a",
" engines ", engines if has_engine_counters(s, r) else "n/a")
if r.detail.runtimes:
@@ -529,10 +536,19 @@ class GpuTaskManager(App):
out.append(Text(inferred_note(r.gpu), style="grey62"))
elif s.wsl_split:
out.append(Text(f"No per-process VRAM: {s.wsl_split}.", style="grey62"))
+ elif s.backend == "drm":
+ out.append(Text(DRM_UNSPLIT_NOTE, style="grey62"))
elif s.backend == "metal":
out.append(Text("GPU memory n/a: footprint can't read other users' processes without root "
"(or hasn't measured this one yet). GPU% comes from Metal's per-process GPU time.",
style="grey62"))
+ if s.backend == "drm" and r.gpu.shared_vram >= 1 << 20:
+ out.append(Text(f"+ {gb(r.gpu.shared_vram)} in VRAM buffers shared with other processes (e.g. windows a "
+ "compositor imports), not counted above: the driver doesn't say which process owns a shared "
+ "buffer, so they stay in 'unattributed'.", style="grey62"))
+ if s.backend == "drm" and "engine" not in r.gpu.available:
+ out.append(Text("Engines n/a: this process uses ROCm/HIP (/dev/kfd), whose compute queues the kernel doesn't "
+ "time per process. The device's AMD gfx load includes its work.", style="grey62"))
if wddm and r.gpu.paged_out >= 64 << 20:
out.append(Text(f"{gb(r.gpu.paged_out)} of {gb(r.gpu.committed)} committed is not accounted for by local/non-local "
"usage. Estimated non-residency, not proof of eviction or disk paging.",
diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.py
index df3a4a1..00bf573 100644
--- a/src/gputaskman/cli.py
+++ b/src/gputaskman/cli.py
@@ -46,7 +46,7 @@ def main() -> None:
ap.error("--interval must be a finite number greater than zero")
if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):
- sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML) and macOS (Apple Silicon) only.")
+ sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu DRM) and macOS (Apple Silicon) only.")
if args.list_gpus:
from .collector import enumerate_adapters, ordered_adapters
diff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.py
index 5fe2f5e..b3a83e6 100644
--- a/src/gputaskman/collector.py
+++ b/src/gputaskman/collector.py
@@ -1,4 +1,4 @@
-"""Assembles one Snapshot from Windows counters, NVML, process inspection and LLM introspection."""
+"""Assembles one Snapshot from Windows counters, NVML or DRM, process inspection and LLM introspection."""
from __future__ import annotations
@@ -111,9 +111,11 @@ class Snapshot:
vram_used: int | None # WDDM adapter-wide dedicated usage
shared_used: int | None
procs: list[ProcRow] = field(default_factory=list)
- backend: str = "wddm" # wddm (Windows) | nvml (native Linux) | wsl (device-wide only) | metal (macOS)
+ backend: str = "wddm" # wddm (Windows) | nvml (native Linux, NVIDIA) | drm (native Linux, AMD) | wsl (device-wide only) | metal (macOS)
host_status: str = "" # WSL only: Windows host counters via interop -- ok | starting | off (...) | ...
wsl_split: str = "" # WSL only: why Linux rows have no (inferred) VRAM, when they don't
+ # False on an APU (amdgpu iGPU): its GTT is ordinary GPU memory, not spill from a full VRAM.
+ shared_is_spill: bool = True
@property
def host_data(self) -> bool:
@@ -172,6 +174,11 @@ class Collector:
DEEP_TOP_N = 25 # memory-map candidates: the top N VRAM users plus anything busy
_mac_driver = "" # "macOS 26.3.1" on the metal backend, read once at startup
SCAN_BUDGET = 0.15 # seconds of memory-map scanning per sample; leftovers wait for the next sample
+ drm_card = None # the selected GPU's amdgpu card, when it is one (drm.DrmCard)
+
+ @property
+ def backend(self) -> str:
+ return "drm" if self.drm_card is not None else BACKEND
def __init__(self, gpu_index: int = 0, scan_budget: float | None = SCAN_BUDGET, host: bool = True) -> None:
self.adapters = [a for a in enumerate_adapters() if not a.software]
@@ -181,6 +188,12 @@ class Collector:
self.amdsmi = AmdSmi(self.adapter, self.adapters)
self.adl = Adl(self.adapter, self.adapters)
self._mac_driver = f"macOS {macos_version()}".strip() if BACKEND == "metal" else ""
+ # Native Linux on an AMD GPU: device metrics from amdgpu's sysfs, processes from DRM fdinfo.
+ drm = getattr(self.counters, "drm", None)
+ if drm is not None and self.adapter and self.adapter.vendor_id != VENDOR_NVIDIA:
+ self.drm_card = next((c for c in drm.cards if c.key == self.adapter.luid), None)
+ if drm is not None and self.drm_card is None:
+ self.counters.drm = None # an NVIDIA GPU is shown: don't walk every process's fds for nothing
self.inspector = ProcInspector()
self.prober = ServerProber()
self.scan_budget = scan_budget # None = scan every candidate now (one-shot output)
@@ -222,6 +235,10 @@ class Collector:
a = adapters.get(luid)
if BACKEND == "metal":
nv = self.counters.device # IOKit's device stats in the shape the header renders (no NVML on macOS)
+ elif self.drm_card is not None:
+ from .drm import read_device
+
+ nv = read_device(self.drm_card, self.gpu_index)
else:
nvs = self.nvml.sample() if self.nvml.ok else []
nv = select_nv_device(self.adapter, nvs, self.gpu_index, self.adapters)
@@ -232,10 +249,12 @@ class Collector:
snap = Snapshot(
taken=time.time(), adapter=self.adapter, adapters=self.adapters, nv=nv,
driver=self._mac_driver or (self.amdsmi.driver if nv and nv.source == "AMD SMI"
- else self.nvml.driver if nv and nv.source == "NVML" else ""),
+ else self.nvml.driver if nv and nv.source == "NVML"
+ else f"kernel {os.uname().release}" if nv and nv.source == "AMD sysfs" else ""),
cuda=self.nvml.cuda_version if nv and nv.source == "NVML" else "", vram_total=total,
vram_used=int(a.dedicated) if a and a.dedicated is not None else None,
- shared_used=int(a.shared) if a and a.shared is not None else None, backend=BACKEND,
+ shared_used=int(a.shared) if a and a.shared is not None else None, backend=self.backend,
+ shared_is_spill=not (self.drm_card is not None and self.drm_card.integrated),
)
snap.host_status = self.host_status
@@ -280,7 +299,9 @@ class Collector:
own = self.prober.get(self.inspector.exclusive_ports(row.pid))
host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end process
log = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else None
- row.llm = build_view(row.detail, row.gpu.shown_vram or 0, row.gpu.shared,
+ # On an APU, GTT is the GPU's ordinary memory (often most of it), not spill: count it as held memory.
+ held = (row.gpu.shown_vram or 0) + (0 if snap.shared_is_spill else row.gpu.shared)
+ row.llm = build_view(row.detail, held, row.gpu.shared if snap.shared_is_spill else 0,
own or host, log, host=host if own else None,
attributed=row.gpu.shown_vram is not None)
diff --git a/src/gputaskman/drm.py b/src/gputaskman/drm.py
new file mode 100644
index 0000000..e62f1a8
--- /dev/null
+++ b/src/gputaskman/drm.py
@@ -0,0 +1,504 @@
+"""Linux AMD GPU backend: cards from sysfs and per-process memory/engine use from DRM fdinfo.
+
+The amdgpu driver exposes device-wide numbers (VRAM, clocks, temperature, power) through sysfs and charges
+each DRM client's resident memory and accumulated engine time to the fds that hold the render node open.
+/proc/<pid>/fdinfo lists those clients, so a process can be attributed its VRAM and the engine time that
+elapsed between two samples -- the Linux stand-in for WDDM's per-process counters.
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import time
+from dataclasses import dataclass
+from functools import lru_cache
+
+from .dxgi import VENDORS, Adapter
+from .linux import pci_key
+from .telemetry import DeviceTelemetry
+from .pdh import AdapterGpu, ProcGpu
+
+DRIVERS = ("amdgpu",)
+
+_SIZE_UNITS = {
+ "": 1,
+ "b": 1,
+ "kb": 1000,
+ "kib": 1024,
+ "mb": 1000**2,
+ "mib": 1024**2,
+ "gb": 1000**3,
+ "gib": 1024**3,
+ "tb": 1000**4,
+ "tib": 1024**4,
+}
+
+
+@dataclass
+class DrmCard:
+ key: int
+ pdev: str
+ sysfs: str
+ driver: str
+ vendor_id: int
+ device_id: int
+ name: str
+ vram_total: int
+ integrated: bool
+ # VCN 4+ (RDNA3 and later) runs decode and encode on one unified ring that fdinfo reports as "enc",
+ # so its busy time is video work of either kind, not encode.
+ unified_video: bool = False
+
+
+def _read(path: str) -> str | None:
+ """Contents of a sysfs/procfs file, stripped; None when it is missing or unreadable."""
+ try:
+ with open(path, encoding="utf-8", errors="replace") as f:
+ return f.read().strip()
+ except OSError:
+ return None
+
+
+def _int(path: str, base: int = 10) -> int | None:
+ value = _read(path)
+ if not value:
+ return None
+ try:
+ return int(value, base)
+ except ValueError:
+ return None
+
+
+def parse_size(value: str) -> int:
+ """'2060 KiB' -> bytes; a bare number is bytes; anything unparseable is 0."""
+ parts = (value or "").strip().split()
+ if not parts:
+ return 0
+ try:
+ number = int(parts[0])
+ except ValueError:
+ return 0
+ unit = parts[1].lower() if len(parts) > 1 else ""
+ multiplier = _SIZE_UNITS.get(unit)
+ return number * multiplier if multiplier is not None else 0
+
+
+def parse_fdinfo(text: str) -> dict | None:
+ """One /proc/<pid>/fdinfo/<fd> file; None when it isn't a DRM fd."""
+ fields: dict[str, str] = {}
+ for line in text.splitlines():
+ key, sep, value = line.partition(":")
+ if sep:
+ fields[key.strip()] = value.strip()
+ if "drm-driver" not in fields:
+ return None
+ try:
+ client_id = int(fields["drm-client-id"]) if "drm-client-id" in fields else None
+ except ValueError:
+ client_id = None
+
+ def memory(resident: str, legacy: str) -> int:
+ return parse_size(fields.get(resident, fields.get(legacy, "0")))
+
+ def shared(name: str) -> int | None:
+ return parse_size(fields[name]) if name in fields else None
+
+ engines: dict[str, int] = {}
+ for key, value in fields.items():
+ if not key.startswith("drm-engine-") or key.startswith("drm-engine-capacity-"):
+ continue
+ token = value.split()[0] if value.split() else "0"
+ try:
+ engines[key[len("drm-engine-"):]] = int(token)
+ except ValueError:
+ engines[key[len("drm-engine-"):]] = 0
+ return {
+ "driver": fields["drm-driver"],
+ "client_id": client_id,
+ "pdev": fields.get("drm-pdev", ""),
+ "vram": memory("drm-resident-vram", "drm-memory-vram"),
+ "gtt": memory("drm-resident-gtt", "drm-memory-gtt"),
+ # Buffers shared with other clients, also counted by every importer. None on older amdgpu
+ # (before the generic DRM memory stats), whose fdinfo can't split private from shared.
+ "shared_vram": shared("drm-shared-vram"),
+ "shared_gtt": shared("drm-shared-gtt"),
+ "cpu": memory("drm-resident-cpu", "drm-memory-cpu"),
+ "engines": engines,
+ }
+
+
+def _uevent(device: str) -> dict[str, str]:
+ out: dict[str, str] = {}
+ for line in (_read(os.path.join(device, "uevent")) or "").splitlines():
+ key, sep, value = line.partition("=")
+ if sep:
+ out[key.strip()] = value.strip()
+ return out
+
+
+def _driver(device: str) -> str:
+ try:
+ return os.path.basename(os.readlink(os.path.join(device, "driver")).rstrip("/"))
+ except OSError:
+ return _uevent(device).get("DRIVER", "")
+
+
+def _hex_id(path: str) -> int:
+ value = _read(path)
+ if not value:
+ return 0
+ try:
+ return int(value, 16)
+ except ValueError:
+ return 0
+
+
+@lru_cache(maxsize=4)
+def _pci_ids(path: str) -> dict[int, dict[int, str]]:
+ """{vendor id: {device id: name}} from pci.ids; subsystem lines (two tabs) are skipped."""
+ out: dict[int, dict[int, str]] = {}
+ vendor: int | None = None
+ try:
+ with open(path, encoding="utf-8", errors="replace") as f:
+ for line in f:
+ if not line.strip() or line.startswith("#"):
+ continue
+ if line.startswith("\t\t"):
+ continue
+ if line.startswith("\t"):
+ if vendor is None:
+ continue
+ parts = line.strip().split(None, 1)
+ if len(parts) != 2:
+ continue
+ try:
+ device = int(parts[0], 16)
+ except ValueError:
+ continue
+ out.setdefault(vendor, {})[device] = parts[1].strip()
+ continue
+ parts = line.split(None, 1)
+ vendor = None
+ if len(parts) == 2:
+ try:
+ vendor = int(parts[0], 16)
+ except ValueError:
+ pass
+ except OSError:
+ return out
+ return out
+
+
+def _vendor_prefix(vendor_id: int) -> str:
+ return VENDORS.get(vendor_id, hex(vendor_id))
+
+
+PCI_IDS = ("/usr/share/hwdata/pci.ids", "/usr/share/misc/pci.ids") # hwdata (Arch, Fedora); Debian/Ubuntu
+
+
+def _card_name(device: str, vendor_id: int, device_id: int, pci_ids: str | None) -> str:
+ product = _read(os.path.join(device, "product_name"))
+ if product:
+ return product
+ prefix = _vendor_prefix(vendor_id)
+ paths = (pci_ids,) if pci_ids else PCI_IDS
+ entry = next((e for e in (_pci_ids(p).get(vendor_id, {}).get(device_id) for p in paths) if e), None)
+ if not entry:
+ return f"{prefix} GPU {device_id:04x}"
+ if "[" in entry and "]" in entry:
+ inner = entry[entry.index("[") + 1:entry.rindex("]")]
+ if inner:
+ return f"{prefix} {inner}"
+ return f"{prefix} {entry}"
+
+
+def cards(root: str = "/sys/class/drm", pci_ids: str | None = None) -> list[DrmCard]:
+ """DRM cards backed by an amdgpu PCI device, discrete first then by VRAM desc."""
+ try:
+ entries = sorted(os.scandir(root), key=lambda e: e.name)
+ except OSError:
+ return []
+ found: list[DrmCard] = []
+ seen: set[str] = set()
+ for entry in entries:
+ if not re.fullmatch(r"card\d+", entry.name):
+ continue
+ device = os.path.join(entry.path, "device")
+ driver = _driver(device)
+ if driver not in DRIVERS:
+ continue
+ uevent = _uevent(device)
+ pdev = uevent.get("PCI_SLOT_NAME") or os.path.basename(os.path.realpath(device))
+ if pdev in seen:
+ continue
+ seen.add(pdev)
+ vendor_id = _hex_id(os.path.join(device, "vendor"))
+ device_id = _hex_id(os.path.join(device, "device"))
+ vram_total = _int(os.path.join(device, "mem_info_vram_total")) or 0
+ found.append(DrmCard(
+ key=pci_key(pdev),
+ pdev=pdev,
+ sysfs=device,
+ driver=driver,
+ vendor_id=vendor_id,
+ device_id=device_id,
+ name=_card_name(device, vendor_id, device_id, pci_ids),
+ vram_total=vram_total,
+ # amdgpu reports the VRAM chips' vendor only for a discrete card; an APU's "VRAM" is a RAM carve-out.
+ integrated=not os.path.exists(os.path.join(device, "mem_info_vram_vendor")),
+ unified_video=_vcn_major(device) >= 4,
+ ))
+ found.sort(key=lambda c: (c.integrated, -c.vram_total))
+ return found
+
+
+def adapters(found: list[DrmCard]) -> list[Adapter]:
+ """AMD cards as bare adapters (per-process data comes from DrmCounters)."""
+ return [
+ Adapter(
+ luid=c.key,
+ name=c.name,
+ vendor_id=c.vendor_id,
+ dedicated_vram=c.vram_total,
+ shared_system=_int(os.path.join(c.sysfs, "mem_info_gtt_total")) or 0,
+ software=False,
+ )
+ for c in found
+ ]
+
+
+def _hwmon_dir(device: str) -> str:
+ hwmon = os.path.join(device, "hwmon")
+ try:
+ names = sorted(os.listdir(hwmon))
+ except OSError:
+ return ""
+ return os.path.join(hwmon, names[0]) if names else ""
+
+
+def _milli(path: str) -> int | None:
+ value = _int(path)
+ return None if value is None else value // 1000
+
+
+def _microwatts(path: str) -> float | None:
+ value = _int(path)
+ return None if value is None else value / 1_000_000
+
+
+def _hwmon_temp(hwmon: str) -> int | None:
+ try:
+ names = sorted(os.listdir(hwmon))
+ except OSError:
+ names = []
+ for name in names:
+ if name.startswith("temp") and name.endswith("_label") and _read(os.path.join(hwmon, name)) == "edge":
+ return _milli(os.path.join(hwmon, f"{name[:-len('_label')]}_input"))
+ return _milli(os.path.join(hwmon, "temp1_input"))
+
+
+def _hwmon_clock(hwmon: str, label: str) -> int | None:
+ try:
+ names = sorted(os.listdir(hwmon))
+ except OSError:
+ names = []
+ for name in names:
+ if name.startswith("freq") and name.endswith("_label") and _read(os.path.join(hwmon, name)) == label:
+ value = _int(os.path.join(hwmon, f"{name[:-len('_label')]}_input"))
+ return None if value is None else value // 1_000_000
+ return None
+
+
+_PCIE_GEN = {2.5: 1, 5.0: 2, 8.0: 3, 16.0: 4, 32.0: 5, 64.0: 6}
+
+
+def read_device(card: DrmCard, index: int) -> DeviceTelemetry:
+ """NVIDIA-shaped device metrics for one AMD card, read from sysfs."""
+ d = card.sysfs
+ hwmon = _hwmon_dir(d)
+ dev = DeviceTelemetry(
+ index=index,
+ name=card.name,
+ pci_bus=card.pdev,
+ driver_model=card.driver,
+ source="AMD sysfs",
+ mem_total=_int(os.path.join(d, "mem_info_vram_total")),
+ mem_used=_int(os.path.join(d, "mem_info_vram_used")),
+ util_gpu=_int(os.path.join(d, "gpu_busy_percent")),
+ util_mem=_int(os.path.join(d, "mem_busy_percent")),
+ )
+ if hwmon:
+ dev.temp = _hwmon_temp(hwmon)
+ power = _microwatts(os.path.join(hwmon, "power1_average"))
+ if power is None:
+ power = _microwatts(os.path.join(hwmon, "power1_input"))
+ dev.power_w = power
+ if power is not None:
+ dev.power_kind = _read(os.path.join(hwmon, "power1_label")) or "" # amdgpu: "PPT" (package power)
+ dev.power_limit_w = _microwatts(os.path.join(hwmon, "power1_cap"))
+ fan_input = _int(os.path.join(hwmon, "fan1_input"))
+ fan_max = _int(os.path.join(hwmon, "fan1_max"))
+ if fan_input is not None and fan_max is not None and fan_max > 0:
+ dev.fan = int(round(100 * fan_input / fan_max))
+ dev.gfx_clock = _hwmon_clock(hwmon, "sclk")
+ dev.mem_clock = _hwmon_clock(hwmon, "mclk")
+ text = _read(os.path.join(d, "pp_dpm_sclk")) or ""
+ clocks = [int(v) for v in re.findall(r"(\d+)\s*Mhz", text, re.IGNORECASE)]
+ dev.sm_clock_max = max(clocks) if clocks else None
+ speed = _read(os.path.join(d, "current_link_speed"))
+ if speed:
+ try:
+ dev.pcie_gen = _PCIE_GEN.get(round(float(speed.split()[0]), 1))
+ except (ValueError, IndexError):
+ pass
+ dev.pcie_width = _int(os.path.join(d, "current_link_width"))
+ return dev
+
+
+def _vcn_major(device: str) -> int:
+ """Major version of the card's video block (VCN, listed as UVD in amdgpu's IP discovery); 0 if unknown."""
+ return _int(os.path.join(device, "ip_discovery", "die", "0", "UVD", "0", "major")) or 0
+
+
+def _engine_key(name: str, unified_video: bool = False) -> str:
+ lowered = name.lower()
+ if unified_video and ("enc" in lowered or "dec" in lowered):
+ return "video" # one ring for both: decode and encode can't be told apart
+ if lowered == "gfx":
+ return "3d"
+ if lowered == "compute":
+ return "compute"
+ if lowered == "enc":
+ return "encode"
+ if lowered == "dec":
+ return "decode"
+ if "dec" in lowered or "jpeg" in lowered:
+ return "decode"
+ if "enc" in lowered:
+ return "encode"
+ if lowered == "dma": # SDMA: buffer copies and uploads
+ return "copy"
+ return "other" # vpe, ...: busy time still counts toward GPU%, like WDDM's unnamed engines
+
+
+class DrmCounters:
+ """Same interface as pdh.GpuCounters: sample() -> ({adapter key: {pid: ProcGpu}}, {adapter key: AdapterGpu})."""
+
+ def __init__(self, found: list[DrmCard] | None = None, proc: str = "/proc",
+ clock=time.monotonic_ns) -> None:
+ self.cards = list(found) if found is not None else cards()
+ self.ok = bool(self.cards)
+ self._proc = proc
+ self._clock = clock
+ self._prev: dict[tuple, dict[str, int]] = {}
+ self._prev_ts: int | None = None
+ if self.ok:
+ self.sample() # engine load is a rate: take the baseline now, like the WDDM counters do
+
+ def _adapters(self) -> dict[int, AdapterGpu]:
+ out: dict[int, AdapterGpu] = {}
+ for card in self.cards:
+ vram = _int(os.path.join(card.sysfs, "mem_info_vram_used"))
+ gtt = _int(os.path.join(card.sysfs, "mem_info_gtt_used"))
+ out[card.key] = AdapterGpu( # an unreadable counter stays None (unknown), not 0
+ dedicated=float(vram) if vram is not None else None,
+ shared=float(gtt) if gtt is not None else None,
+ )
+ return out
+
+ def _clients(self) -> tuple[dict[tuple, tuple[int, int, dict]], set[int]]:
+ """One entry per DRM client: (lowest pid, lowest fd number, fdinfo); plus the pids holding /dev/kfd."""
+ by_pdev = {c.pdev: c.key for c in self.cards}
+ clients: dict[tuple, tuple[int, int, dict]] = {}
+ kfd: set[int] = set()
+ try:
+ pids = os.scandir(self._proc)
+ except OSError:
+ return clients, kfd
+ with pids:
+ for pid_entry in pids:
+ if not pid_entry.name.isdigit():
+ continue
+ pid = int(pid_entry.name)
+ try:
+ fds = os.scandir(os.path.join(pid_entry.path, "fd"))
+ except OSError:
+ continue
+ with fds:
+ for fd in fds:
+ try:
+ target = os.readlink(fd.path)
+ except OSError:
+ continue
+ if target == "/dev/kfd":
+ kfd.add(pid)
+ if not target.startswith("/dev/dri/"):
+ continue
+ try:
+ with open(os.path.join(pid_entry.path, "fdinfo", fd.name),
+ encoding="utf-8", errors="replace") as f:
+ info = parse_fdinfo(f.read())
+ except OSError:
+ continue
+ if not info or info["driver"] not in DRIVERS or info["pdev"] not in by_pdev:
+ continue
+ fd_number = int(fd.name) if fd.name.isdigit() else 0
+ client_id = info["client_id"]
+ if client_id is None:
+ key = (info["pdev"], pid, fd_number)
+ else:
+ key = (info["pdev"], client_id)
+ best = clients.get(key)
+ if best is None or (pid, fd_number) < (best[0], best[1]):
+ clients[key] = (pid, fd_number, info)
+ return clients, kfd
+
+ def sample(self) -> tuple[dict[int, dict[int, ProcGpu]], dict[int, AdapterGpu]]:
+ now = self._clock()
+ by_pdev = {c.pdev: c.key for c in self.cards}
+ unified = {c.pdev for c in self.cards if c.unified_video}
+ procs: dict[int, dict[int, ProcGpu]] = {c.key: {} for c in self.cards}
+ adapters = self._adapters()
+ clients, kfd = self._clients()
+ delta = now - self._prev_ts if self._prev_ts is not None else 0
+ previous: dict[tuple, dict[str, int]] = {}
+ for client_key, (pid, _fd, info) in clients.items():
+ key = by_pdev[info["pdev"]]
+ # No commit counter in fdinfo, so Paged stays unavailable rather than 0. ROCm/HIP (a /dev/kfd
+ # holder) submits compute through user-mode KFD queues the kernel doesn't time per process, so
+ # its engine load is unknown, not 0 (AGENTS.md rule 2); its VRAM is still in fdinfo.
+ available = {"dedicated", "shared"} if pid in kfd else {"dedicated", "shared", "engine"}
+ proc = procs[key].setdefault(pid, ProcGpu(attributed=True, available=available))
+ # A buffer shared between clients is resident in each of them and fdinfo doesn't say which owns it.
+ # Only private bytes are charged (AGENTS.md rule 1); shared ones are reported apart, owner unknown.
+ if info["shared_vram"] is None:
+ # Older amdgpu: resident bytes include every imported buffer, so summing them would count
+ # shared buffers once per importer. Unknown rather than double counted (rules 1 and 9).
+ proc.attributed = False
+ proc.available -= {"dedicated", "shared"}
+ else:
+ shared = float(info["shared_vram"])
+ proc.dedicated += max(float(info["vram"]) - shared, 0.0)
+ proc.shared_vram += shared
+ proc.shared += max(float(info["gtt"]) - float(info["shared_gtt"] or 0), 0.0)
+ old = self._prev.get(client_key)
+ engines: dict[str, float] = {}
+ for name, ns in info["engines"].items():
+ pct = 0.0
+ if old is not None and delta > 0 and name in old:
+ elapsed = ns - old[name]
+ if elapsed > 0:
+ pct = 100.0 * elapsed / delta
+ pct = min(100.0, max(0.0, pct))
+ engine = _engine_key(name, info["pdev"] in unified)
+ engines[engine] = engines.get(engine, 0.0) + pct
+ for engine, pct in engines.items():
+ proc.engines[engine] = min(100.0, proc.engines[engine] + pct)
+ previous[client_key] = dict(info["engines"])
+ self._prev = previous
+ self._prev_ts = now
+ return procs, adapters
+
+ def close(self) -> None:
+ """No handles are held."""
\ No newline at end of file
diff --git a/src/gputaskman/linux.py b/src/gputaskman/linux.py
index cf27143..a719e83 100644
--- a/src/gputaskman/linux.py
+++ b/src/gputaskman/linux.py
@@ -1,4 +1,4 @@
-"""Linux GPU backend: adapters and per-process usage from NVML, with an honest WSL mode.
+"""Linux GPU backend: adapters and per-process usage from NVML (and DRM fdinfo for AMD), with an honest WSL mode.
Native Linux: NVML attributes VRAM (running compute/graphics processes) and SM/encoder/decoder load
(process utilization samples) to Linux PIDs, which plays the role the WDDM counters play on Windows.
@@ -69,6 +69,17 @@ def _nvml_devices() -> list[tuple[int, object]]:
def enumerate_adapters() -> list[Adapter]:
+ """NVIDIA GPUs through NVML, plus (native Linux only) AMD GPUs through the amdgpu DRM driver."""
+ out = _nvml_adapters()
+ if not is_wsl():
+ from . import drm
+
+ seen = {a.luid for a in out}
+ out += [a for a in drm.adapters(drm.cards()) if a.luid not in seen]
+ return out
+
+
+def _nvml_adapters() -> list[Adapter]:
if N is None:
return []
try:
@@ -215,6 +226,14 @@ class LinuxCounters:
# {adapter key: {host pid: ProcGpu}}: NVML processes in another PID namespace that couldn't be
# matched to a local process. Their PIDs must never be looked up in this /proc.
self.outside: dict[int, dict[int, ProcGpu]] = {}
+ # AMD (amdgpu) GPUs: per-process VRAM, GTT and engine time from DRM fdinfo. Native Linux only;
+ # under WSL the GPU is /dev/dxg and no DRM driver is involved.
+ self.drm = None
+ if not self.wsl:
+ from .drm import DrmCounters
+
+ drm = DrmCounters()
+ self.drm = drm if drm.ok else None
if N is None:
return
try:
@@ -287,6 +306,10 @@ class LinuxCounters:
for key, _ in self._devices:
for pid in users:
procs[key][pid] = ProcGpu(attributed=False)
+ if self.drm is not None:
+ drm_procs, drm_adapters = self.drm.sample()
+ procs.update(drm_procs)
+ adapters.update(drm_adapters)
return procs, adapters
def _resolve_namespaces(self, procs: dict[int, dict[int, ProcGpu]]) -> None:
@@ -323,6 +346,8 @@ class LinuxCounters:
per.update(moved)
def close(self) -> None:
+ if self.drm is not None:
+ self.drm.close()
if self.ok:
try:
N.nvmlShutdown()
diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.py
index aa99ea6..ce40615 100644
--- a/src/gputaskman/pdh.py
+++ b/src/gputaskman/pdh.py
@@ -84,6 +84,9 @@ class ProcGpu:
# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.
inferred: float | None = None
inferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)
+ # amdgpu (DRM fdinfo): resident VRAM in buffers shared with other DRM clients, kept out of `dedicated`. fdinfo
+ # can't say which client owns a shared buffer, so it is shown apart and left in the adapter's unattributed part.
+ shared_vram: float = 0.0
@property
def shown_vram(self) -> float | None:
diff --git a/src/gputaskman/procinfo.py b/src/gputaskman/procinfo.py
index c3d09d7..e179ab6 100644
--- a/src/gputaskman/procinfo.py
+++ b/src/gputaskman/procinfo.py
@@ -438,7 +438,7 @@ def classify(d: ProcDetail, engines: dict[str, float]) -> str:
return "decision?"
if "llama.cpp" in d.runtimes or n.startswith(LLM_NAMES):
return "LLM"
- video = engines.get("decode", 0) + engines.get("encode", 0) > 1
+ video = engines.get("decode", 0) + engines.get("encode", 0) + engines.get("video", 0) > 1
if d.is_ai and not video:
return "AI"
if n in BROWSERS:
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index 9dfb586..48717d4 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -34,6 +34,11 @@ def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:
return s.backend == "wddm" or row.origin == "windows"
+def has_spill_counters(s: Snapshot, row: ProcRow) -> bool:
+ """Per-process GPU memory in system RAM: WDDM's non-local usage, or amdgpu's GTT on Linux."""
+ return (has_wddm_counters(s, row) or (s.backend == "drm" and not row.host)) and "shared" in row.gpu.available
+
+
def has_engine_counters(s: Snapshot, row: ProcRow) -> bool:
return (s.backend != "wsl" or row.origin == "windows") and "engine" in row.gpu.available
@@ -53,6 +58,11 @@ ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer
"gputaskman runs.")
+DRM_UNSPLIT_NOTE = ("VRAM n/a: this kernel's amdgpu fdinfo has no drm-shared-* keys, so resident memory includes "
+ "every buffer imported from other processes and would be counted once per importer. Newer "
+ "kernels report shared buffers separately.")
+
+
def inferred_note(g) -> str:
return ORDER_NOTE if g.inferred_by == "order" else INFERRED_NOTE
@@ -73,7 +83,8 @@ def to_dict(s: Snapshot) -> dict:
"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,
"vram_inferred_by": p.gpu.inferred_by or None,
"vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed and "dedicated" in p.gpu.available else None,
- "shared_bytes": int(p.gpu.shared) if wddm and "shared" in p.gpu.available else None,
+ "vram_shared_bytes": int(p.gpu.shared_vram) if s.backend == "drm" and not p.host else None,
+ "shared_bytes": int(p.gpu.shared) if has_spill_counters(s, p) else None,
"committed_bytes": int(p.gpu.committed) if wddm and "committed" in p.gpu.available else None,
"paged_out_bytes": int(p.gpu.paged_out) if wddm and p.gpu.paged_available else None,
"gpu_pct": round(p.gpu.gpu_pct, 1) if engines else None,
@@ -129,7 +140,7 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
used = gb(s.vram_used) if s.vram_used is not None else "N/A"
shared = gb(s.shared_used) if s.shared_used is not None else "N/A"
memory = f"VRAM {used} / {gb(s.vram_total)}"
- if s.backend == "wddm" or s.host_data:
+ if s.backend in ("wddm", "drm") or s.host_data:
memory += f" shared (in system RAM) {shared}"
out.append(memory)
if s.backend == "wsl":
@@ -155,7 +166,7 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
out.append(f"{'PID':>{w}} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")
for p in shown:
wddm = has_wddm_counters(s, p)
- spill = gb(p.gpu.shared) if wddm and p.gpu.shared >= 1 << 20 else ""
+ spill = gb(p.gpu.shared) if has_spill_counters(s, p) and p.gpu.shared >= 1 << 20 else ""
paged = gb(p.gpu.paged_out) if wddm and p.gpu.paged_out >= 1 << 20 else ""
if wddm and "shared" not in p.gpu.available:
spill = "N/A"
@@ -169,6 +180,8 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
out.append(INFERRED_NOTE)
if any(p.gpu.inferred_by == "order" for p in shown):
out.append(ORDER_NOTE)
+ if s.backend == "drm" and any(not p.host and not p.gpu.attributed for p in shown):
+ out.append(DRM_UNSPLIT_NOTE)
if s.backend == "wsl" and s.wsl_split and any(not p.host and not p.gpu.attributed and p.gpu.inferred is None
for p in s.procs):
out.append(f"Linux VRAM split: {s.wsl_split}")
diff --git a/tests/test_drm.py b/tests/test_drm.py
new file mode 100644
index 0000000..114049b
--- /dev/null
+++ b/tests/test_drm.py
@@ -0,0 +1,399 @@
+from __future__ import annotations
+
+import os
+import tempfile
+import unittest
+from unittest.mock import patch
+from pathlib import Path
+
+from gputaskman import drm
+
+GiB = 1 << 30
+
+FDINFO = """pos:\t0
+flags:\t02100002
+mnt_id:\t28
+ino:\t415
+drm-driver:\tamdgpu
+drm-client-id:\t26
+drm-pdev:\t0000:03:00.0
+pasid:\t25
+drm-total-cpu:\t0
+drm-shared-cpu:\t0
+drm-resident-cpu:\t0
+drm-purgeable-cpu:\t0
+drm-total-gtt:\t2 MiB
+drm-shared-gtt:\t0
+drm-resident-gtt:\t2060 KiB
+drm-purgeable-gtt:\t0
+drm-total-vram:\t12 KiB
+drm-shared-vram:\t0
+drm-resident-vram:\t0
+drm-purgeable-vram:\t0
+drm-memory-vram:\t0 KiB
+drm-memory-gtt: \t2060 KiB
+drm-memory-cpu: \t0 KiB
+amd-evicted-vram:\t12 KiB
+amd-requested-vram:\t12 KiB
+amd-requested-gtt:\t2048 KiB
+drm-engine-gfx:\t5308400172 ns
+drm-engine-compute:\t0 ns
+drm-engine-enc:\t1200 ns
+drm-engine-capacity-gfx:\t1
+"""
+
+
+def _make_card(root: Path, name, pdev, vendor, device, vram, driver="amdgpu", gtt=0, apu=False) -> Path:
+ dev = root / "drm" / name / "device"
+ dev.mkdir(parents=True)
+ (dev / "vendor").write_text(f"0x{vendor:04x}\n")
+ (dev / "device").write_text(f"0x{device:04x}\n")
+ (dev / "mem_info_vram_total").write_text(str(vram))
+ (dev / "mem_info_gtt_total").write_text(str(gtt))
+ if not apu:
+ (dev / "mem_info_vram_vendor").write_text("samsung\n") # only discrete cards have VRAM chips
+ driver_dir = root / "drivers" / driver
+ driver_dir.mkdir(parents=True, exist_ok=True)
+ os.symlink(driver_dir, dev / "driver")
+ (dev / "uevent").write_text(f"DRIVER={driver}\nPCI_SLOT_NAME={pdev}\n")
+ return dev
+
+
+def _card(dev, pdev, name="AMD GPU", vram=16 * GiB, vendor=0x1002, device=0x7550) -> drm.DrmCard:
+ return drm.DrmCard(
+ key=drm.pci_key(pdev), pdev=pdev, sysfs=str(dev), driver="amdgpu",
+ vendor_id=vendor, device_id=device, name=name, vram_total=vram,
+ integrated=vram <= 4 * GiB,
+ )
+
+
+def _write_fd(proc: Path, pid, fd, text, target="/dev/dri/renderD128") -> None:
+ """One process fd: the fdinfo contents plus the /proc/<pid>/fd symlink DrmCounters filters on."""
+ fdinfo = proc / str(pid) / "fdinfo"
+ fdinfo.mkdir(parents=True, exist_ok=True)
+ (fdinfo / str(fd)).write_text(text)
+ fds = proc / str(pid) / "fd"
+ fds.mkdir(parents=True, exist_ok=True)
+ link = fds / str(fd)
+ if not link.is_symlink():
+ os.symlink(target, link)
+
+
+def _fdinfo(client, pdev, gfx=0, vram=0, gtt=0, shared_vram=0) -> str:
+ return (
+ "pos:\t0\n"
+ "drm-driver:\tamdgpu\n"
+ f"drm-client-id:\t{client}\n"
+ f"drm-pdev:\t{pdev}\n"
+ f"drm-memory-vram:\t{vram} KiB\n"
+ f"drm-shared-vram:\t{shared_vram} KiB\n"
+ f"drm-memory-gtt: \t{gtt} KiB\n"
+ "drm-memory-cpu: \t0 KiB\n"
+ f"drm-engine-gfx:\t{gfx} ns\n"
+ f"drm-engine-capacity-gfx:\t1\n"
+ )
+
+
+class ParseTests(unittest.TestCase):
+ def test_parse_size(self):
+ self.assertEqual(drm.parse_size("2060 KiB"), 2060 * 1024)
+ self.assertEqual(drm.parse_size("2 MiB"), 2 * 1024**2)
+ self.assertEqual(drm.parse_size("1 GiB"), 1024**3)
+ self.assertEqual(drm.parse_size("12 KiB"), 12 * 1024)
+ self.assertEqual(drm.parse_size("0"), 0)
+ self.assertEqual(drm.parse_size("512"), 512)
+ self.assertEqual(drm.parse_size(" 4 KiB "), 4096)
+ self.assertEqual(drm.parse_size("bogus"), 0)
+ self.assertEqual(drm.parse_size("12 bananas"), 0)
+
+ def test_parse_fdinfo_example(self):
+ info = drm.parse_fdinfo(FDINFO)
+ self.assertIsNotNone(info)
+ self.assertEqual(info["driver"], "amdgpu")
+ self.assertEqual(info["client_id"], 26)
+ self.assertEqual(info["pdev"], "0000:03:00.0")
+ self.assertEqual(info["vram"], 0)
+ self.assertEqual(info["gtt"], 2060 * 1024)
+ self.assertEqual(info["cpu"], 0)
+ self.assertEqual(info["engines"], {"gfx": 5308400172, "compute": 0, "enc": 1200})
+
+ def test_parse_fdinfo_non_drm(self):
+ self.assertIsNone(drm.parse_fdinfo("pos:\t0\nflags:\t02\nmnt_id:\t28\n"))
+
+ def test_parse_fdinfo_resident_fallback(self):
+ info = drm.parse_fdinfo(
+ "drm-driver:\tamdgpu\n"
+ "drm-resident-vram:\t8 KiB\n"
+ "drm-resident-gtt:\t3 MiB\n"
+ "drm-resident-cpu:\t0\n"
+ )
+ self.assertIsNotNone(info)
+ self.assertEqual(info["vram"], 8 * 1024)
+ self.assertEqual(info["gtt"], 3 * 1024**2)
+ self.assertEqual(info["cpu"], 0)
+ self.assertEqual(info["engines"], {})
+ self.assertIsNone(info["client_id"])
+ self.assertIsNone(info["shared_vram"]) # no drm-shared-* keys: unknown, not 0
+
+ def test_parse_fdinfo_prefers_resident_keys(self):
+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-resident-vram:\t8 KiB\ndrm-memory-vram:\t9 KiB\n"
+ "drm-shared-vram:\t0\ndrm-shared-gtt:\t4 KiB\n")
+ self.assertEqual(info["vram"], 8 * 1024)
+ self.assertEqual(info["shared_vram"], 0)
+ self.assertEqual(info["shared_gtt"], 4 * 1024)
+
+ def test_parse_fdinfo_shared_vram(self):
+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-memory-vram:\t300 MiB\ndrm-shared-vram:\t200 MiB\n")
+ self.assertEqual(info["vram"], 300 * 1024**2)
+ self.assertEqual(info["shared_vram"], 200 * 1024**2) # part of vram, not on top of it
+
+
+PCI_IDS = """# fake
+1002 Advanced Micro Devices, Inc. [AMD/ATI]
+\t7550 Navi 48 [Radeon RX 9070/9070 XT/9070 GRE]
+\t\t7550 1234 Subsystem line that must be skipped
+\t13c0 Granite Ridge [Radeon 610M]
+1003 Somebody else
+\t9999 Not this vendor
+"""
+
+
+class CardsTests(unittest.TestCase):
+ def test_cards(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ root = Path(tmp)
+ # An APU with a large BIOS carve-out is still integrated: the flag doesn't depend on size.
+ _make_card(root, "card0", "0000:05:00.0", 0x1002, 0x13C0, 8 * GiB, gtt=3 * GiB, apu=True)
+ _make_card(root, "card1", "0000:03:00.0", 0x1002, 0x7550, 16 * GiB, gtt=32 * GiB)
+ _make_card(root, "card2", "0000:0a:00.0", 0x10DE, 0x9999, 24 * GiB, driver="nvidia")
+ for junk in ("card1-DP-1", "renderD128"):
+ (root / "drm" / junk).mkdir(parents=True)
+ pci_ids = root / "pci.ids"
+ pci_ids.write_text(PCI_IDS)
+
+ found = drm.cards(str(root / "drm"), str(pci_ids))
+
+ self.assertEqual([c.pdev for c in found], ["0000:03:00.0", "0000:05:00.0"])
+ self.assertEqual([c.name for c in found], [
+ "AMD Radeon RX 9070/9070 XT/9070 GRE",
+ "AMD Radeon 610M",
+ ])
+ self.assertFalse(found[0].integrated)
+ self.assertTrue(found[1].integrated)
+ self.assertEqual(found[0].vram_total, 16 * GiB)
+ self.assertEqual(found[0].key, drm.pci_key("0000:03:00.0"))
+ self.assertEqual(found[0].vendor_id, 0x1002)
+ self.assertEqual(found[0].device_id, 0x7550)
+
+ adapters = drm.adapters(found)
+ self.assertEqual([a.luid for a in adapters], [c.key for c in found])
+ self.assertEqual(adapters[0].dedicated_vram, 16 * GiB)
+ self.assertEqual(adapters[0].shared_system, 32 * GiB)
+ self.assertEqual(adapters[1].shared_system, 3 * GiB)
+ self.assertFalse(any(a.software for a in adapters))
+
+
+class ReadDeviceTests(unittest.TestCase):
+ def test_read_device(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ dev = Path(tmp) / "card0" / "device"
+ dev.mkdir(parents=True)
+ (dev / "mem_info_vram_total").write_text(str(16 * GiB))
+ (dev / "mem_info_vram_used").write_text(str(3 * GiB))
+ (dev / "gpu_busy_percent").write_text("42\n")
+ (dev / "mem_busy_percent").write_text("7\n")
+ (dev / "current_link_speed").write_text("32.0 GT/s PCIe\n")
+ (dev / "current_link_width").write_text("16\n")
+ (dev / "pp_dpm_sclk").write_text("0: 500Mhz\n1: 2100Mhz\n2: 2570Mhz *\nS: 44Mhz *\n")
+ hwmon = dev / "hwmon" / "hwmon0"
+ hwmon.mkdir(parents=True)
+ (hwmon / "temp1_label").write_text("edge\n")
+ (hwmon / "temp1_input").write_text("57000\n")
+ (hwmon / "temp2_label").write_text("junction\n")
+ (hwmon / "temp2_input").write_text("65000\n")
+ (hwmon / "power1_average").write_text("47000000\n")
+ (hwmon / "power1_label").write_text("PPT\n")
+ (hwmon / "power1_cap").write_text("340000000\n")
+ (hwmon / "freq1_label").write_text("sclk\n")
+ (hwmon / "freq1_input").write_text("27000000\n")
+ (hwmon / "freq2_label").write_text("mclk\n")
+ (hwmon / "freq2_input").write_text("1258000000\n")
+ (hwmon / "fan1_input").write_text("1000\n")
+ (hwmon / "fan1_max").write_text("2000\n")
+
+ card = _card(dev, "0000:03:00.0", name="AMD Radeon RX 9070")
+ d = drm.read_device(card, 0)
+
+ self.assertEqual(d.index, 0)
+ self.assertEqual(d.name, "AMD Radeon RX 9070")
+ self.assertEqual(d.pci_bus, "0000:03:00.0")
+ self.assertEqual(d.driver_model, "amdgpu")
+ self.assertEqual(d.mem_total, 16 * GiB)
+ self.assertEqual(d.mem_used, 3 * GiB)
+ self.assertEqual(d.util_gpu, 42)
+ self.assertEqual(d.util_mem, 7)
+ self.assertEqual(d.temp, 57)
+ self.assertEqual(d.power_w, 47.0)
+ self.assertEqual(d.power_limit_w, 340.0)
+ self.assertEqual(d.fan, 50)
+ self.assertEqual(d.gfx_clock, 27)
+ self.assertEqual(d.source, "AMD sysfs")
+ self.assertEqual(d.power_kind, "PPT")
+ self.assertEqual(d.mem_clock, 1258)
+ self.assertEqual(d.sm_clock_max, 2570)
+ self.assertEqual(d.pcie_gen, 5)
+ self.assertEqual(d.pcie_width, 16)
+ self.assertIsNone(d.enc)
+ self.assertIsNone(d.dec)
+
+
+class DrmCountersTests(unittest.TestCase):
+ def test_drm_counters(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ tmp = Path(tmp)
+ pdev = "0000:03:00.0"
+ dev = tmp / "drm" / "card1" / "device"
+ dev.mkdir(parents=True)
+ (dev / "mem_info_vram_total").write_text(str(16 * GiB))
+ (dev / "mem_info_vram_used").write_text(str(1 * GiB))
+ (dev / "mem_info_gtt_total").write_text(str(8 * GiB))
+ (dev / "mem_info_gtt_used").write_text(str(64 * 1024))
+ card = _card(dev, pdev)
+
+ proc = tmp / "proc"
+ _write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_000_000_000, vram=3, gtt=2, shared_vram=2))
+ _write_fd(proc, 200, 11, _fdinfo(26, pdev, gfx=1_000_000_000, vram=1, gtt=2))
+ _write_fd(proc, 300, 12, _fdinfo(27, "0000:ff:00.0", gfx=9_999_999_999, vram=4))
+ # DRM-looking fdinfo behind a non-DRM fd: the /proc/<pid>/fd target rules it out.
+ _write_fd(proc, 400, 13, _fdinfo(28, pdev, gfx=9_999_999_999, vram=4), target="socket:[123]")
+
+ times = iter([0, 1_000_000_000, 2_000_000_000]) # the constructor takes the baseline at 0
+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))
+
+ self.assertTrue(counters.ok)
+ procs, adapters = counters.sample()
+ self.assertEqual(list(procs), [card.key])
+ self.assertIn(100, procs[card.key])
+ self.assertNotIn(200, procs[card.key]) # same client, counted once, on the lowest pid
+ self.assertNotIn(300, procs[card.key]) # unknown pdev
+ self.assertNotIn(400, procs[card.key]) # non-DRM fd
+ self.assertEqual(procs[card.key][100].dedicated, 1024) # private only: 3 KiB resident - 2 KiB shared
+ self.assertEqual(procs[card.key][100].shared_vram, 2048)
+ self.assertFalse(procs[card.key][100].paged_available) # no commit counter: unavailable, not 0
+ self.assertEqual(procs[card.key][100].shared, 2048)
+ self.assertEqual(procs[card.key][100].committed, 0)
+ self.assertEqual(procs[card.key][100].engines.get("3d", 0.0), 0.0) # idle since the baseline
+ self.assertEqual(adapters[card.key].dedicated, 1 * GiB)
+ self.assertEqual(adapters[card.key].shared, 64 * 1024)
+
+ (dev / "mem_info_vram_used").write_text(str(2 * GiB))
+ _write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_500_000_000, vram=1, gtt=2))
+
+ procs, adapters = counters.sample()
+ self.assertEqual(procs[card.key][100].engines["3d"], 50.0)
+ self.assertIn("engine", procs[card.key][100].available)
+ self.assertEqual(adapters[card.key].dedicated, 2 * GiB)
+ counters.close()
+
+ def test_rocm_process_engine_load_is_unknown(self):
+ # A HIP process (llama.cpp-ROCm) holds /dev/kfd next to its render node: its VRAM is in fdinfo, but
+ # its compute runs on KFD user queues with no per-process busy time, so engines are unavailable.
+ with tempfile.TemporaryDirectory() as tmp:
+ tmp = Path(tmp)
+ pdev = "0000:03:00.0"
+ dev = tmp / "drm" / "card1" / "device"
+ dev.mkdir(parents=True)
+ card = _card(dev, pdev)
+ proc = tmp / "proc"
+ _write_fd(proc, 100, 3, "pos:\t0\nflags:\t02\n", target="/dev/kfd")
+ _write_fd(proc, 100, 7, _fdinfo(500, pdev, vram=3891436))
+ _write_fd(proc, 200, 7, _fdinfo(501, pdev, gfx=5, vram=4)) # a graphics client, for contrast
+ times = iter([0, 1_000_000_000])
+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))
+ procs, _ = counters.sample()
+ self.assertEqual(procs[card.key][100].dedicated, 3891436 * 1024)
+ self.assertNotIn("engine", procs[card.key][100].available)
+ self.assertIn("engine", procs[card.key][200].available)
+
+
+class DrmEdgeCaseTests(unittest.TestCase):
+ def _counters(self, tmp, fds, unified_video=False, sysfs=True):
+ pdev = "0000:03:00.0"
+ dev = tmp / "drm" / "card1" / "device"
+ dev.mkdir(parents=True)
+ if sysfs:
+ (dev / "mem_info_vram_used").write_text(str(GiB))
+ (dev / "mem_info_gtt_used").write_text("0")
+ card = _card(dev, pdev)
+ card.unified_video = unified_video
+ proc = tmp / "proc"
+ for pid, fd, text in fds:
+ _write_fd(proc, pid, fd, text.replace("PDEV", pdev))
+ times = iter([0, 1_000_000_000, 2_000_000_000])
+ return card, proc, drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))
+
+ def test_old_kernel_without_shared_keys_is_not_summed(self):
+ old = "drm-driver:\tamdgpu\ndrm-client-id:\t5\ndrm-pdev:\tPDEV\ndrm-memory-vram:\t4096 KiB\n"
+ with tempfile.TemporaryDirectory() as tmp:
+ card, _, counters = self._counters(Path(tmp), [(100, 5, old)])
+ g = counters.sample()[0][card.key][100]
+ self.assertFalse(g.attributed) # resident includes imports: would double count
+ self.assertEqual(g.dedicated, 0)
+ self.assertNotIn("dedicated", g.available)
+
+ def test_shared_gtt_left_out_of_spill(self):
+ text = _fdinfo(7, "PDEV", gtt=10) + "drm-shared-gtt:\t4 KiB\n"
+ with tempfile.TemporaryDirectory() as tmp:
+ card, _, counters = self._counters(Path(tmp), [(100, 5, text)])
+ self.assertEqual(counters.sample()[0][card.key][100].shared, 6 * 1024)
+
+ def test_unified_vcn_reports_video(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ text = "drm-driver:\tamdgpu\ndrm-client-id:\t9\ndrm-pdev:\tPDEV\ndrm-shared-vram:\t0\ndrm-engine-enc:\t{} ns\n"
+ card, proc, counters = self._counters(Path(tmp), [(100, 5, text.format(0))], unified_video=True)
+ (proc / "100" / "fdinfo" / "5").write_text(text.format(1_000_000_000).replace("PDEV", card.pdev))
+ engines = counters.sample()[0][card.key][100].engines
+ self.assertEqual(engines["video"], 100.0)
+ self.assertNotIn("encode", engines)
+
+ def test_missing_sysfs_memory_is_unknown(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, _, counters = self._counters(Path(tmp), [], sysfs=False)
+ a = counters.sample()[1][card.key]
+ self.assertIsNone(a.dedicated)
+ self.assertIsNone(a.shared)
+ self.assertIsNone(drm.read_device(card, 0).mem_used)
+
+ def test_vcn_major_from_ip_discovery(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ uvd = Path(tmp) / "ip_discovery" / "die" / "0" / "UVD" / "0"
+ uvd.mkdir(parents=True)
+ (uvd / "major").write_text("5\n")
+ self.assertEqual(drm._vcn_major(tmp), 5)
+ self.assertEqual(drm._vcn_major(str(Path(tmp) / "none")), 0)
+
+
+class EngineAndSelectionTests(unittest.TestCase):
+ def test_engine_names(self):
+ self.assertEqual(drm._engine_key("gfx"), "3d")
+ self.assertEqual(drm._engine_key("compute"), "compute")
+ self.assertEqual(drm._engine_key("dma"), "copy")
+ self.assertEqual(drm._engine_key("enc_1"), "encode")
+ self.assertEqual(drm._engine_key("jpeg"), "decode")
+ self.assertEqual(drm._engine_key("vpe"), "other")
+ self.assertEqual(drm._engine_key("enc", unified_video=True), "video")
+ self.assertEqual(drm._engine_key("gfx", unified_video=True), "3d")
+
+ def test_amd_selectable_next_to_nvidia_on_linux(self):
+ from gputaskman import collector
+ from gputaskman.dxgi import Adapter
+
+ nvidia = Adapter(1, "NVIDIA GPU", 0x10DE, 8 * GiB, 0, False)
+ amd = Adapter(2, "AMD Radeon 610M", 0x1002, 2 * GiB, 0, False)
+ with patch.object(collector, "BACKEND", "nvml"):
+ self.assertIs(collector.select_adapter([amd, nvidia], 0), nvidia)
+ self.assertIs(collector.select_adapter([amd, nvidia], 1), amd)
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tests/test_linux.py b/tests/test_linux.py
index 158e3a1..ad9419f 100644
--- a/tests/test_linux.py
+++ b/tests/test_linux.py
@@ -15,6 +15,16 @@ from gputaskman.procinfo import ProcDetail, _lib_name, _match_runtimes, classify
from gputaskman.report import to_dict, to_text
GiB = 1 << 30
+_no_drm = patch("gputaskman.drm.cards", return_value=[]) # keep the test machine's real AMD GPUs out of NVML tests
+
+
+def setUpModule() -> None:
+ _no_drm.start()
+
+
+def tearDownModule() -> None:
+ _no_drm.stop()
+
NOT_AVAILABLE = (1 << 64) - 1 # NVML_VALUE_NOT_AVAILABLE as an unsigned long long