A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
Add AMD GPU support on native Linux via amdgpu sysfs and DRM fdinfo
Pinta365571f3d0a14e0
diff --git a/AGENTS.md b/AGENTS.mdindex e471859..e1f252b 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -47,9 +47,9 @@ these rules:```cli.py ──► Collector.sample() (collector.py)├─ adapters dxgi.py (Windows) / linux.enumerate_adapters / macos.enumerate_adapters- ├─ counters pdh.py GpuCounters (Windows PDH) / linux.py LinuxCounters (NVML) /+ ├─ counters pdh.py GpuCounters (Windows PDH) / linux.py LinuxCounters (NVML, + drm.py for amdgpu) /│ macos.py MacCounters (IOKit Metal clients + footprint poller)- ├─ device metrics nvml.py (macOS: MacCounters.device from IOKit)+ ├─ device metrics nvml.py; drm.py read_device (AMD, Linux) (macOS: MacCounters.device from IOKit)├─ WSL host wslhost.py: HostStream (powershell PDH via interop), ContextTracker├─ processes procinfo.py: runtime DLLs, mapped files, cmdline, classify()├─ model naming stores.py (Ollama, LM Studio, Hugging Face), gguf.py (header, kind, sizes)@@ -60,7 +60,9 @@ cli.py diff ──► diff.py: compares two saved --json snapshots (no collectio```The backend is chosen at import time in `collector.py`: `BACKEND` is `wddm` on Windows, `nvml` on-native Linux, `wsl` under WSL, and `metal` on macOS. One-shot output (`--once`, `--json`) uses `scan_budget=None` to+native Linux, `wsl` under WSL, and `metal` on macOS. On native Linux, `Collector.backend` becomes `drm`+when the selected GPU is an amdgpu card (device readings from sysfs, processes from DRM fdinfo). One-shot+output (`--once`, `--json`) uses `scan_budget=None` toidentify everything immediately; the TUI spreads scans across refreshes.## Where to change things@@ -124,6 +126,7 @@ run builds artifacts without publishing.## Known limitsValidated live: Windows 11 with an RTX 4080 SUPER (WDDM, plus WSL2), a Linux container with an-RTX 3060 Ti, and macOS 26.3.1 on an M4 Mac mini (Metal; own-user processes, not run as root).-Not yet validated on hardware: bare-metal Linux, multi-GPU systems, and GPU-backed WinML on+RTX 3060 Ti, macOS 26.3.1 on an M4 Mac mini (Metal; own-user processes, not run as root), and+bare-metal Linux with an amdgpu Radeon RX 9070-series card plus a Radeon 610M (DRM path).+Not yet validated on hardware: bare-metal Linux with NVIDIA, multi-GPU systems, and GPU-backed WinML onCopilot+ PCs. Planned work is listed under **Next** in the README.diff --git a/README.md b/README.mdindex bc61a3b..cbc21f7 100644--- a/README.md+++ b/README.md@@ -2,8 +2,8 @@**An `htop` for what's inside your GPU, not just how full it is.**-Windows + NVIDIA/AMD, with NVIDIA support on Linux and WSL (see [Linux and WSL](#linux-and-wsl)) and-macOS on Apple Silicon (see [macOS](#macos-apple-silicon)). A terminal UI+Windows + NVIDIA/AMD, with Linux and WSL support (NVIDIA everywhere, AMD on native Linux; see+[Linux and WSL](#linux-and-wsl)) and macOS on Apple Silicon (see [macOS](#macos-apple-silicon)). A terminal UIthat connects per-process GPU memory and engine activity with loaded runtimes, model files, andLLM buffer breakdowns.@@ -97,7 +97,8 @@ skips the consistency check, so do not use it after changing dependencies.Tagged releases include portable Linux builds: `gputaskman-VERSION-linux-x64.tar.gz` and`gputaskman-VERSION-linux-arm64.tar.gz`. They need glibc 2.35 or newer (Ubuntu 22.04, Debian 12,-or newer) and an NVIDIA driver, and work natively, in containers, and in WSL:+or newer) and an NVIDIA driver (or, on native Linux, an AMD GPU on the `amdgpu` driver), and work+natively, in containers, and in WSL:```bashtar -xzf gputaskman-VERSION-linux-x64.tar.gz@@ -150,6 +151,49 @@ An Ollama `qwen3:0.6b` runner was detected and named from Ollama's store with `aevidence, its weights / KV-cache / compute split shown, and the TUI rendered. Bare-metal Linux(same PID namespace, no matching needed) is covered by tests but has not been run on hardware.+**AMD GPUs on native Linux** (the `amdgpu` driver) need no ROCm, AMD SMI, or extra package: the+kernel already exposes what's needed.++| | Source |+|---|---|+| Device VRAM, GTT, load, temperature, power, clocks, fan, PCIe link | `/sys/class/drm/card*/device` and its `hwmon` |+| Per-process VRAM and GTT | `drm-resident-vram` / `drm-resident-gtt` (older kernels: `drm-memory-*`) and `drm-shared-*` in `/proc/<pid>/fdinfo/<fd>` |+| Per-process engine load (gfx, compute, video) | `drm-engine-*` busy time, sampled twice; `n/a` for ROCm/HIP processes (see below) |++GTT is GPU-accessible system RAM, so it fills the SPILL column (and `shared (in system RAM)` for+the device), like WDDM's non-local usage. On an APU (an integrated GPU, detected by amdgpu not+reporting a VRAM vendor, a heuristic validated on one APU) GTT is the GPU's normal memory and "VRAM"+is only a RAM carve-out, so GTT is still shown but never flagged as spill, and an LLM's memory is its+VRAM plus GTT. The SDMA engine (`dma`) fills the COPY column. Paged stays blank: there is no commit+counter. A DRM client+shared between processes (fd passing, fork) is counted once, for the lowest PID. A *buffer* shared+between clients (a window a compositor imports, for example) is resident in each of them, and fdinfo+does not say which one owns it. VRAM therefore shows each process's private bytes (resident minus+shared), which add up without double counting; shared buffers stay in *unattributed* and are listed+per process, owner unknown, in the inspector and JSON (`vram_shared_bytes`); shared GTT is left out+of SPILL the same way. Older kernels whose amdgpu fdinfo has no `drm-shared-*` keys can't make that+split, so per-process VRAM shows `n/a` there instead of double counting. amdgpu has no device-wide+video engine counter, so the device's encode/decode load stays N/A; per-process video load is+measured. On VCN 4 and later (RDNA3 and newer, including the RX 9070) decode and encode share one+ring, so it is reported as `video` rather than split. Only processes you can inspect are listed; run with `sudo` to include other users'. `--gpu 1`+selects the next card, such as an APU's integrated GPU next to a discrete one. A ROCm/HIP process+(tagged `HIP`) has its VRAM in fdinfo like any other. Its GPU% is not: HIP submits compute through user-mode queues on `/dev/kfd`, which the kernel+does not time per process, so a process holding `/dev/kfd` shows engines as `n/a` (`null` in JSON)+rather than a measured-looking 0. The device's `AMD gfx` load does include that work.+This path is native-Linux only; under WSL the GPU is `/dev/dxg`, with no DRM driver involved.++Live check, bare-metal Arch-based Linux (kernel 7.2.15, `amdgpu`), Radeon RX 9070-series (Navi 48)+plus a Radeon 610M iGPU: both cards were found, named, and read; VAAPI HEVC encode and decode with `ffmpeg`+each showed as a `video` row at ~95-100% video load (on the RX 9070 both were charged to amdgpu's+unified `enc` ring, hence `video`); desktop processes' private VRAM summed+to less than the card's used VRAM, with the compositor's imported windows listed as shared.+llama.cpp's ROCm server image (`ghcr.io/ggml-org/llama.cpp:server-rocm`, Qwen3-4B-Instruct-2507+Q4_K_M, `-ngl 99 -c 8192`, next to a running game) showed as an `LLM` row with `llama.cpp+ggml+HIP`+and 3.95 GB of private VRAM, matching the 2.49 GB of weights plus 1.21 GB of f16 KV cache plus compute+buffers, and the 4.11 GB KFD reports for the process in `/sys/class/kfd/kfd/proc/<pid>/vram_*`.+While it generated at 103 tok/s the device read 99% gfx and 339 W, but fdinfo had no engine lines+for it, which is why ROCm rows show engines as `n/a`. PyTorch-ROCm has not been run on this path.+**WSL** is different, because Windows owns the GPU. Linux cannot see how much VRAM its ownprocesses use: `nvidia-smi` lists no processes, NVML's per-process samples carry *Windows* PIDs(never looked up in Linux's `/proc`), and Windows charges all Linux GPU memory to the VM worker@@ -501,6 +545,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:| GGUF header, model-store manifests | ms, cached | Once per file / every 60 s || Local API probes | ms, local only | llama-server `/v1/models` only when its model path changes || `psutil.open_files()` | 0.6-2.5 s per process | Never |+| AMD on Linux: DRM fdinfo scan (`readlink` every fd, read fdinfo of `/dev/dri` ones) | ~9 ms for ~8,000 fds (Linux desktop) | Every refresh, only when an amdgpu card is shown || macOS: `ioreg` device stats + Metal clients | ~10 ms | Every refresh || macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above || macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) |@@ -627,6 +672,7 @@ downloads are bundled in this repository.|---|---|| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration || `linux.py` | Linux backend: NVML adapters and per-process usage, container PID matching, WSL detection, `/dev/dxg` holders |+| `drm.py` | AMD on native Linux: amdgpu cards and device readings from sysfs, per-process VRAM/GTT/engine time from DRM fdinfo || `wslhost.py` | WSL: interop check, Windows host counter stream, adapter pairing, context matching || `macos.py` | macOS backend: IOKit device stats and per-process GPU time, `footprint` GPU memory poller, libproc memory maps || `nvml.py` | NVIDIA device readings |@@ -644,7 +690,7 @@ downloads are bundled in this repository.- Model identity for non-GGUF runtimes (ONNX Runtime, PyTorch, TensorRT) beyond installed candidates.- Explicit provenance and availability for every counter and estimate.- Reliable multi-GPU matching and per-device model placement.-- Bare-metal Linux validation on hardware; AMD and Intel GPUs on Linux through DRM fdinfo.+- Bare-metal Linux validation on NVIDIA hardware; Intel GPUs (`i915`/`xe`) through the same DRM fdinfo path as AMD.- macOS: a portable build, Ollama/llama.cpp/MLX validation, and compressed GPU memory (footprint's`swapped`) as the paged column.- WSL: per-process GPU load for Linux rows, and Ollama's systemd journal as a log source.diff --git a/src/gputaskman/app.py b/src/gputaskman/app.pyindex 74cade3..8f934ae 100644--- a/src/gputaskman/app.py+++ b/src/gputaskman/app.py@@ -20,7 +20,8 @@ from textual.worker import get_current_workerfrom .collector import BACKEND, Collector, ProcRow, Snapshotfrom .llm import gbfrom .nvml import NvDevice, format_metric-from .report import METAL_NOTE, WSL_EMPTY, has_engine_counters, has_wddm_counters, inferred_note, vram_label, wsl_note+from .report import (DRM_UNSPLIT_NOTE, METAL_NOTE, WSL_EMPTY, has_engine_counters, has_spill_counters, has_wddm_counters,+ inferred_note, vram_label, wsl_note)KIND_STYLE = {"LLM": "bold magenta", "decision": "bold orchid", "decision?": "orchid",@@ -59,8 +60,8 @@ HELP_MD = """\| Layer | Source | Works for ||---|---|---|-| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI | availability varies by GPU and driver |-| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl) | workloads visible to WDDM |+| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI; amdgpu sysfs on Linux | availability varies by GPU and driver |+| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl); DRM fdinfo for amdgpu on Linux | workloads visible to WDDM or DRM || **Which runtime** (CUDA, HIP, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything || **Which model file** | memory-mapped files + command line | llama.cpp, stable-diffusion.cpp, LM Studio, Ollama || **Model name & kind** (LLM, decision, embedding, speech) | GGUF header, Ollama/LM Studio/Hugging Face stores, local server APIs | GGUF family |@@ -78,7 +79,11 @@ GGUF kind is missing. It does not confirm the kind, GPU residency, or execution.candidates remain unproven; engine activity is measured per process, not per model.**Linux / WSL:** on native Linux, per-process VRAM and SM/encoder/decoder load come from NVML-(no spill or commit counters exist there, so those columns stay blank). Under **WSL**, Windows+(no spill or commit counters exist there, so those columns stay blank). **AMD GPUs on Linux**+(`amdgpu`) use sysfs for the device and DRM fdinfo (`/proc/<pid>/fdinfo`) per process: VRAM,+GTT (GPU memory in system RAM, shown as SPILL), and gfx/compute/video engine time (`n/a` for ROCm/HIP+processes: their compute runs on `/dev/kfd` queues the kernel doesn't time per process). Only processes+you can inspect are listed; run with `sudo` to see other users'. Under **WSL**, Windowscharges all Linux GPU memory to the VM (`vmwp.exe`), so VRAM and load are device-wide only; rowsare Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up share. When WSLinterop is enabled, the Windows host's counters are read through `powershell.exe`: **WSL VM**@@ -472,8 +477,9 @@ class GpuTaskManager(App):wddm = has_wddm_counters(s, r)engines = has_engine_counters(s, r)split = engines and s.backend != "metal" # Metal reports one GPU time, no per-engine split- spill = Text(gb(r.gpu.shared) if wddm and r.gpu.shared >= 1 << 20 else "", justify="right",- style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")+ spill = Text(gb(r.gpu.shared) if has_spill_counters(s, r) and r.gpu.shared >= 1 << 20 else "", justify="right",+ style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") and s.shared_is_spill+ else "yellow")paged = Text(gb(r.gpu.paged_out) if wddm and r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")if wddm and "shared" not in r.gpu.available:spill = Text("n/a", justify="right", style="grey62")@@ -488,7 +494,7 @@ class GpuTaskManager(App):pct(r.gpu.gpu_pct if engines else None),pct(e.get("3d", 0) + e.get("compute", 0) if split else None),pct(e.get("copy", 0) if split else None),- pct(e.get("decode", 0) + e.get("encode", 0) if split else None),+ pct(e.get("decode", 0) + e.get("encode", 0) + e.get("video", 0) if split else None),Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),key=r.key,)@@ -515,7 +521,8 @@ class GpuTaskManager(App):wddm = has_wddm_counters(s, r)engines = " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle"hdr = Text.assemble(("VRAM ", "bold"), (vram_label(r.gpu), "bold"), " spill(RAM) ",- (gb(r.gpu.shared) if wddm and "shared" in r.gpu.available else "n/a", "bold red" if r.gpu.shared > 256 << 20 else ""),+ (gb(r.gpu.shared) if has_spill_counters(s, r) else "n/a",+ "bold red" if r.gpu.shared > 256 << 20 and s.shared_is_spill else "")," paged(est) ", gb(r.gpu.paged_out) if wddm and r.gpu.paged_available else "n/a"," engines ", engines if has_engine_counters(s, r) else "n/a")if r.detail.runtimes:@@ -529,10 +536,19 @@ class GpuTaskManager(App):out.append(Text(inferred_note(r.gpu), style="grey62"))elif s.wsl_split:out.append(Text(f"No per-process VRAM: {s.wsl_split}.", style="grey62"))+ elif s.backend == "drm":+ out.append(Text(DRM_UNSPLIT_NOTE, style="grey62"))elif s.backend == "metal":out.append(Text("GPU memory n/a: footprint can't read other users' processes without root ""(or hasn't measured this one yet). GPU% comes from Metal's per-process GPU time.",style="grey62"))+ if s.backend == "drm" and r.gpu.shared_vram >= 1 << 20:+ out.append(Text(f"+ {gb(r.gpu.shared_vram)} in VRAM buffers shared with other processes (e.g. windows a "+ "compositor imports), not counted above: the driver doesn't say which process owns a shared "+ "buffer, so they stay in 'unattributed'.", style="grey62"))+ if s.backend == "drm" and "engine" not in r.gpu.available:+ out.append(Text("Engines n/a: this process uses ROCm/HIP (/dev/kfd), whose compute queues the kernel doesn't "+ "time per process. The device's AMD gfx load includes its work.", style="grey62"))if wddm and r.gpu.paged_out >= 64 << 20:out.append(Text(f"{gb(r.gpu.paged_out)} of {gb(r.gpu.committed)} committed is not accounted for by local/non-local ""usage. Estimated non-residency, not proof of eviction or disk paging.",diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.pyindex df3a4a1..00bf573 100644--- a/src/gputaskman/cli.py+++ b/src/gputaskman/cli.py@@ -46,7 +46,7 @@ def main() -> None:ap.error("--interval must be a finite number greater than zero")if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):- sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML) and macOS (Apple Silicon) only.")+ sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu DRM) and macOS (Apple Silicon) only.")if args.list_gpus:from .collector import enumerate_adapters, ordered_adaptersdiff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.pyindex 5fe2f5e..b3a83e6 100644--- a/src/gputaskman/collector.py+++ b/src/gputaskman/collector.py@@ -1,4 +1,4 @@-"""Assembles one Snapshot from Windows counters, NVML, process inspection and LLM introspection."""+"""Assembles one Snapshot from Windows counters, NVML or DRM, process inspection and LLM introspection."""from __future__ import annotations@@ -111,9 +111,11 @@ class Snapshot:vram_used: int | None # WDDM adapter-wide dedicated usageshared_used: int | Noneprocs: list[ProcRow] = field(default_factory=list)- backend: str = "wddm" # wddm (Windows) | nvml (native Linux) | wsl (device-wide only) | metal (macOS)+ backend: str = "wddm" # wddm (Windows) | nvml (native Linux, NVIDIA) | drm (native Linux, AMD) | wsl (device-wide only) | metal (macOS)host_status: str = "" # WSL only: Windows host counters via interop -- ok | starting | off (...) | ...wsl_split: str = "" # WSL only: why Linux rows have no (inferred) VRAM, when they don't+ # False on an APU (amdgpu iGPU): its GTT is ordinary GPU memory, not spill from a full VRAM.+ shared_is_spill: bool = True@propertydef host_data(self) -> bool:@@ -172,6 +174,11 @@ class Collector:DEEP_TOP_N = 25 # memory-map candidates: the top N VRAM users plus anything busy_mac_driver = "" # "macOS 26.3.1" on the metal backend, read once at startupSCAN_BUDGET = 0.15 # seconds of memory-map scanning per sample; leftovers wait for the next sample+ drm_card = None # the selected GPU's amdgpu card, when it is one (drm.DrmCard)++ @property+ def backend(self) -> str:+ return "drm" if self.drm_card is not None else BACKENDdef __init__(self, gpu_index: int = 0, scan_budget: float | None = SCAN_BUDGET, host: bool = True) -> None:self.adapters = [a for a in enumerate_adapters() if not a.software]@@ -181,6 +188,12 @@ class Collector:self.amdsmi = AmdSmi(self.adapter, self.adapters)self.adl = Adl(self.adapter, self.adapters)self._mac_driver = f"macOS {macos_version()}".strip() if BACKEND == "metal" else ""+ # Native Linux on an AMD GPU: device metrics from amdgpu's sysfs, processes from DRM fdinfo.+ drm = getattr(self.counters, "drm", None)+ if drm is not None and self.adapter and self.adapter.vendor_id != VENDOR_NVIDIA:+ self.drm_card = next((c for c in drm.cards if c.key == self.adapter.luid), None)+ if drm is not None and self.drm_card is None:+ self.counters.drm = None # an NVIDIA GPU is shown: don't walk every process's fds for nothingself.inspector = ProcInspector()self.prober = ServerProber()self.scan_budget = scan_budget # None = scan every candidate now (one-shot output)@@ -222,6 +235,10 @@ class Collector:a = adapters.get(luid)if BACKEND == "metal":nv = self.counters.device # IOKit's device stats in the shape the header renders (no NVML on macOS)+ elif self.drm_card is not None:+ from .drm import read_device++ nv = read_device(self.drm_card, self.gpu_index)else:nvs = self.nvml.sample() if self.nvml.ok else []nv = select_nv_device(self.adapter, nvs, self.gpu_index, self.adapters)@@ -232,10 +249,12 @@ class Collector:snap = Snapshot(taken=time.time(), adapter=self.adapter, adapters=self.adapters, nv=nv,driver=self._mac_driver or (self.amdsmi.driver if nv and nv.source == "AMD SMI"- else self.nvml.driver if nv and nv.source == "NVML" else ""),+ else self.nvml.driver if nv and nv.source == "NVML"+ else f"kernel {os.uname().release}" if nv and nv.source == "AMD sysfs" else ""),cuda=self.nvml.cuda_version if nv and nv.source == "NVML" else "", vram_total=total,vram_used=int(a.dedicated) if a and a.dedicated is not None else None,- shared_used=int(a.shared) if a and a.shared is not None else None, backend=BACKEND,+ shared_used=int(a.shared) if a and a.shared is not None else None, backend=self.backend,+ shared_is_spill=not (self.drm_card is not None and self.drm_card.integrated),)snap.host_status = self.host_status@@ -280,7 +299,9 @@ class Collector:own = self.prober.get(self.inspector.exclusive_ports(row.pid))host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end processlog = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else None- row.llm = build_view(row.detail, row.gpu.shown_vram or 0, row.gpu.shared,+ # On an APU, GTT is the GPU's ordinary memory (often most of it), not spill: count it as held memory.+ held = (row.gpu.shown_vram or 0) + (0 if snap.shared_is_spill else row.gpu.shared)+ row.llm = build_view(row.detail, held, row.gpu.shared if snap.shared_is_spill else 0,own or host, log, host=host if own else None,attributed=row.gpu.shown_vram is not None)diff --git a/src/gputaskman/drm.py b/src/gputaskman/drm.pynew file mode 100644index 0000000..e62f1a8--- /dev/null+++ b/src/gputaskman/drm.py@@ -0,0 +1,504 @@+"""Linux AMD GPU backend: cards from sysfs and per-process memory/engine use from DRM fdinfo.++The amdgpu driver exposes device-wide numbers (VRAM, clocks, temperature, power) through sysfs and charges+each DRM client's resident memory and accumulated engine time to the fds that hold the render node open.+/proc/<pid>/fdinfo lists those clients, so a process can be attributed its VRAM and the engine time that+elapsed between two samples -- the Linux stand-in for WDDM's per-process counters.+"""++from __future__ import annotations++import os+import re+import time+from dataclasses import dataclass+from functools import lru_cache++from .dxgi import VENDORS, Adapter+from .linux import pci_key+from .telemetry import DeviceTelemetry+from .pdh import AdapterGpu, ProcGpu++DRIVERS = ("amdgpu",)++_SIZE_UNITS = {+ "": 1,+ "b": 1,+ "kb": 1000,+ "kib": 1024,+ "mb": 1000**2,+ "mib": 1024**2,+ "gb": 1000**3,+ "gib": 1024**3,+ "tb": 1000**4,+ "tib": 1024**4,+}+++@dataclass+class DrmCard:+ key: int+ pdev: str+ sysfs: str+ driver: str+ vendor_id: int+ device_id: int+ name: str+ vram_total: int+ integrated: bool+ # VCN 4+ (RDNA3 and later) runs decode and encode on one unified ring that fdinfo reports as "enc",+ # so its busy time is video work of either kind, not encode.+ unified_video: bool = False+++def _read(path: str) -> str | None:+ """Contents of a sysfs/procfs file, stripped; None when it is missing or unreadable."""+ try:+ with open(path, encoding="utf-8", errors="replace") as f:+ return f.read().strip()+ except OSError:+ return None+++def _int(path: str, base: int = 10) -> int | None:+ value = _read(path)+ if not value:+ return None+ try:+ return int(value, base)+ except ValueError:+ return None+++def parse_size(value: str) -> int:+ """'2060 KiB' -> bytes; a bare number is bytes; anything unparseable is 0."""+ parts = (value or "").strip().split()+ if not parts:+ return 0+ try:+ number = int(parts[0])+ except ValueError:+ return 0+ unit = parts[1].lower() if len(parts) > 1 else ""+ multiplier = _SIZE_UNITS.get(unit)+ return number * multiplier if multiplier is not None else 0+++def parse_fdinfo(text: str) -> dict | None:+ """One /proc/<pid>/fdinfo/<fd> file; None when it isn't a DRM fd."""+ fields: dict[str, str] = {}+ for line in text.splitlines():+ key, sep, value = line.partition(":")+ if sep:+ fields[key.strip()] = value.strip()+ if "drm-driver" not in fields:+ return None+ try:+ client_id = int(fields["drm-client-id"]) if "drm-client-id" in fields else None+ except ValueError:+ client_id = None++ def memory(resident: str, legacy: str) -> int:+ return parse_size(fields.get(resident, fields.get(legacy, "0")))++ def shared(name: str) -> int | None:+ return parse_size(fields[name]) if name in fields else None++ engines: dict[str, int] = {}+ for key, value in fields.items():+ if not key.startswith("drm-engine-") or key.startswith("drm-engine-capacity-"):+ continue+ token = value.split()[0] if value.split() else "0"+ try:+ engines[key[len("drm-engine-"):]] = int(token)+ except ValueError:+ engines[key[len("drm-engine-"):]] = 0+ return {+ "driver": fields["drm-driver"],+ "client_id": client_id,+ "pdev": fields.get("drm-pdev", ""),+ "vram": memory("drm-resident-vram", "drm-memory-vram"),+ "gtt": memory("drm-resident-gtt", "drm-memory-gtt"),+ # Buffers shared with other clients, also counted by every importer. None on older amdgpu+ # (before the generic DRM memory stats), whose fdinfo can't split private from shared.+ "shared_vram": shared("drm-shared-vram"),+ "shared_gtt": shared("drm-shared-gtt"),+ "cpu": memory("drm-resident-cpu", "drm-memory-cpu"),+ "engines": engines,+ }+++def _uevent(device: str) -> dict[str, str]:+ out: dict[str, str] = {}+ for line in (_read(os.path.join(device, "uevent")) or "").splitlines():+ key, sep, value = line.partition("=")+ if sep:+ out[key.strip()] = value.strip()+ return out+++def _driver(device: str) -> str:+ try:+ return os.path.basename(os.readlink(os.path.join(device, "driver")).rstrip("/"))+ except OSError:+ return _uevent(device).get("DRIVER", "")+++def _hex_id(path: str) -> int:+ value = _read(path)+ if not value:+ return 0+ try:+ return int(value, 16)+ except ValueError:+ return 0+++@lru_cache(maxsize=4)+def _pci_ids(path: str) -> dict[int, dict[int, str]]:+ """{vendor id: {device id: name}} from pci.ids; subsystem lines (two tabs) are skipped."""+ out: dict[int, dict[int, str]] = {}+ vendor: int | None = None+ try:+ with open(path, encoding="utf-8", errors="replace") as f:+ for line in f:+ if not line.strip() or line.startswith("#"):+ continue+ if line.startswith("\t\t"):+ continue+ if line.startswith("\t"):+ if vendor is None:+ continue+ parts = line.strip().split(None, 1)+ if len(parts) != 2:+ continue+ try:+ device = int(parts[0], 16)+ except ValueError:+ continue+ out.setdefault(vendor, {})[device] = parts[1].strip()+ continue+ parts = line.split(None, 1)+ vendor = None+ if len(parts) == 2:+ try:+ vendor = int(parts[0], 16)+ except ValueError:+ pass+ except OSError:+ return out+ return out+++def _vendor_prefix(vendor_id: int) -> str:+ return VENDORS.get(vendor_id, hex(vendor_id))+++PCI_IDS = ("/usr/share/hwdata/pci.ids", "/usr/share/misc/pci.ids") # hwdata (Arch, Fedora); Debian/Ubuntu+++def _card_name(device: str, vendor_id: int, device_id: int, pci_ids: str | None) -> str:+ product = _read(os.path.join(device, "product_name"))+ if product:+ return product+ prefix = _vendor_prefix(vendor_id)+ paths = (pci_ids,) if pci_ids else PCI_IDS+ entry = next((e for e in (_pci_ids(p).get(vendor_id, {}).get(device_id) for p in paths) if e), None)+ if not entry:+ return f"{prefix} GPU {device_id:04x}"+ if "[" in entry and "]" in entry:+ inner = entry[entry.index("[") + 1:entry.rindex("]")]+ if inner:+ return f"{prefix} {inner}"+ return f"{prefix} {entry}"+++def cards(root: str = "/sys/class/drm", pci_ids: str | None = None) -> list[DrmCard]:+ """DRM cards backed by an amdgpu PCI device, discrete first then by VRAM desc."""+ try:+ entries = sorted(os.scandir(root), key=lambda e: e.name)+ except OSError:+ return []+ found: list[DrmCard] = []+ seen: set[str] = set()+ for entry in entries:+ if not re.fullmatch(r"card\d+", entry.name):+ continue+ device = os.path.join(entry.path, "device")+ driver = _driver(device)+ if driver not in DRIVERS:+ continue+ uevent = _uevent(device)+ pdev = uevent.get("PCI_SLOT_NAME") or os.path.basename(os.path.realpath(device))+ if pdev in seen:+ continue+ seen.add(pdev)+ vendor_id = _hex_id(os.path.join(device, "vendor"))+ device_id = _hex_id(os.path.join(device, "device"))+ vram_total = _int(os.path.join(device, "mem_info_vram_total")) or 0+ found.append(DrmCard(+ key=pci_key(pdev),+ pdev=pdev,+ sysfs=device,+ driver=driver,+ vendor_id=vendor_id,+ device_id=device_id,+ name=_card_name(device, vendor_id, device_id, pci_ids),+ vram_total=vram_total,+ # amdgpu reports the VRAM chips' vendor only for a discrete card; an APU's "VRAM" is a RAM carve-out.+ integrated=not os.path.exists(os.path.join(device, "mem_info_vram_vendor")),+ unified_video=_vcn_major(device) >= 4,+ ))+ found.sort(key=lambda c: (c.integrated, -c.vram_total))+ return found+++def adapters(found: list[DrmCard]) -> list[Adapter]:+ """AMD cards as bare adapters (per-process data comes from DrmCounters)."""+ return [+ Adapter(+ luid=c.key,+ name=c.name,+ vendor_id=c.vendor_id,+ dedicated_vram=c.vram_total,+ shared_system=_int(os.path.join(c.sysfs, "mem_info_gtt_total")) or 0,+ software=False,+ )+ for c in found+ ]+++def _hwmon_dir(device: str) -> str:+ hwmon = os.path.join(device, "hwmon")+ try:+ names = sorted(os.listdir(hwmon))+ except OSError:+ return ""+ return os.path.join(hwmon, names[0]) if names else ""+++def _milli(path: str) -> int | None:+ value = _int(path)+ return None if value is None else value // 1000+++def _microwatts(path: str) -> float | None:+ value = _int(path)+ return None if value is None else value / 1_000_000+++def _hwmon_temp(hwmon: str) -> int | None:+ try:+ names = sorted(os.listdir(hwmon))+ except OSError:+ names = []+ for name in names:+ if name.startswith("temp") and name.endswith("_label") and _read(os.path.join(hwmon, name)) == "edge":+ return _milli(os.path.join(hwmon, f"{name[:-len('_label')]}_input"))+ return _milli(os.path.join(hwmon, "temp1_input"))+++def _hwmon_clock(hwmon: str, label: str) -> int | None:+ try:+ names = sorted(os.listdir(hwmon))+ except OSError:+ names = []+ for name in names:+ if name.startswith("freq") and name.endswith("_label") and _read(os.path.join(hwmon, name)) == label:+ value = _int(os.path.join(hwmon, f"{name[:-len('_label')]}_input"))+ return None if value is None else value // 1_000_000+ return None+++_PCIE_GEN = {2.5: 1, 5.0: 2, 8.0: 3, 16.0: 4, 32.0: 5, 64.0: 6}+++def read_device(card: DrmCard, index: int) -> DeviceTelemetry:+ """NVIDIA-shaped device metrics for one AMD card, read from sysfs."""+ d = card.sysfs+ hwmon = _hwmon_dir(d)+ dev = DeviceTelemetry(+ index=index,+ name=card.name,+ pci_bus=card.pdev,+ driver_model=card.driver,+ source="AMD sysfs",+ mem_total=_int(os.path.join(d, "mem_info_vram_total")),+ mem_used=_int(os.path.join(d, "mem_info_vram_used")),+ util_gpu=_int(os.path.join(d, "gpu_busy_percent")),+ util_mem=_int(os.path.join(d, "mem_busy_percent")),+ )+ if hwmon:+ dev.temp = _hwmon_temp(hwmon)+ power = _microwatts(os.path.join(hwmon, "power1_average"))+ if power is None:+ power = _microwatts(os.path.join(hwmon, "power1_input"))+ dev.power_w = power+ if power is not None:+ dev.power_kind = _read(os.path.join(hwmon, "power1_label")) or "" # amdgpu: "PPT" (package power)+ dev.power_limit_w = _microwatts(os.path.join(hwmon, "power1_cap"))+ fan_input = _int(os.path.join(hwmon, "fan1_input"))+ fan_max = _int(os.path.join(hwmon, "fan1_max"))+ if fan_input is not None and fan_max is not None and fan_max > 0:+ dev.fan = int(round(100 * fan_input / fan_max))+ dev.gfx_clock = _hwmon_clock(hwmon, "sclk")+ dev.mem_clock = _hwmon_clock(hwmon, "mclk")+ text = _read(os.path.join(d, "pp_dpm_sclk")) or ""+ clocks = [int(v) for v in re.findall(r"(\d+)\s*Mhz", text, re.IGNORECASE)]+ dev.sm_clock_max = max(clocks) if clocks else None+ speed = _read(os.path.join(d, "current_link_speed"))+ if speed:+ try:+ dev.pcie_gen = _PCIE_GEN.get(round(float(speed.split()[0]), 1))+ except (ValueError, IndexError):+ pass+ dev.pcie_width = _int(os.path.join(d, "current_link_width"))+ return dev+++def _vcn_major(device: str) -> int:+ """Major version of the card's video block (VCN, listed as UVD in amdgpu's IP discovery); 0 if unknown."""+ return _int(os.path.join(device, "ip_discovery", "die", "0", "UVD", "0", "major")) or 0+++def _engine_key(name: str, unified_video: bool = False) -> str:+ lowered = name.lower()+ if unified_video and ("enc" in lowered or "dec" in lowered):+ return "video" # one ring for both: decode and encode can't be told apart+ if lowered == "gfx":+ return "3d"+ if lowered == "compute":+ return "compute"+ if lowered == "enc":+ return "encode"+ if lowered == "dec":+ return "decode"+ if "dec" in lowered or "jpeg" in lowered:+ return "decode"+ if "enc" in lowered:+ return "encode"+ if lowered == "dma": # SDMA: buffer copies and uploads+ return "copy"+ return "other" # vpe, ...: busy time still counts toward GPU%, like WDDM's unnamed engines+++class DrmCounters:+ """Same interface as pdh.GpuCounters: sample() -> ({adapter key: {pid: ProcGpu}}, {adapter key: AdapterGpu})."""++ def __init__(self, found: list[DrmCard] | None = None, proc: str = "/proc",+ clock=time.monotonic_ns) -> None:+ self.cards = list(found) if found is not None else cards()+ self.ok = bool(self.cards)+ self._proc = proc+ self._clock = clock+ self._prev: dict[tuple, dict[str, int]] = {}+ self._prev_ts: int | None = None+ if self.ok:+ self.sample() # engine load is a rate: take the baseline now, like the WDDM counters do++ def _adapters(self) -> dict[int, AdapterGpu]:+ out: dict[int, AdapterGpu] = {}+ for card in self.cards:+ vram = _int(os.path.join(card.sysfs, "mem_info_vram_used"))+ gtt = _int(os.path.join(card.sysfs, "mem_info_gtt_used"))+ out[card.key] = AdapterGpu( # an unreadable counter stays None (unknown), not 0+ dedicated=float(vram) if vram is not None else None,+ shared=float(gtt) if gtt is not None else None,+ )+ return out++ def _clients(self) -> tuple[dict[tuple, tuple[int, int, dict]], set[int]]:+ """One entry per DRM client: (lowest pid, lowest fd number, fdinfo); plus the pids holding /dev/kfd."""+ by_pdev = {c.pdev: c.key for c in self.cards}+ clients: dict[tuple, tuple[int, int, dict]] = {}+ kfd: set[int] = set()+ try:+ pids = os.scandir(self._proc)+ except OSError:+ return clients, kfd+ with pids:+ for pid_entry in pids:+ if not pid_entry.name.isdigit():+ continue+ pid = int(pid_entry.name)+ try:+ fds = os.scandir(os.path.join(pid_entry.path, "fd"))+ except OSError:+ continue+ with fds:+ for fd in fds:+ try:+ target = os.readlink(fd.path)+ except OSError:+ continue+ if target == "/dev/kfd":+ kfd.add(pid)+ if not target.startswith("/dev/dri/"):+ continue+ try:+ with open(os.path.join(pid_entry.path, "fdinfo", fd.name),+ encoding="utf-8", errors="replace") as f:+ info = parse_fdinfo(f.read())+ except OSError:+ continue+ if not info or info["driver"] not in DRIVERS or info["pdev"] not in by_pdev:+ continue+ fd_number = int(fd.name) if fd.name.isdigit() else 0+ client_id = info["client_id"]+ if client_id is None:+ key = (info["pdev"], pid, fd_number)+ else:+ key = (info["pdev"], client_id)+ best = clients.get(key)+ if best is None or (pid, fd_number) < (best[0], best[1]):+ clients[key] = (pid, fd_number, info)+ return clients, kfd++ def sample(self) -> tuple[dict[int, dict[int, ProcGpu]], dict[int, AdapterGpu]]:+ now = self._clock()+ by_pdev = {c.pdev: c.key for c in self.cards}+ unified = {c.pdev for c in self.cards if c.unified_video}+ procs: dict[int, dict[int, ProcGpu]] = {c.key: {} for c in self.cards}+ adapters = self._adapters()+ clients, kfd = self._clients()+ delta = now - self._prev_ts if self._prev_ts is not None else 0+ previous: dict[tuple, dict[str, int]] = {}+ for client_key, (pid, _fd, info) in clients.items():+ key = by_pdev[info["pdev"]]+ # No commit counter in fdinfo, so Paged stays unavailable rather than 0. ROCm/HIP (a /dev/kfd+ # holder) submits compute through user-mode KFD queues the kernel doesn't time per process, so+ # its engine load is unknown, not 0 (AGENTS.md rule 2); its VRAM is still in fdinfo.+ available = {"dedicated", "shared"} if pid in kfd else {"dedicated", "shared", "engine"}+ proc = procs[key].setdefault(pid, ProcGpu(attributed=True, available=available))+ # A buffer shared between clients is resident in each of them and fdinfo doesn't say which owns it.+ # Only private bytes are charged (AGENTS.md rule 1); shared ones are reported apart, owner unknown.+ if info["shared_vram"] is None:+ # Older amdgpu: resident bytes include every imported buffer, so summing them would count+ # shared buffers once per importer. Unknown rather than double counted (rules 1 and 9).+ proc.attributed = False+ proc.available -= {"dedicated", "shared"}+ else:+ shared = float(info["shared_vram"])+ proc.dedicated += max(float(info["vram"]) - shared, 0.0)+ proc.shared_vram += shared+ proc.shared += max(float(info["gtt"]) - float(info["shared_gtt"] or 0), 0.0)+ old = self._prev.get(client_key)+ engines: dict[str, float] = {}+ for name, ns in info["engines"].items():+ pct = 0.0+ if old is not None and delta > 0 and name in old:+ elapsed = ns - old[name]+ if elapsed > 0:+ pct = 100.0 * elapsed / delta+ pct = min(100.0, max(0.0, pct))+ engine = _engine_key(name, info["pdev"] in unified)+ engines[engine] = engines.get(engine, 0.0) + pct+ for engine, pct in engines.items():+ proc.engines[engine] = min(100.0, proc.engines[engine] + pct)+ previous[client_key] = dict(info["engines"])+ self._prev = previous+ self._prev_ts = now+ return procs, adapters++ def close(self) -> None:+ """No handles are held."""\ No newline at end of filediff --git a/src/gputaskman/linux.py b/src/gputaskman/linux.pyindex cf27143..a719e83 100644--- a/src/gputaskman/linux.py+++ b/src/gputaskman/linux.py@@ -1,4 +1,4 @@-"""Linux GPU backend: adapters and per-process usage from NVML, with an honest WSL mode.+"""Linux GPU backend: adapters and per-process usage from NVML (and DRM fdinfo for AMD), with an honest WSL mode.Native Linux: NVML attributes VRAM (running compute/graphics processes) and SM/encoder/decoder load(process utilization samples) to Linux PIDs, which plays the role the WDDM counters play on Windows.@@ -69,6 +69,17 @@ def _nvml_devices() -> list[tuple[int, object]]:def enumerate_adapters() -> list[Adapter]:+ """NVIDIA GPUs through NVML, plus (native Linux only) AMD GPUs through the amdgpu DRM driver."""+ out = _nvml_adapters()+ if not is_wsl():+ from . import drm++ seen = {a.luid for a in out}+ out += [a for a in drm.adapters(drm.cards()) if a.luid not in seen]+ return out+++def _nvml_adapters() -> list[Adapter]:if N is None:return []try:@@ -215,6 +226,14 @@ class LinuxCounters:# {adapter key: {host pid: ProcGpu}}: NVML processes in another PID namespace that couldn't be# matched to a local process. Their PIDs must never be looked up in this /proc.self.outside: dict[int, dict[int, ProcGpu]] = {}+ # AMD (amdgpu) GPUs: per-process VRAM, GTT and engine time from DRM fdinfo. Native Linux only;+ # under WSL the GPU is /dev/dxg and no DRM driver is involved.+ self.drm = None+ if not self.wsl:+ from .drm import DrmCounters++ drm = DrmCounters()+ self.drm = drm if drm.ok else Noneif N is None:returntry:@@ -287,6 +306,10 @@ class LinuxCounters:for key, _ in self._devices:for pid in users:procs[key][pid] = ProcGpu(attributed=False)+ if self.drm is not None:+ drm_procs, drm_adapters = self.drm.sample()+ procs.update(drm_procs)+ adapters.update(drm_adapters)return procs, adaptersdef _resolve_namespaces(self, procs: dict[int, dict[int, ProcGpu]]) -> None:@@ -323,6 +346,8 @@ class LinuxCounters:per.update(moved)def close(self) -> None:+ if self.drm is not None:+ self.drm.close()if self.ok:try:N.nvmlShutdown()diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.pyindex aa99ea6..ce40615 100644--- a/src/gputaskman/pdh.py+++ b/src/gputaskman/pdh.py@@ -84,6 +84,9 @@ class ProcGpu:# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.inferred: float | None = Noneinferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)+ # amdgpu (DRM fdinfo): resident VRAM in buffers shared with other DRM clients, kept out of `dedicated`. fdinfo+ # can't say which client owns a shared buffer, so it is shown apart and left in the adapter's unattributed part.+ shared_vram: float = 0.0@propertydef shown_vram(self) -> float | None:diff --git a/src/gputaskman/procinfo.py b/src/gputaskman/procinfo.pyindex c3d09d7..e179ab6 100644--- a/src/gputaskman/procinfo.py+++ b/src/gputaskman/procinfo.py@@ -438,7 +438,7 @@ def classify(d: ProcDetail, engines: dict[str, float]) -> str:return "decision?"if "llama.cpp" in d.runtimes or n.startswith(LLM_NAMES):return "LLM"- video = engines.get("decode", 0) + engines.get("encode", 0) > 1+ video = engines.get("decode", 0) + engines.get("encode", 0) + engines.get("video", 0) > 1if d.is_ai and not video:return "AI"if n in BROWSERS:diff --git a/src/gputaskman/report.py b/src/gputaskman/report.pyindex 9dfb586..48717d4 100644--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@ -34,6 +34,11 @@ def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:return s.backend == "wddm" or row.origin == "windows"+def has_spill_counters(s: Snapshot, row: ProcRow) -> bool:+ """Per-process GPU memory in system RAM: WDDM's non-local usage, or amdgpu's GTT on Linux."""+ return (has_wddm_counters(s, row) or (s.backend == "drm" and not row.host)) and "shared" in row.gpu.available++def has_engine_counters(s: Snapshot, row: ProcRow) -> bool:return (s.backend != "wsl" or row.origin == "windows") and "engine" in row.gpu.available@@ -53,6 +58,11 @@ ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer"gputaskman runs.")+DRM_UNSPLIT_NOTE = ("VRAM n/a: this kernel's amdgpu fdinfo has no drm-shared-* keys, so resident memory includes "+ "every buffer imported from other processes and would be counted once per importer. Newer "+ "kernels report shared buffers separately.")++def inferred_note(g) -> str:return ORDER_NOTE if g.inferred_by == "order" else INFERRED_NOTE@@ -73,7 +83,8 @@ def to_dict(s: Snapshot) -> dict:"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,"vram_inferred_by": p.gpu.inferred_by or None,"vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed and "dedicated" in p.gpu.available else None,- "shared_bytes": int(p.gpu.shared) if wddm and "shared" in p.gpu.available else None,+ "vram_shared_bytes": int(p.gpu.shared_vram) if s.backend == "drm" and not p.host else None,+ "shared_bytes": int(p.gpu.shared) if has_spill_counters(s, p) else None,"committed_bytes": int(p.gpu.committed) if wddm and "committed" in p.gpu.available else None,"paged_out_bytes": int(p.gpu.paged_out) if wddm and p.gpu.paged_available else None,"gpu_pct": round(p.gpu.gpu_pct, 1) if engines else None,@@ -129,7 +140,7 @@ def to_text(s: Snapshot, limit: int = 30) -> str:used = gb(s.vram_used) if s.vram_used is not None else "N/A"shared = gb(s.shared_used) if s.shared_used is not None else "N/A"memory = f"VRAM {used} / {gb(s.vram_total)}"- if s.backend == "wddm" or s.host_data:+ if s.backend in ("wddm", "drm") or s.host_data:memory += f" shared (in system RAM) {shared}"out.append(memory)if s.backend == "wsl":@@ -155,7 +166,7 @@ def to_text(s: Snapshot, limit: int = 30) -> str:out.append(f"{'PID':>{w}} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")for p in shown:wddm = has_wddm_counters(s, p)- spill = gb(p.gpu.shared) if wddm and p.gpu.shared >= 1 << 20 else ""+ spill = gb(p.gpu.shared) if has_spill_counters(s, p) and p.gpu.shared >= 1 << 20 else ""paged = gb(p.gpu.paged_out) if wddm and p.gpu.paged_out >= 1 << 20 else ""if wddm and "shared" not in p.gpu.available:spill = "N/A"@@ -169,6 +180,8 @@ def to_text(s: Snapshot, limit: int = 30) -> str:out.append(INFERRED_NOTE)if any(p.gpu.inferred_by == "order" for p in shown):out.append(ORDER_NOTE)+ if s.backend == "drm" and any(not p.host and not p.gpu.attributed for p in shown):+ out.append(DRM_UNSPLIT_NOTE)if s.backend == "wsl" and s.wsl_split and any(not p.host and not p.gpu.attributed and p.gpu.inferred is Nonefor p in s.procs):out.append(f"Linux VRAM split: {s.wsl_split}")diff --git a/tests/test_drm.py b/tests/test_drm.pynew file mode 100644index 0000000..114049b--- /dev/null+++ b/tests/test_drm.py@@ -0,0 +1,399 @@+from __future__ import annotations++import os+import tempfile+import unittest+from unittest.mock import patch+from pathlib import Path++from gputaskman import drm++GiB = 1 << 30++FDINFO = """pos:\t0+flags:\t02100002+mnt_id:\t28+ino:\t415+drm-driver:\tamdgpu+drm-client-id:\t26+drm-pdev:\t0000:03:00.0+pasid:\t25+drm-total-cpu:\t0+drm-shared-cpu:\t0+drm-resident-cpu:\t0+drm-purgeable-cpu:\t0+drm-total-gtt:\t2 MiB+drm-shared-gtt:\t0+drm-resident-gtt:\t2060 KiB+drm-purgeable-gtt:\t0+drm-total-vram:\t12 KiB+drm-shared-vram:\t0+drm-resident-vram:\t0+drm-purgeable-vram:\t0+drm-memory-vram:\t0 KiB+drm-memory-gtt: \t2060 KiB+drm-memory-cpu: \t0 KiB+amd-evicted-vram:\t12 KiB+amd-requested-vram:\t12 KiB+amd-requested-gtt:\t2048 KiB+drm-engine-gfx:\t5308400172 ns+drm-engine-compute:\t0 ns+drm-engine-enc:\t1200 ns+drm-engine-capacity-gfx:\t1+"""+++def _make_card(root: Path, name, pdev, vendor, device, vram, driver="amdgpu", gtt=0, apu=False) -> Path:+ dev = root / "drm" / name / "device"+ dev.mkdir(parents=True)+ (dev / "vendor").write_text(f"0x{vendor:04x}\n")+ (dev / "device").write_text(f"0x{device:04x}\n")+ (dev / "mem_info_vram_total").write_text(str(vram))+ (dev / "mem_info_gtt_total").write_text(str(gtt))+ if not apu:+ (dev / "mem_info_vram_vendor").write_text("samsung\n") # only discrete cards have VRAM chips+ driver_dir = root / "drivers" / driver+ driver_dir.mkdir(parents=True, exist_ok=True)+ os.symlink(driver_dir, dev / "driver")+ (dev / "uevent").write_text(f"DRIVER={driver}\nPCI_SLOT_NAME={pdev}\n")+ return dev+++def _card(dev, pdev, name="AMD GPU", vram=16 * GiB, vendor=0x1002, device=0x7550) -> drm.DrmCard:+ return drm.DrmCard(+ key=drm.pci_key(pdev), pdev=pdev, sysfs=str(dev), driver="amdgpu",+ vendor_id=vendor, device_id=device, name=name, vram_total=vram,+ integrated=vram <= 4 * GiB,+ )+++def _write_fd(proc: Path, pid, fd, text, target="/dev/dri/renderD128") -> None:+ """One process fd: the fdinfo contents plus the /proc/<pid>/fd symlink DrmCounters filters on."""+ fdinfo = proc / str(pid) / "fdinfo"+ fdinfo.mkdir(parents=True, exist_ok=True)+ (fdinfo / str(fd)).write_text(text)+ fds = proc / str(pid) / "fd"+ fds.mkdir(parents=True, exist_ok=True)+ link = fds / str(fd)+ if not link.is_symlink():+ os.symlink(target, link)+++def _fdinfo(client, pdev, gfx=0, vram=0, gtt=0, shared_vram=0) -> str:+ return (+ "pos:\t0\n"+ "drm-driver:\tamdgpu\n"+ f"drm-client-id:\t{client}\n"+ f"drm-pdev:\t{pdev}\n"+ f"drm-memory-vram:\t{vram} KiB\n"+ f"drm-shared-vram:\t{shared_vram} KiB\n"+ f"drm-memory-gtt: \t{gtt} KiB\n"+ "drm-memory-cpu: \t0 KiB\n"+ f"drm-engine-gfx:\t{gfx} ns\n"+ f"drm-engine-capacity-gfx:\t1\n"+ )+++class ParseTests(unittest.TestCase):+ def test_parse_size(self):+ self.assertEqual(drm.parse_size("2060 KiB"), 2060 * 1024)+ self.assertEqual(drm.parse_size("2 MiB"), 2 * 1024**2)+ self.assertEqual(drm.parse_size("1 GiB"), 1024**3)+ self.assertEqual(drm.parse_size("12 KiB"), 12 * 1024)+ self.assertEqual(drm.parse_size("0"), 0)+ self.assertEqual(drm.parse_size("512"), 512)+ self.assertEqual(drm.parse_size(" 4 KiB "), 4096)+ self.assertEqual(drm.parse_size("bogus"), 0)+ self.assertEqual(drm.parse_size("12 bananas"), 0)++ def test_parse_fdinfo_example(self):+ info = drm.parse_fdinfo(FDINFO)+ self.assertIsNotNone(info)+ self.assertEqual(info["driver"], "amdgpu")+ self.assertEqual(info["client_id"], 26)+ self.assertEqual(info["pdev"], "0000:03:00.0")+ self.assertEqual(info["vram"], 0)+ self.assertEqual(info["gtt"], 2060 * 1024)+ self.assertEqual(info["cpu"], 0)+ self.assertEqual(info["engines"], {"gfx": 5308400172, "compute": 0, "enc": 1200})++ def test_parse_fdinfo_non_drm(self):+ self.assertIsNone(drm.parse_fdinfo("pos:\t0\nflags:\t02\nmnt_id:\t28\n"))++ def test_parse_fdinfo_resident_fallback(self):+ info = drm.parse_fdinfo(+ "drm-driver:\tamdgpu\n"+ "drm-resident-vram:\t8 KiB\n"+ "drm-resident-gtt:\t3 MiB\n"+ "drm-resident-cpu:\t0\n"+ )+ self.assertIsNotNone(info)+ self.assertEqual(info["vram"], 8 * 1024)+ self.assertEqual(info["gtt"], 3 * 1024**2)+ self.assertEqual(info["cpu"], 0)+ self.assertEqual(info["engines"], {})+ self.assertIsNone(info["client_id"])+ self.assertIsNone(info["shared_vram"]) # no drm-shared-* keys: unknown, not 0++ def test_parse_fdinfo_prefers_resident_keys(self):+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-resident-vram:\t8 KiB\ndrm-memory-vram:\t9 KiB\n"+ "drm-shared-vram:\t0\ndrm-shared-gtt:\t4 KiB\n")+ self.assertEqual(info["vram"], 8 * 1024)+ self.assertEqual(info["shared_vram"], 0)+ self.assertEqual(info["shared_gtt"], 4 * 1024)++ def test_parse_fdinfo_shared_vram(self):+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-memory-vram:\t300 MiB\ndrm-shared-vram:\t200 MiB\n")+ self.assertEqual(info["vram"], 300 * 1024**2)+ self.assertEqual(info["shared_vram"], 200 * 1024**2) # part of vram, not on top of it+++PCI_IDS = """# fake+1002 Advanced Micro Devices, Inc. [AMD/ATI]+\t7550 Navi 48 [Radeon RX 9070/9070 XT/9070 GRE]+\t\t7550 1234 Subsystem line that must be skipped+\t13c0 Granite Ridge [Radeon 610M]+1003 Somebody else+\t9999 Not this vendor+"""+++class CardsTests(unittest.TestCase):+ def test_cards(self):+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ # An APU with a large BIOS carve-out is still integrated: the flag doesn't depend on size.+ _make_card(root, "card0", "0000:05:00.0", 0x1002, 0x13C0, 8 * GiB, gtt=3 * GiB, apu=True)+ _make_card(root, "card1", "0000:03:00.0", 0x1002, 0x7550, 16 * GiB, gtt=32 * GiB)+ _make_card(root, "card2", "0000:0a:00.0", 0x10DE, 0x9999, 24 * GiB, driver="nvidia")+ for junk in ("card1-DP-1", "renderD128"):+ (root / "drm" / junk).mkdir(parents=True)+ pci_ids = root / "pci.ids"+ pci_ids.write_text(PCI_IDS)++ found = drm.cards(str(root / "drm"), str(pci_ids))++ self.assertEqual([c.pdev for c in found], ["0000:03:00.0", "0000:05:00.0"])+ self.assertEqual([c.name for c in found], [+ "AMD Radeon RX 9070/9070 XT/9070 GRE",+ "AMD Radeon 610M",+ ])+ self.assertFalse(found[0].integrated)+ self.assertTrue(found[1].integrated)+ self.assertEqual(found[0].vram_total, 16 * GiB)+ self.assertEqual(found[0].key, drm.pci_key("0000:03:00.0"))+ self.assertEqual(found[0].vendor_id, 0x1002)+ self.assertEqual(found[0].device_id, 0x7550)++ adapters = drm.adapters(found)+ self.assertEqual([a.luid for a in adapters], [c.key for c in found])+ self.assertEqual(adapters[0].dedicated_vram, 16 * GiB)+ self.assertEqual(adapters[0].shared_system, 32 * GiB)+ self.assertEqual(adapters[1].shared_system, 3 * GiB)+ self.assertFalse(any(a.software for a in adapters))+++class ReadDeviceTests(unittest.TestCase):+ def test_read_device(self):+ with tempfile.TemporaryDirectory() as tmp:+ dev = Path(tmp) / "card0" / "device"+ dev.mkdir(parents=True)+ (dev / "mem_info_vram_total").write_text(str(16 * GiB))+ (dev / "mem_info_vram_used").write_text(str(3 * GiB))+ (dev / "gpu_busy_percent").write_text("42\n")+ (dev / "mem_busy_percent").write_text("7\n")+ (dev / "current_link_speed").write_text("32.0 GT/s PCIe\n")+ (dev / "current_link_width").write_text("16\n")+ (dev / "pp_dpm_sclk").write_text("0: 500Mhz\n1: 2100Mhz\n2: 2570Mhz *\nS: 44Mhz *\n")+ hwmon = dev / "hwmon" / "hwmon0"+ hwmon.mkdir(parents=True)+ (hwmon / "temp1_label").write_text("edge\n")+ (hwmon / "temp1_input").write_text("57000\n")+ (hwmon / "temp2_label").write_text("junction\n")+ (hwmon / "temp2_input").write_text("65000\n")+ (hwmon / "power1_average").write_text("47000000\n")+ (hwmon / "power1_label").write_text("PPT\n")+ (hwmon / "power1_cap").write_text("340000000\n")+ (hwmon / "freq1_label").write_text("sclk\n")+ (hwmon / "freq1_input").write_text("27000000\n")+ (hwmon / "freq2_label").write_text("mclk\n")+ (hwmon / "freq2_input").write_text("1258000000\n")+ (hwmon / "fan1_input").write_text("1000\n")+ (hwmon / "fan1_max").write_text("2000\n")++ card = _card(dev, "0000:03:00.0", name="AMD Radeon RX 9070")+ d = drm.read_device(card, 0)++ self.assertEqual(d.index, 0)+ self.assertEqual(d.name, "AMD Radeon RX 9070")+ self.assertEqual(d.pci_bus, "0000:03:00.0")+ self.assertEqual(d.driver_model, "amdgpu")+ self.assertEqual(d.mem_total, 16 * GiB)+ self.assertEqual(d.mem_used, 3 * GiB)+ self.assertEqual(d.util_gpu, 42)+ self.assertEqual(d.util_mem, 7)+ self.assertEqual(d.temp, 57)+ self.assertEqual(d.power_w, 47.0)+ self.assertEqual(d.power_limit_w, 340.0)+ self.assertEqual(d.fan, 50)+ self.assertEqual(d.gfx_clock, 27)+ self.assertEqual(d.source, "AMD sysfs")+ self.assertEqual(d.power_kind, "PPT")+ self.assertEqual(d.mem_clock, 1258)+ self.assertEqual(d.sm_clock_max, 2570)+ self.assertEqual(d.pcie_gen, 5)+ self.assertEqual(d.pcie_width, 16)+ self.assertIsNone(d.enc)+ self.assertIsNone(d.dec)+++class DrmCountersTests(unittest.TestCase):+ def test_drm_counters(self):+ with tempfile.TemporaryDirectory() as tmp:+ tmp = Path(tmp)+ pdev = "0000:03:00.0"+ dev = tmp / "drm" / "card1" / "device"+ dev.mkdir(parents=True)+ (dev / "mem_info_vram_total").write_text(str(16 * GiB))+ (dev / "mem_info_vram_used").write_text(str(1 * GiB))+ (dev / "mem_info_gtt_total").write_text(str(8 * GiB))+ (dev / "mem_info_gtt_used").write_text(str(64 * 1024))+ card = _card(dev, pdev)++ proc = tmp / "proc"+ _write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_000_000_000, vram=3, gtt=2, shared_vram=2))+ _write_fd(proc, 200, 11, _fdinfo(26, pdev, gfx=1_000_000_000, vram=1, gtt=2))+ _write_fd(proc, 300, 12, _fdinfo(27, "0000:ff:00.0", gfx=9_999_999_999, vram=4))+ # DRM-looking fdinfo behind a non-DRM fd: the /proc/<pid>/fd target rules it out.+ _write_fd(proc, 400, 13, _fdinfo(28, pdev, gfx=9_999_999_999, vram=4), target="socket:[123]")++ times = iter([0, 1_000_000_000, 2_000_000_000]) # the constructor takes the baseline at 0+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))++ self.assertTrue(counters.ok)+ procs, adapters = counters.sample()+ self.assertEqual(list(procs), [card.key])+ self.assertIn(100, procs[card.key])+ self.assertNotIn(200, procs[card.key]) # same client, counted once, on the lowest pid+ self.assertNotIn(300, procs[card.key]) # unknown pdev+ self.assertNotIn(400, procs[card.key]) # non-DRM fd+ self.assertEqual(procs[card.key][100].dedicated, 1024) # private only: 3 KiB resident - 2 KiB shared+ self.assertEqual(procs[card.key][100].shared_vram, 2048)+ self.assertFalse(procs[card.key][100].paged_available) # no commit counter: unavailable, not 0+ self.assertEqual(procs[card.key][100].shared, 2048)+ self.assertEqual(procs[card.key][100].committed, 0)+ self.assertEqual(procs[card.key][100].engines.get("3d", 0.0), 0.0) # idle since the baseline+ self.assertEqual(adapters[card.key].dedicated, 1 * GiB)+ self.assertEqual(adapters[card.key].shared, 64 * 1024)++ (dev / "mem_info_vram_used").write_text(str(2 * GiB))+ _write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_500_000_000, vram=1, gtt=2))++ procs, adapters = counters.sample()+ self.assertEqual(procs[card.key][100].engines["3d"], 50.0)+ self.assertIn("engine", procs[card.key][100].available)+ self.assertEqual(adapters[card.key].dedicated, 2 * GiB)+ counters.close()++ def test_rocm_process_engine_load_is_unknown(self):+ # A HIP process (llama.cpp-ROCm) holds /dev/kfd next to its render node: its VRAM is in fdinfo, but+ # its compute runs on KFD user queues with no per-process busy time, so engines are unavailable.+ with tempfile.TemporaryDirectory() as tmp:+ tmp = Path(tmp)+ pdev = "0000:03:00.0"+ dev = tmp / "drm" / "card1" / "device"+ dev.mkdir(parents=True)+ card = _card(dev, pdev)+ proc = tmp / "proc"+ _write_fd(proc, 100, 3, "pos:\t0\nflags:\t02\n", target="/dev/kfd")+ _write_fd(proc, 100, 7, _fdinfo(500, pdev, vram=3891436))+ _write_fd(proc, 200, 7, _fdinfo(501, pdev, gfx=5, vram=4)) # a graphics client, for contrast+ times = iter([0, 1_000_000_000])+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))+ procs, _ = counters.sample()+ self.assertEqual(procs[card.key][100].dedicated, 3891436 * 1024)+ self.assertNotIn("engine", procs[card.key][100].available)+ self.assertIn("engine", procs[card.key][200].available)+++class DrmEdgeCaseTests(unittest.TestCase):+ def _counters(self, tmp, fds, unified_video=False, sysfs=True):+ pdev = "0000:03:00.0"+ dev = tmp / "drm" / "card1" / "device"+ dev.mkdir(parents=True)+ if sysfs:+ (dev / "mem_info_vram_used").write_text(str(GiB))+ (dev / "mem_info_gtt_used").write_text("0")+ card = _card(dev, pdev)+ card.unified_video = unified_video+ proc = tmp / "proc"+ for pid, fd, text in fds:+ _write_fd(proc, pid, fd, text.replace("PDEV", pdev))+ times = iter([0, 1_000_000_000, 2_000_000_000])+ return card, proc, drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))++ def test_old_kernel_without_shared_keys_is_not_summed(self):+ old = "drm-driver:\tamdgpu\ndrm-client-id:\t5\ndrm-pdev:\tPDEV\ndrm-memory-vram:\t4096 KiB\n"+ with tempfile.TemporaryDirectory() as tmp:+ card, _, counters = self._counters(Path(tmp), [(100, 5, old)])+ g = counters.sample()[0][card.key][100]+ self.assertFalse(g.attributed) # resident includes imports: would double count+ self.assertEqual(g.dedicated, 0)+ self.assertNotIn("dedicated", g.available)++ def test_shared_gtt_left_out_of_spill(self):+ text = _fdinfo(7, "PDEV", gtt=10) + "drm-shared-gtt:\t4 KiB\n"+ with tempfile.TemporaryDirectory() as tmp:+ card, _, counters = self._counters(Path(tmp), [(100, 5, text)])+ self.assertEqual(counters.sample()[0][card.key][100].shared, 6 * 1024)++ def test_unified_vcn_reports_video(self):+ with tempfile.TemporaryDirectory() as tmp:+ text = "drm-driver:\tamdgpu\ndrm-client-id:\t9\ndrm-pdev:\tPDEV\ndrm-shared-vram:\t0\ndrm-engine-enc:\t{} ns\n"+ card, proc, counters = self._counters(Path(tmp), [(100, 5, text.format(0))], unified_video=True)+ (proc / "100" / "fdinfo" / "5").write_text(text.format(1_000_000_000).replace("PDEV", card.pdev))+ engines = counters.sample()[0][card.key][100].engines+ self.assertEqual(engines["video"], 100.0)+ self.assertNotIn("encode", engines)++ def test_missing_sysfs_memory_is_unknown(self):+ with tempfile.TemporaryDirectory() as tmp:+ card, _, counters = self._counters(Path(tmp), [], sysfs=False)+ a = counters.sample()[1][card.key]+ self.assertIsNone(a.dedicated)+ self.assertIsNone(a.shared)+ self.assertIsNone(drm.read_device(card, 0).mem_used)++ def test_vcn_major_from_ip_discovery(self):+ with tempfile.TemporaryDirectory() as tmp:+ uvd = Path(tmp) / "ip_discovery" / "die" / "0" / "UVD" / "0"+ uvd.mkdir(parents=True)+ (uvd / "major").write_text("5\n")+ self.assertEqual(drm._vcn_major(tmp), 5)+ self.assertEqual(drm._vcn_major(str(Path(tmp) / "none")), 0)+++class EngineAndSelectionTests(unittest.TestCase):+ def test_engine_names(self):+ self.assertEqual(drm._engine_key("gfx"), "3d")+ self.assertEqual(drm._engine_key("compute"), "compute")+ self.assertEqual(drm._engine_key("dma"), "copy")+ self.assertEqual(drm._engine_key("enc_1"), "encode")+ self.assertEqual(drm._engine_key("jpeg"), "decode")+ self.assertEqual(drm._engine_key("vpe"), "other")+ self.assertEqual(drm._engine_key("enc", unified_video=True), "video")+ self.assertEqual(drm._engine_key("gfx", unified_video=True), "3d")++ def test_amd_selectable_next_to_nvidia_on_linux(self):+ from gputaskman import collector+ from gputaskman.dxgi import Adapter++ nvidia = Adapter(1, "NVIDIA GPU", 0x10DE, 8 * GiB, 0, False)+ amd = Adapter(2, "AMD Radeon 610M", 0x1002, 2 * GiB, 0, False)+ with patch.object(collector, "BACKEND", "nvml"):+ self.assertIs(collector.select_adapter([amd, nvidia], 0), nvidia)+ self.assertIs(collector.select_adapter([amd, nvidia], 1), amd)+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_linux.py b/tests/test_linux.pyindex 158e3a1..ad9419f 100644--- a/tests/test_linux.py+++ b/tests/test_linux.py@@ -15,6 +15,16 @@ from gputaskman.procinfo import ProcDetail, _lib_name, _match_runtimes, classifyfrom gputaskman.report import to_dict, to_textGiB = 1 << 30+_no_drm = patch("gputaskman.drm.cards", return_value=[]) # keep the test machine's real AMD GPUs out of NVML tests+++def setUpModule() -> None:+ _no_drm.start()+++def tearDownModule() -> None:+ _no_drm.stop()+NOT_AVAILABLE = (1 << 64) - 1 # NVML_VALUE_NOT_AVAILABLE as an unsigned long long