Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

feat: adding support for AMD, tested with Vulkan and ROCm/HIP processes (ex. llama) on a Radeon 9060XT 16GB, Windows.

Kdbf61a8a6459
diff --git a/README.md b/README.md
index ea05653..b560062 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
**An `htop` for what's inside your GPU, not just how full it is.**
-Windows + NVIDIA. A terminal UI that connects per-process GPU memory and engine activity
+Windows + NVIDIA or AMD. A terminal UI that connects per-process GPU memory and engine activity
with loaded runtimes, model files, and LLM buffer breakdowns.
Task Manager says "15 GB used." This tries to answer: *by what? Model weights? KV cache?
@@ -36,14 +36,14 @@ or PowerShell in the extracted folder:
.\gputaskman.exe --json
```
-No Python, uv, CUDA Toolkit, model download, or installation is needed. An NVIDIA GPU with a
-current Windows WDDM driver is still required. Builds are currently **unsigned**; code signing
+No Python, uv, CUDA Toolkit, model download, or installation is needed. A supported GPU with a
+current Windows WDDM driver is required. Builds are currently **unsigned**; code signing
and an installer are not part of this initial distribution. Each ZIP includes licenses and a
`.zip.sha256` companion file is published for integrity checks.
## Clone and run
-Use **64-bit Windows 10/11**, an **NVIDIA GPU with a current WDDM driver**, and a terminal
+Use **64-bit Windows 10/11**, an **NVIDIA or AMD GPU with a current WDDM driver**, and a terminal
(Windows Terminal is recommended). Live workload checks have included an RTX 4080 SUPER and an
RTX Spark N1X on Windows ARM64. CUDA workloads are supported on ARM64 with a compatible NVIDIA
driver and ARM64 CUDA runtime.
@@ -67,7 +67,7 @@ uv run --locked gputaskman
**That's it.** uv obtains the pinned Python version if needed, creates `.venv`, and installs
the locked dependencies. The first run needs internet access. There is no CUDA Toolkit,
Visual Studio, administrator session, API key, or model download required to monitor the GPU.
-NVML ships with the NVIDIA driver.
+NVML ships with the NVIDIA driver; AMD ADL is supplied by supported AMD Windows drivers.
Already have Python 3.11+ and prefer pip?
@@ -91,7 +91,7 @@ skips the consistency check, so do not use it after changing dependencies.
- **Device view:** VRAM usage, GPU activity, memory activity, PCIe traffic, temperature, power,
video-engine clock, and responsive history charts with an all-charts or single-metric view.
- **Process view:** local GPU memory, GPU-accessible system RAM, estimated non-resident commit,
- engine activity, and detected runtimes such as CUDA, Vulkan, Direct3D, ggml and ONNX Runtime.
+ engine activity, and detected runtimes such as CUDA, HIP, Vulkan, Direct3D, ggml and ONNX Runtime.
- **Model inspector:** which model each process holds (LLM, decision, embedding, speech, or diffusion), how
it was identified, architecture, quantization, layer count, weights, estimated KV cost, and
buffer sizes from compatible load logs.
@@ -140,28 +140,45 @@ uv run --locked gputaskman --help
| `q` | Quit |
A wide terminal (roughly 140 columns or more) makes the full table easier to read.
-All eight history charts are shown by default: GPU, SM clock, VRAM, memory activity,
+All eight history charts are shown by default: GPU, SM clock (GFX clock on AMD), VRAM, memory activity,
temperature, power, encoder, and decoder. They stack in narrow terminals and share rows as
the window widens. Click the **Charts** dropdown to focus on one metric or return to **All charts**.
A blank sample means no reading; zero is drawn at the baseline. Percentages use a 0–100%
-scale, temperature uses 0–120°C, and the power and SM clock scales are displayed below the charts.
+scale, temperature uses 0–120°C, and the power and clock scales are displayed below the charts.
The dashboard and inspector scroll independently in short terminals so the process list stays usable.
Model/runtime discovery is throttled, so newly loaded models may take about 15 seconds to appear.
-`--gpu N` selects a detected NVIDIA graphics adapter, ordered by dedicated memory; an out-of-range
+`--list-gpus` lists selectable hardware adapters with their vendor and LUID. `--gpu N` selects
+one of these adapters: NVIDIA adapters are listed first, followed by other vendors, with each
+group ordered by dedicated memory. An out-of-range
index is rejected rather than silently selecting another device. Compute-only devices such as an
-NPU are not graphics adapters. **Single-NVIDIA-GPU systems are the validated configuration**.
-On multi-GPU systems, adapter names may be ambiguous, DXGI and NVML ordering may differ, and buffer
-logs can span devices; cross-device attribution is not yet reliable.
+NPU are not graphics adapters. AMD and NVIDIA hardware coverage is described below; configurations
+with multiple GPUs and mixed vendors need further validation.
+Device telemetry is matched to WDDM adapters by PCI location, not display name or enumeration order.
+If a unique match cannot be established, telemetry is unavailable; buffer logs can still span devices,
+so model-buffer attribution across GPUs is not yet reliable. This conservative matching also applies
+to NVIDIA and may suppress telemetry when Windows does not expose a usable PCI address.
+
+AMD adapters can be selected and use the existing Windows GPU process counters; HIP runtime DLLs
+are recognized as runtime evidence. On supported drivers, the AMD Display Library supplied by the
+Windows driver can provide graphics activity, clocks, edge temperature, board or ASIC power, fan
+and PCIe lane count. An optional AMD SMI Python installation is a fallback and may provide graphics
+activity, edge temperature and socket power. Both require a PCI match to the selected WDDM adapter.
+The SMI library and its native dependencies must be available to the Python environment; it is not
+bundled in the portable build. The RX 9060 XT HIP workload and ADL telemetry have been observed on
+this machine; AMD SMI telemetry and Vulkan workloads remain unvalidated.
+A loaded DLL does not prove GPU execution.
Unavailable utilization, temperature, power, clock and PCIe readings are displayed as `N/A` in
text and the TUI and as `null` in JSON. For example, a missing NVML power limit means that the
driver did not expose that reading; it does not by itself indicate a hardware fault.
+Missing per-process memory or engine samples likewise show `N/A`/`null`; the paged estimate is
+unavailable when any of its input counters is missing.
## How can it know what's inside?
It combines evidence rather than reading arbitrary GPU memory. Windows' graphics kernel supplies
-per-process memory and engine counters, while NVML supplies device metrics. Process DLLs suggest
+per-process memory and engine counters, while NVML or AMD ADL/SMI supplies device metrics. Process DLLs suggest
the runtime, and mapped files or command-line arguments identify candidate models. GGUF metadata
and tensor offsets let it estimate weights and KV-cache costs; compatible runtime logs and local
server APIs add allocated buffer sizes and live state. That can explain a llama.cpp process well,
@@ -169,7 +186,7 @@ but a runtime DLL or filename alone does not prove which allocation contains whi
| Question | Evidence | Confidence / limitation |
|---|---|---|
-| How full and busy is the card? | WDDM adapter counters and NVML | Measured; driver support varies by metric |
+| How full and busy is the card? | WDDM adapter counters and NVML or AMD ADL/SMI | Measured; driver support varies by metric |
| Which process uses GPU memory? | WDDM process counters | Measured accounting, not a resource ownership graph |
| Which engines are active? | WDDM engine counters | CUDA can appear on the **3D** engine |
| Which runtime is loaded? | Mapped DLL names | Detection, not proof that every library is doing GPU work |
@@ -351,7 +368,7 @@ file paths, but model and process names still require review.
|---|---|
| `uv` or `git` is not recognized | Install the prerequisites, then open a new terminal |
| Windows-only message | Run on native Windows, not Linux, macOS, or WSL |
-| Empty or incomplete GPU readings | Check the NVIDIA driver and WDDM 2.0+ support in `dxdiag`; use a normal local desktop session |
+| Empty or incomplete GPU readings | Check the GPU driver and WDDM 2.0+ support in `dxdiag`; use a normal local desktop session |
| Missing DLLs or model details | Protected processes may deny inspection; basic counters can still work without elevation |
| No exact LLM breakdown | Enable compatible buffer logging, or use the GGUF estimate; API/log access is optional |
| The screen looks cramped | Widen the terminal or use `--once` / `--json` |
@@ -380,7 +397,7 @@ uv run --locked --group release python tools\build_portable.py --arch x64
The builder verifies the interpreter and executable architectures, includes Textual resources
and dependency licenses, and writes the complete portable ZIP and SHA-256 file to `dist`.
-It does not bundle Windows/NVIDIA DLLs, inference servers, or models. Keep uv for development;
+It does not bundle Windows or GPU driver DLLs, inference servers, or models. Keep uv for development;
the executable ZIP is for running the monitor without a Python environment.
The **Portable Windows release** GitHub Actions workflow builds on `windows-2022` (x64) and
@@ -404,6 +421,7 @@ downloads are bundled in this repository.
|---|---|
| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration |
| `nvml.py` | NVIDIA device readings |
+| `adl.py`, `amdsmi.py` | Optional AMD device readings |
| `procinfo.py` | Runtime, mapped-file, and process discovery |
| `stores.py` | Friendly names from Ollama, LM Studio, and Hugging Face model stores |
| `gguf.py` | Model metadata, kind, and size estimates |
diff --git a/src/gputaskman/adl.py b/src/gputaskman/adl.py
new file mode 100644
index 0000000..dc48027
--- /dev/null
+++ b/src/gputaskman/adl.py
@@ -0,0 +1,101 @@
+"""Read-only AMD Display Library telemetry supplied by the Windows driver.
+
+ADL2_New_QueryPMLogData_Get is deprecated by AMD; failures leave data unavailable.
+Definitions: https://github.com/GPUOpen-LibrariesAndSDKs/display-library/tree/master/include
+"""
+from __future__ import annotations
+
+import ctypes
+import os
+
+from .dxgi import Adapter, VENDOR_AMD
+from .telemetry import DeviceTelemetry
+
+
+class _AdapterInfo(ctypes.Structure):
+ _fields_ = [("size", ctypes.c_int), ("index", ctypes.c_int), ("udid", ctypes.c_char * 256),
+ ("bus", ctypes.c_int), ("device", ctypes.c_int), ("function", ctypes.c_int),
+ ("vendor", ctypes.c_int), ("name", ctypes.c_char * 256),
+ ("display", ctypes.c_char * 256), ("present", ctypes.c_int), ("exists", ctypes.c_int),
+ ("driver", ctypes.c_char * 256), ("driver_ext", ctypes.c_char * 256),
+ ("pnp", ctypes.c_char * 256), ("os_index", ctypes.c_int)]
+
+
+class _Sensor(ctypes.Structure):
+ _fields_ = [("supported", ctypes.c_int), ("value", ctypes.c_int)]
+
+
+class _Metrics(ctypes.Structure):
+ _fields_ = [("size", ctypes.c_int), ("sensors", _Sensor * 256)]
+
+
+class Adl:
+ def __init__(self, adapter: Adapter | None, adapters: list[Adapter]):
+ self.dll = None
+ self.context = ctypes.c_void_p()
+ self.index = None
+ self.driver = ""
+ if not adapter or adapter.vendor_id != VENDOR_AMD or adapter.pci_address is None:
+ return
+ if sum(a.pci_address == adapter.pci_address for a in adapters if not a.software) != 1:
+ return
+ try:
+ # System32 is the AMD driver's installation location, not the working directory.
+ self.dll = ctypes.CDLL(os.path.join(os.environ.get("SystemRoot", r"C:\Windows"), "System32", "atiadlxx.dll"))
+ crt = ctypes.CDLL("msvcrt")
+ crt.malloc.argtypes = [ctypes.c_size_t]
+ crt.malloc.restype = ctypes.c_void_p
+ self._malloc = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_int)(crt.malloc)
+ self.dll.ADL2_Main_Control_Create.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(ctypes.c_void_p)]
+ self.dll.ADL2_Main_Control_Destroy.argtypes = [ctypes.c_void_p]
+ self.dll.ADL2_Adapter_NumberOfAdapters_Get.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_int)]
+ self.dll.ADL2_Adapter_AdapterInfo_Get.argtypes = [ctypes.c_void_p, ctypes.POINTER(_AdapterInfo), ctypes.c_int]
+ self.dll.ADL2_New_QueryPMLogData_Get.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(_Metrics)]
+ if self.dll.ADL2_Main_Control_Create(self._malloc, 1, ctypes.byref(self.context)) != 0:
+ return
+ count = ctypes.c_int()
+ if self.dll.ADL2_Adapter_NumberOfAdapters_Get(self.context, ctypes.byref(count)) != 0 or not 0 < count.value <= 256:
+ return
+ entries = (_AdapterInfo * count.value)()
+ for entry in entries:
+ entry.size = ctypes.sizeof(_AdapterInfo)
+ if self.dll.ADL2_Adapter_AdapterInfo_Get(self.context, entries, ctypes.sizeof(entries)) != 0:
+ return
+ matches = {entry.index for entry in entries if (entry.bus, entry.device, entry.function) == adapter.pci_address}
+ # ADL may return several logical display handles for the same physical GPU.
+ if matches:
+ self.index = min(matches)
+ except (OSError, AttributeError, ValueError):
+ self.close()
+ finally:
+ if self.index is None:
+ self.close()
+
+ def sample(self) -> DeviceTelemetry | None:
+ if self.index is None:
+ return None
+ metrics = _Metrics()
+ metrics.size = ctypes.sizeof(metrics)
+ try:
+ status = self.dll.ADL2_New_QueryPMLogData_Get(self.context, self.index, ctypes.byref(metrics))
+ except (OSError, AttributeError):
+ return None
+ if status != 0 or not any(item.supported for item in metrics.sensors):
+ return None
+ def sensor(index):
+ item = metrics.sensors[index]
+ return item.value if item.supported and item.value >= 0 else None
+ board_power = sensor(73)
+ asic_power = sensor(23)
+ return DeviceTelemetry(index=self.index, source="AMD ADL", gfx_clock=sensor(1), mem_clock=sensor(2),
+ temp=sensor(8), fan=sensor(15), util_gpu=sensor(19),
+ power_w=board_power if board_power is not None else asic_power,
+ power_kind="board" if board_power is not None else "ASIC" if asic_power is not None else "",
+ pcie_width=sensor(41))
+
+ def close(self):
+ self.index = None
+ if self.dll is not None and self.context.value:
+ context = self.context
+ self.context = ctypes.c_void_p()
+ self.dll.ADL2_Main_Control_Destroy(context)
diff --git a/src/gputaskman/amdsmi.py b/src/gputaskman/amdsmi.py
new file mode 100644
index 0000000..a94f1ef
--- /dev/null
+++ b/src/gputaskman/amdsmi.py
@@ -0,0 +1,80 @@
+"""Optional read-only AMD SMI telemetry. WDDM collection works without this module."""
+from __future__ import annotations
+
+import importlib
+import math
+import re
+
+from .dxgi import Adapter, VENDOR_AMD
+from .telemetry import DeviceTelemetry
+
+
+def _number(value):
+ return value if isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) else None
+
+
+def _bdf(value):
+ match = re.fullmatch(r"([0-9a-fA-F]+):([0-9a-fA-F]+):([0-9a-fA-F]+)\.([0-9a-fA-F]+)", str(value))
+ if not match:
+ return None
+ domain, bus, device, function = (int(part, 16) for part in match.groups())
+ # WDDM's ADAPTERADDRESS has no domain. Refuse nonzero domains rather than guess.
+ return (bus, device, function) if domain == 0 else None
+
+
+class AmdSmi:
+ def __init__(self, adapter: Adapter | None, adapters: list[Adapter], api=None):
+ self.api = None
+ self.handle = None
+ self.driver = ""
+ self.initialized = False
+ if not adapter or adapter.vendor_id != VENDOR_AMD or adapter.pci_address is None:
+ return
+ if sum(a.pci_address == adapter.pci_address for a in adapters if not a.software) != 1:
+ return
+ try:
+ self.api = api or importlib.import_module("amdsmi")
+ self.api.amdsmi_init()
+ self.initialized = True
+ matches = [h for h in self.api.amdsmi_get_processor_handles()
+ if _bdf(self._call("amdsmi_get_gpu_device_bdf", h)) == adapter.pci_address]
+ if len(matches) == 1:
+ self.handle = matches[0]
+ info = self._call("amdsmi_get_gpu_driver_info", self.handle) or {}
+ if isinstance(info, dict):
+ self.driver = str(info.get("driver_version") or "")
+ except Exception:
+ self.close()
+
+ def _call(self, name, *args):
+ try:
+ return getattr(self.api, name)(*args)
+ except Exception:
+ return None
+
+ def sample(self) -> DeviceTelemetry | None:
+ if self.handle is None:
+ return None
+ d = DeviceTelemetry(index=0, source="AMD SMI")
+ activity = self._call("amdsmi_get_gpu_activity", self.handle)
+ if isinstance(activity, dict):
+ d.util_gpu = _number(activity.get("gfx_activity"))
+ power = self._call("amdsmi_get_power_info", self.handle)
+ if isinstance(power, dict):
+ d.power_w = _number(power.get("socket_power"))
+ d.power_kind = "socket" if d.power_w is not None else ""
+ d.power_limit_w = _number(power.get("power_limit"))
+ try:
+ temp = self._call("amdsmi_get_temp_metric", self.handle,
+ self.api.AmdSmiTemperatureType.EDGE, self.api.AmdSmiTemperatureMetric.CURRENT)
+ value = _number(temp)
+ d.temp = value / 1000 if value is not None else None
+ except AttributeError:
+ pass
+ return d
+
+ def close(self):
+ self.handle = None
+ if self.initialized:
+ self._call("amdsmi_shut_down")
+ self.initialized = False
diff --git a/src/gputaskman/app.py b/src/gputaskman/app.py
index 41dbfe9..f022019 100644
--- a/src/gputaskman/app.py
+++ b/src/gputaskman/app.py
@@ -42,8 +42,8 @@ EXPLAIN = {
"Deeper: edge://gpu, or DevTools > Performance with the GPU track.",
"3D/game": "Textures, meshes, render targets and descriptor heaps. Windows only sees opaque allocations; "
"per-resource names need a graphics debugger (PIX on Windows, NVIDIA Nsight Graphics).",
- "video": "Decoded/encoded video frames and codec surfaces (NVDEC/NVENC).",
- "compute": "CUDA allocations are opaque to the OS. For more detail trace cudaMalloc with Nsight Systems.",
+ "video": "Decoded/encoded video frames and codec surfaces (NVIDIA NVDEC/ENC)",
+ "compute": "Compute allocations are opaque to the OS. Runtime-specific profilers can provide more detail (CUDA: cudaMalloc with Nsight Systems).",
"diffusion": "Model components are identified from launch arguments; GPU allocations cannot be split by component.",
"AI": "ML runtime allocations (tensors, workspaces). PyTorch: torch.cuda.memory_summary(); "
"ONNX Runtime/DirectML pools are opaque from outside.",
@@ -55,9 +55,9 @@ HELP_MD = """\
| Layer | Source | Works for |
|---|---|---|
-| **Device** totals, load, bandwidth, PCIe, power | NVML (`nvml.dll`) | NVIDIA |
-| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl) | every API incl. **CUDA** |
-| **Which runtime** (CUDA, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything |
+| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI | availability varies by GPU and driver |
+| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl) | workloads visible to WDDM |
+| **Which runtime** (CUDA, HIP, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything |
| **Which model file** | memory-mapped files + command line | llama.cpp, stable-diffusion.cpp, LM Studio, Ollama |
| **Model name & kind** (LLM, decision, embedding, speech) | GGUF header, Ollama/LM Studio/Hugging Face stores, local server APIs | GGUF family |
| **Weights / KV cache / compute** split | GGUF header math, llama.cpp/Ollama logs, local server APIs | GGUF/llama.cpp family |
@@ -91,7 +91,8 @@ non-resident committed memory, not a measured eviction count or proof of pagefil
not necessarily current residency. A game's textures cannot be enumerated here.
**GPU%** = busiest engine (like Task Manager). On GeForce/WDDM, CUDA kernels are usually reported
-on the **3D** engine.
+on the **3D** engine. AMD device GPU charts show graphics-engine activity, which is not
+necessarily equivalent to NVML GPU utilization. Vulkan does not expose another process's HIP internals.
Keys: `↑/↓` select · `s` sort · `c` chart · `a` show all · `p` pause · `?` this help · `q` quit
@@ -241,9 +242,17 @@ class GpuTaskManager(App):
def apply(self, snap: Snapshot) -> None:
self.snap = snap
nv = snap.nv
+ amd = bool(snap.adapter and snap.adapter.vendor_id == 0x1002)
+ if getattr(self, "_amd_chart_labels", None) != amd:
+ self._amd_chart_labels = amd
+ selector = self.query_one("#chart-mode", Select)
+ selected = selector.value
+ selector.set_options([("All charts", -1), *(("GFX clock" if amd and name == "SM clock" else name, index)
+ for index, name in enumerate(self.CHARTS))])
+ selector.value = selected
if nv:
self.hist_util.append(nv.util_gpu)
- self.hist_clock.append(nv.sm_clock)
+ self.hist_clock.append(nv.gfx_clock if amd else nv.sm_clock)
self.hist_bw.append(nv.util_mem)
self.hist_temp.append(nv.temp)
self.hist_power.append(nv.power_w)
@@ -253,7 +262,8 @@ class GpuTaskManager(App):
for history in (self.hist_util, self.hist_clock, self.hist_bw, self.hist_temp,
self.hist_power, self.hist_enc, self.hist_dec):
history.append(None)
- self.hist_vram.append(100 * snap.vram_used / snap.vram_total if snap.vram_total else None)
+ self.hist_vram.append(100 * snap.vram_used / snap.vram_total
+ if snap.vram_total and snap.vram_used is not None else None)
if not self._auto_selected:
llm = next((r for r in sorted(snap.procs, key=lambda r: -r.gpu.dedicated) if r.llm), None)
if llm:
@@ -267,7 +277,8 @@ class GpuTaskManager(App):
def render_device(self, s: Snapshot) -> Group:
nv = s.nv
name = s.adapter.name if s.adapter else (nv.name if nv else "GPU")
- head = Text.assemble((name, "bold"), f" driver {s.driver} CUDA {s.cuda} ",
+ info = (f" driver {s.driver}" if s.driver else "") + (f" CUDA {s.cuda}" if s.cuda else "")
+ head = Text.assemble((name, "bold"), info + " ",
(nv.driver_model if nv else "", "bold yellow"), f" {nv.pci_bus if nv else ''}")
if self.paused:
head.append(" [PAUSED]", style="bold red")
@@ -277,15 +288,16 @@ class GpuTaskManager(App):
for r in s.procs:
by_kind[r.kind] = by_kind.get(r.kind, 0) + r.gpu.dedicated
attributed = sum(by_kind.values())
- scale = min(1.0, s.vram_used / attributed) if attributed else 1.0
+ scale = min(1.0, s.vram_used / attributed) if attributed and s.vram_used is not None else 1.0
order = sorted(by_kind.items(), key=lambda kv: -kv[1])
- parts = [(v * scale, KIND_STYLE.get(k, "white")) for k, v in order]
- other = max(s.vram_used - attributed * scale, 0)
+ parts = [(v * scale, KIND_STYLE.get(k, "white")) for k, v in order] if s.vram_used is not None else []
+ other = max(s.vram_used - attributed * scale, 0) if s.vram_used is not None and s.attributed is not None else 0
parts.append((other, "grey35"))
width = max(min(self.size.width - 40, 90), 20)
vram = Text("VRAM ")
vram.append_text(stacked_bar(parts, s.vram_total, width))
- vram.append(f" {gb(s.vram_used)} / {gb(s.vram_total)}", style="bold")
+ used_label = gb(s.vram_used) if s.vram_used is not None else "N/A"
+ vram.append(f" {used_label} / {gb(s.vram_total)}", style="bold")
legend = Text(" ")
for k, v in order[:7]:
legend.append("■ ", style=KIND_STYLE.get(k, "white"))
@@ -293,8 +305,9 @@ class GpuTaskManager(App):
if other > 32 << 20:
legend.append("■ ", style="grey35")
legend.append(f"unattributed {gb(other)} ")
- legend.append(f"free {gb(max(s.vram_total - s.vram_used, 0))}", style="grey62")
- if s.shared_used > 64 << 20:
+ if s.vram_used is not None:
+ legend.append(f"free {gb(max(s.vram_total - s.vram_used, 0))}", style="grey62")
+ if s.shared_used is not None and s.shared_used > 64 << 20:
legend.append(f" shared(RAM) {gb(s.shared_used)}", style="yellow")
if scale < 1.0:
legend.append(" [bar normalized; labels are raw]", style="grey62")
@@ -305,18 +318,22 @@ class GpuTaskManager(App):
tx = nv.pcie_tx_kbs / 1024 if nv.pcie_tx_kbs is not None else None
pstate = f"P{nv.pstate}" if nv.pstate is not None else "P N/A"
m = Text()
- m.append(f"GPU {format_metric(nv.util_gpu, unit='%')}", style=f"bold {self.CHART_COLORS['GPU']}")
+ label = "AMD gfx" if nv.source.startswith("AMD ") else "GPU"
+ m.append(f"{label} {format_metric(nv.util_gpu, unit='%')}", style=f"bold {self.CHART_COLORS['GPU']}")
m.append(f" Mem activity {format_metric(nv.util_mem, unit='%')}", style=self.CHART_COLORS["Mem activity"])
m.append(f" PCIe↓{format_metric(rx, '.0f'):>5} ↑{format_metric(tx, '.0f'):>4} MB/s")
if nv.enc or nv.dec:
m.append(f" enc {format_metric(nv.enc, unit='%')} dec {format_metric(nv.dec, unit='%')}")
lines.append(m)
+ clock_label = (f"gfx {format_metric(nv.gfx_clock)} MHz" if nv.source.startswith("AMD ") else
+ f"SM {format_metric(nv.sm_clock)}/{format_metric(nv.sm_clock_max)} MHz")
+ power_kind = f" {nv.power_kind}" if nv.power_kind else ""
st = Text(f"{format_metric(nv.temp, unit='°C')} {format_metric(nv.power_w, '.0f')}/"
- f"{format_metric(nv.power_limit_w, '.0f')} W {pstate} "
- f"SM {format_metric(nv.sm_clock)}/{format_metric(nv.sm_clock_max)} MHz "
+ f"{format_metric(nv.power_limit_w, '.0f')} W{power_kind} {pstate} "
+ f"{clock_label} "
f"mem {format_metric(nv.mem_clock)} MHz video {format_metric(nv.video_clock)} MHz "
f"PCIe gen{format_metric(nv.pcie_gen)} x{format_metric(nv.pcie_width)} "
- f"CC {nv.compute_cap or 'N/A'}", style="grey70")
+ + (f"CC {nv.compute_cap or 'N/A'}" if nv.source == "NVML" else nv.source), style="grey70")
if nv.throttle:
st.append(" throttle: " + ",".join(nv.throttle), style="yellow")
lines.append(st)
@@ -361,7 +378,10 @@ class GpuTaskManager(App):
for index in indices:
values, scale, unit, style = self.chart_series(index, self.snap.nv)
current = values[-1] if values else None
- cell = Text(f"{self.CHARTS[index]:<12} ", style=style)
+ name = self.CHARTS[index]
+ if getattr(self, "_amd_chart_labels", False) and name == "SM clock":
+ name = "GFX clock"
+ cell = Text(f"{name:<12} ", style=style)
cell.append(f"{format_metric(current, '.0f', unit):>8} ", style="bold")
cell.append(spark(values, max(1, cell_width - 22), scale))
cells.append(cell)
@@ -372,7 +392,7 @@ class GpuTaskManager(App):
_, clock_scale, _, _ = self.chart_series(self.CHARTS.index("SM clock"), self.snap.nv)
self.query_one("#chart-note", Static).update(
f"Recent samples · blank = no reading · power 0–{power_scale:.0f} W · "
- f"SM clock 0–{clock_scale:.0f} MHz"
+ f"{'GFX' if getattr(self, '_amd_chart_labels', False) else 'SM'} clock 0–{clock_scale:.0f} MHz"
)
def on_resize(self) -> None:
@@ -398,17 +418,20 @@ class GpuTaskManager(App):
for r in rows:
e = r.gpu.engines
- spill = Text(gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",
+ spill = Text("N/A" if "shared" not in r.gpu.available else gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",
style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")
- paged = Text(gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")
+ paged = Text("N/A" if not r.gpu.paged_available else gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")
+ vram = gb(r.gpu.dedicated) if "dedicated" in r.gpu.available else "N/A"
+ activity = pct(r.gpu.gpu_pct) if "engine" in r.gpu.available else Text("N/A", justify="right")
+ engine = lambda value: pct(value) if "engine" in r.gpu.available else Text("N/A", justify="right")
t.add_row(
Text(str(r.pid), justify="right"),
Text(r.name),
Text(r.kind, style=KIND_STYLE.get(r.kind, "")),
Text(" ".join(r.detail.runtimes[:3]), style="grey70"),
- Text(gb(r.gpu.dedicated), justify="right", style="bold"), spill, paged, pct(r.gpu.gpu_pct),
- pct(e.get("3d", 0) + e.get("compute", 0)), pct(e.get("copy", 0)),
- pct(e.get("decode", 0) + e.get("encode", 0)),
+ Text(vram, justify="right", style="bold"), spill, paged, activity,
+ engine(e.get("3d", 0) + e.get("compute", 0)), engine(e.get("copy", 0)),
+ engine(e.get("decode", 0) + e.get("encode", 0)),
Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),
key=str(r.pid),
)
@@ -429,15 +452,15 @@ class GpuTaskManager(App):
panel.border_title = f"{r.name} ({r.pid}) - {r.kind}"
out: list = []
e = r.gpu.engines
- hdr = Text.assemble(("VRAM ", "bold"), (gb(r.gpu.dedicated), "bold"), " spill(RAM) ",
- (gb(r.gpu.shared), "bold red" if r.gpu.shared > 256 << 20 else ""),
- " paged(est) ", gb(r.gpu.paged_out),
- " engines ", " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle")
+ hdr = Text.assemble(("VRAM ", "bold"), (gb(r.gpu.dedicated) if "dedicated" in r.gpu.available else "N/A", "bold"), " spill(RAM) ",
+ (gb(r.gpu.shared) if "shared" in r.gpu.available else "N/A", "bold red" if r.gpu.shared > 256 << 20 else ""),
+ " paged(est) ", gb(r.gpu.paged_out) if r.gpu.paged_available else "N/A",
+ " engines ", (" ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle") if "engine" in r.gpu.available else "N/A")
if r.detail.runtimes:
hdr.append(" runtime ", style="grey62")
hdr.append(" ".join(r.detail.runtimes), style="cyan")
out.append(hdr)
- if r.gpu.paged_out >= 64 << 20:
+ if r.gpu.paged_available and r.gpu.paged_out >= 64 << 20:
out.append(Text(f"{gb(r.gpu.paged_out)} of {gb(r.gpu.committed)} committed is not accounted for by local/non-local "
"usage. Estimated non-residency, not proof of eviction or disk paging.",
style="yellow"))
diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.py
index 59df309..bb2fa81 100644
--- a/src/gputaskman/cli.py
+++ b/src/gputaskman/cli.py
@@ -13,6 +13,7 @@ def main() -> None:
ap.add_argument("--once", action="store_true", help="print one snapshot and exit (no TUI)")
ap.add_argument("--json", action="store_true", help="print one JSON snapshot and exit (implies --once)")
ap.add_argument("--gpu", type=int, default=0, help="GPU index (default 0)")
+ ap.add_argument("--list-gpus", action="store_true", help="list selectable GPU adapters and exit")
ap.add_argument("-i", "--interval", type=float, default=1.0, help="refresh seconds (default 1)")
args = ap.parse_args()
if args.gpu < 0:
@@ -23,6 +24,13 @@ def main() -> None:
if sys.platform != "win32":
sys.exit("gputaskman currently supports Windows (WDDM) only.")
+ if args.list_gpus:
+ from .collector import enumerate_adapters, ordered_adapters
+
+ for i, adapter in enumerate(ordered_adapters(enumerate_adapters())):
+ print(f"{i}: {adapter.name} ({adapter.vendor}, LUID 0x{adapter.luid & 0xffffffffffffffff:016x})")
+ return
+
if args.gpu:
from .collector import enumerate_adapters, select_adapter
diff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.py
index 216b4ae..1e3829b 100644
--- a/src/gputaskman/collector.py
+++ b/src/gputaskman/collector.py
@@ -8,12 +8,15 @@ from dataclasses import dataclass, field
import psutil
+from .amdsmi import AmdSmi, _bdf
+from .adl import Adl
from .dxgi import VENDOR_NVIDIA, Adapter, enumerate_adapters
from .handy import HandyEvidence, inspect_handy
from .llm import LlmView, ServerProber, build_view, gb, ollama_log_path
from .nvml import Nvml, NvDevice
from .pdh import GpuCounters, ProcGpu
from .procinfo import AI_KINDS, ProcDetail, ProcInspector, classify
+from .telemetry import DeviceTelemetry
MiB = 1 << 20
@@ -69,23 +72,23 @@ class Snapshot:
taken: float
adapter: Adapter | None
adapters: list[Adapter]
- nv: NvDevice | None
+ nv: DeviceTelemetry | None # legacy name; may contain telemetry from either provider
driver: str
cuda: str
vram_total: int
- vram_used: int # WDDM adapter-wide dedicated usage (ground truth)
- shared_used: int
+ vram_used: int | None # WDDM adapter-wide dedicated usage
+ shared_used: int | None
procs: list[ProcRow] = field(default_factory=list)
@property
- def attributed(self) -> int:
+ def attributed(self) -> int | None:
+ if any("dedicated" not in p.gpu.available for p in self.procs):
+ return None
return int(sum(p.gpu.dedicated for p in self.procs))
def select_adapter(adapters: list[Adapter], gpu_index: int) -> Adapter | None:
- hardware = [a for a in adapters if not a.software]
- choices = [a for a in hardware if a.vendor_id == VENDOR_NVIDIA] or hardware
- choices.sort(key=lambda a: -a.dedicated_vram)
+ choices = ordered_adapters(adapters)
if gpu_index < 0:
raise ValueError("--gpu must be a non-negative index")
if gpu_index >= len(choices):
@@ -95,15 +98,26 @@ def select_adapter(adapters: list[Adapter], gpu_index: int) -> Adapter | None:
return choices[gpu_index]
-def select_nv_device(adapter: Adapter | None, devices: list[NvDevice], gpu_index: int) -> NvDevice | None:
+def ordered_adapters(adapters: list[Adapter]) -> list[Adapter]:
+ """List all hardware adapters; retain NVIDIA-first ordering for existing users."""
+ return sorted((a for a in adapters if not a.software),
+ key=lambda a: (a.vendor_id != VENDOR_NVIDIA, -a.dedicated_vram, a.luid))
+
+
+def select_nv_device(adapter: Adapter | None, devices: list[NvDevice], gpu_index: int,
+ adapters: list[Adapter] | None = None) -> NvDevice | None:
if adapter:
if adapter.vendor_id != VENDOR_NVIDIA:
return None
- matches = [d for d in devices if d.name.casefold() == adapter.name.casefold()]
+ if adapter.pci_address is None:
+ return None
+ if adapters is not None and sum(a.pci_address == adapter.pci_address for a in adapters) != 1:
+ return None
+ matches = [d for d in devices if _bdf(d.pci_bus) == adapter.pci_address and d.driver_model != "MCDM"]
if len(matches) == 1:
return matches[0]
- fallback = devices[gpu_index] if gpu_index < len(devices) else None
- return None if adapter and fallback and fallback.driver_model == "MCDM" else fallback
+ return None
+ return devices[gpu_index] if gpu_index < len(devices) else None
class Collector:
@@ -115,6 +129,8 @@ class Collector:
self.adapter = select_adapter(self.adapters, gpu_index)
self.counters = GpuCounters()
self.nvml = Nvml()
+ self.amdsmi = AmdSmi(self.adapter, self.adapters)
+ self.adl = Adl(self.adapter, self.adapters)
self.inspector = ProcInspector()
self.prober = ServerProber()
self.scan_budget = scan_budget # None = scan every candidate now (one-shot output)
@@ -133,7 +149,7 @@ class Collector:
d.parents.append(p.pid)
except (psutil.Error, OSError):
pass
- return [port for ppid in d.parents for port in self.inspector.ports(ppid)]
+ return [port for ppid in d.parents for port in self.inspector.exclusive_ports(ppid)]
def sample(self) -> Snapshot:
per_luid, adapters = self.counters.sample()
@@ -141,14 +157,17 @@ class Collector:
procs = per_luid.get(luid, {})
a = adapters.get(luid)
nvs = self.nvml.sample() if self.nvml.ok else []
- nv = select_nv_device(self.adapter, nvs, self.gpu_index)
+ nv = select_nv_device(self.adapter, nvs, self.gpu_index, self.adapters)
+ if self.adapter and self.adapter.vendor_id == 0x1002:
+ nv = self.adl.sample() or self.amdsmi.sample()
total = (nv.mem_total if nv else 0) or (self.adapter.dedicated_vram if self.adapter else 0)
snap = Snapshot(
taken=time.time(), adapter=self.adapter, adapters=self.adapters, nv=nv,
- driver=self.nvml.driver, cuda=self.nvml.cuda_version, vram_total=total,
- vram_used=int(a.dedicated) if a else (nv.mem_used if nv else 0),
- shared_used=int(a.shared) if a else 0,
+ driver=(self.amdsmi.driver if nv and nv.source == "AMD SMI" else self.nvml.driver if nv and nv.source == "NVML" else ""),
+ cuda=self.nvml.cuda_version if nv and nv.source == "NVML" else "", vram_total=total,
+ vram_used=int(a.dedicated) if a and a.dedicated is not None else None,
+ shared_used=int(a.shared) if a and a.shared is not None else None,
)
live = [(pid, g) for pid, g in procs.items()
@@ -173,9 +192,10 @@ class Collector:
for row in snap.procs:
if os.path.basename(row.detail.exe).lower() == "handy.exe":
row.handy = inspect_handy(row.detail.create_time)
- if row.detail.is_diffusion or row.kind not in AI_KINDS or not (row.detail.models or row.kind == "LLM"):
+ if (row.detail.is_diffusion or row.kind not in AI_KINDS or not (row.detail.models or row.kind == "LLM")
+ or not {"dedicated", "shared"} <= row.gpu.available):
continue
- own = self.prober.get(row.detail.ports)
+ own = self.prober.get(self.inspector.exclusive_ports(row.pid))
host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end process
log = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else None
row.llm = build_view(row.detail, row.gpu.dedicated, row.gpu.shared, own or host, log,
@@ -187,3 +207,5 @@ class Collector:
def close(self) -> None:
self.counters.close()
self.nvml.close()
+ self.amdsmi.close()
+ self.adl.close()
diff --git a/src/gputaskman/dxgi.py b/src/gputaskman/dxgi.py
index 9af8d2b..93bb675 100644
--- a/src/gputaskman/dxgi.py
+++ b/src/gputaskman/dxgi.py
@@ -32,6 +32,7 @@ class _GUID(ctypes.Structure):
_IID_IDXGIFactory1 = _GUID(0x770AAE78, 0xF26F, 0x4DBA, (ctypes.c_ubyte * 8)(0xA8, 0x29, 0x25, 0x3C, 0x83, 0xD1, 0xB3, 0x87))
VENDOR_NVIDIA = 0x10DE
+VENDOR_AMD = 0x1002
VENDORS = {0x10DE: "NVIDIA", 0x1002: "AMD", 0x8086: "Intel", 0x1414: "Microsoft", 0x5143: "Qualcomm"}
@@ -43,6 +44,7 @@ class Adapter:
dedicated_vram: int
shared_system: int
software: bool
+ pci_address: tuple[int, int, int] | None = None
@property
def vendor(self) -> str:
@@ -54,6 +56,38 @@ def _vcall(obj: ctypes.c_void_p, index: int, restype, *argtypes):
return ctypes.WINFUNCTYPE(restype, ctypes.c_void_p, *argtypes)(vtbl[index])
+class _OpenAdapter(ctypes.Structure):
+ _fields_ = [("luid", _LUID), ("handle", wintypes.UINT)]
+
+
+class _AdapterAddress(ctypes.Structure):
+ _fields_ = [("bus", wintypes.UINT), ("device", wintypes.UINT), ("function", wintypes.UINT)]
+
+
+class _QueryAdapter(ctypes.Structure):
+ _fields_ = [("handle", wintypes.UINT), ("kind", wintypes.UINT),
+ ("data", ctypes.c_void_p), ("size", wintypes.UINT)]
+
+
+def adapter_pci_address(luid: int) -> tuple[int, int, int] | None:
+ """Obtain the WDDM adapter's bus/device/function; the PCI domain is not exposed."""
+ try:
+ gdi = ctypes.WinDLL("gdi32")
+ opened = _OpenAdapter(_LUID(luid & 0xffffffff, ctypes.c_int32(luid >> 32).value), 0)
+ if gdi.D3DKMTOpenAdapterFromLuid(ctypes.byref(opened)) != 0:
+ return None
+ try:
+ address = _AdapterAddress()
+ query = _QueryAdapter(opened.handle, 6, ctypes.addressof(address), ctypes.sizeof(address))
+ if gdi.D3DKMTQueryAdapterInfo(ctypes.byref(query)) != 0:
+ return None
+ return address.bus, address.device, address.function
+ finally:
+ gdi.D3DKMTCloseAdapter(ctypes.byref(wintypes.UINT(opened.handle)))
+ except (OSError, AttributeError):
+ return None
+
+
def enumerate_adapters() -> list[Adapter]:
try:
dxgi = ctypes.WinDLL("dxgi")
@@ -73,7 +107,7 @@ def enumerate_adapters() -> list[Adapter]:
try:
desc = _DESC1()
if _vcall(adapter, 10, ctypes.c_long, ctypes.POINTER(_DESC1))(adapter, ctypes.byref(desc)) == 0:
- luid = (desc.AdapterLuid.HighPart << 32) | desc.AdapterLuid.LowPart
+ luid = ((desc.AdapterLuid.HighPart & 0xffffffff) << 32) | desc.AdapterLuid.LowPart
out.append(Adapter(
luid=luid,
name=desc.Description,
@@ -81,6 +115,7 @@ def enumerate_adapters() -> list[Adapter]:
dedicated_vram=desc.DedicatedVideoMemory,
shared_system=desc.SharedSystemMemory,
software=bool(desc.Flags & 2),
+ pci_address=adapter_pci_address(luid),
))
finally:
_vcall(adapter, 2, wintypes.ULONG)(adapter) # Release
diff --git a/src/gputaskman/nvml.py b/src/gputaskman/nvml.py
index 46e8c52..39d936d 100644
--- a/src/gputaskman/nvml.py
+++ b/src/gputaskman/nvml.py
@@ -2,7 +2,7 @@
from __future__ import annotations
-from dataclasses import dataclass, field
+from .telemetry import DeviceTelemetry as NvDevice
try:
import pynvml as N
@@ -10,36 +10,6 @@ except ImportError: # pragma: no cover
N = None
-@dataclass
-class NvDevice:
- index: int
- name: str = "?"
- pci_bus: str = ""
- driver_model: str = "" # WDDM / TCC / MCDM
- mem_total: int = 0
- mem_used: int = 0
- mem_reserved: int = 0
- util_gpu: float | None = None # % of time a kernel was running
- util_mem: float | None = None # % of time device memory was being read/written (bandwidth pressure)
- enc: float | None = None
- dec: float | None = None
- temp: int | None = None
- power_w: float | None = None
- power_limit_w: float | None = None
- fan: int | None = None
- pstate: int | None = None
- sm_clock: int | None = None
- sm_clock_max: int | None = None
- mem_clock: int | None = None
- video_clock: int | None = None
- pcie_rx_kbs: int | None = None # host -> GPU
- pcie_tx_kbs: int | None = None # GPU -> host
- pcie_gen: int | None = None
- pcie_width: int | None = None
- throttle: list[str] = field(default_factory=list)
- compute_cap: str = ""
-
-
def format_metric(value: float | int | None, spec: str = "", unit: str = "") -> str:
return f"{format(value, spec)}{unit}" if value is not None else "N/A"
diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.py
index 3f16e41..fafbc43 100644
--- a/src/gputaskman/pdh.py
+++ b/src/gputaskman/pdh.py
@@ -63,10 +63,17 @@ class ProcGpu:
shared: float = 0.0 # bytes resident in shared (system RAM) GPU memory -- "spill" ("Non Local Usage")
committed: float = 0.0 # all GPU memory this process allocated, resident or not ("Total Committed")
engines: dict[str, float] = field(default_factory=lambda: defaultdict(float)) # engtype -> util %
+ available: set[str] = field(default_factory=lambda: {"dedicated", "shared", "committed", "engine"})
+
+ @property
+ def paged_available(self) -> bool:
+ return {"dedicated", "shared", "committed"} <= self.available
@property
def paged_out(self) -> float:
"""Commit minus local/non-local usage: a non-residency estimate, not an eviction trace."""
+ if not self.paged_available:
+ return 0.0 # internal sentinel; callers must check paged_available before display
return max(self.committed - self.dedicated - self.shared, 0.0)
@property
@@ -77,8 +84,8 @@ class ProcGpu:
@dataclass
class AdapterGpu:
- dedicated: float = 0.0
- shared: float = 0.0
+ dedicated: float | None = None
+ shared: float | None = None
class GpuCounters:
@@ -136,13 +143,14 @@ class GpuCounters:
def sample(self) -> tuple[dict[int, dict[int, ProcGpu]], dict[int, AdapterGpu]]:
"""Returns ({luid: {pid: ProcGpu}}, {luid: AdapterGpu})."""
- _pdh.PdhCollectQueryData(self._query)
+ if _pdh.PdhCollectQueryData(self._query) != 0:
+ return {}, {}
procs: dict[int, dict[int, ProcGpu]] = defaultdict(dict)
def proc(luid: int, pid: int) -> ProcGpu:
p = procs[luid].get(pid)
if p is None:
- p = procs[luid][pid] = ProcGpu()
+ p = procs[luid][pid] = ProcGpu(available=set())
return p
for attr in ("dedicated", "shared", "committed"):
@@ -151,10 +159,13 @@ class GpuCounters:
if m:
p = proc(luid_key(m[2], m[3]), int(m[1]))
setattr(p, attr, getattr(p, attr) + val)
+ p.available.add(attr)
for name, val in self._read("engine"):
m = _ENGINE_RE.match(name)
if m:
- proc(luid_key(m[2], m[3]), int(m[1])).engines[_norm_engine(m[6])] += val
+ p = proc(luid_key(m[2], m[3]), int(m[1]))
+ p.engines[_norm_engine(m[6])] += val
+ p.available.add("engine")
adapters: dict[int, AdapterGpu] = defaultdict(AdapterGpu)
for attr, key in (("dedicated", "a_dedicated"), ("shared", "a_shared")):
@@ -162,7 +173,7 @@ class GpuCounters:
m = _ADAPTER_RE.match(name)
if m:
a = adapters[luid_key(m[1], m[2])]
- setattr(a, attr, getattr(a, attr) + val)
+ setattr(a, attr, (getattr(a, attr) or 0.0) + val)
return procs, adapters
def close(self) -> None:
diff --git a/src/gputaskman/procinfo.py b/src/gputaskman/procinfo.py
index 6ef0601..bb308b5 100644
--- a/src/gputaskman/procinfo.py
+++ b/src/gputaskman/procinfo.py
@@ -24,6 +24,7 @@ RUNTIMES: list[tuple[str, tuple[str, ...]]] = [
("DirectML", ("directml.dll",)),
("cuDNN", ("cudnn*.dll",)),
("CUDA", ("nvcuda.dll", "nvcuda64.dll", "cudart64_*.dll", "cublas64_*.dll")),
+ ("HIP", ("amdhip64.dll", "amdhip64_*.dll")),
("Vulkan", ("vulkan-1.dll",)),
("D3D12", ("d3d12core.dll",)),
("OpenGL", ("opengl32.dll",)),
@@ -267,6 +268,12 @@ class ProcInspector:
self._refresh_ports()
return self._ports.get(pid, [])
+ def exclusive_ports(self, pid: int) -> list[int]:
+ """Only probe ports with one known listener; localhost may route shared ports to another PID."""
+ self._refresh_ports()
+ others = {port for other, ports in self._ports.items() if other != pid for port in ports}
+ return [port for port in self._ports.get(pid, []) if port not in others]
+
@staticmethod
def _new(pid: int) -> ProcDetail:
d = ProcDetail(pid=pid)
@@ -370,7 +377,7 @@ def classify(d: ProcDetail, engines: dict[str, float]) -> str:
return "browser"
if video or "NVENC" in d.runtimes:
return "video"
- if "CUDA" in d.runtimes:
+ if "CUDA" in d.runtimes or "HIP" in d.runtimes:
return "compute"
if ("D3D12" in d.runtimes or "Vulkan" in d.runtimes) and engines.get("3d", 0) > 10:
return "3D/game"
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index 289d871..7106f84 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -17,9 +17,12 @@ def to_dict(s: Snapshot) -> dict:
for p in s.procs:
item = {
"pid": p.pid, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,
- "vram_bytes": int(p.gpu.dedicated), "shared_bytes": int(p.gpu.shared),
- "committed_bytes": int(p.gpu.committed), "paged_out_bytes": int(p.gpu.paged_out),
- "gpu_pct": round(p.gpu.gpu_pct, 1), "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v},
+ "vram_bytes": int(p.gpu.dedicated) if "dedicated" in p.gpu.available else None,
+ "shared_bytes": int(p.gpu.shared) if "shared" in p.gpu.available else None,
+ "committed_bytes": int(p.gpu.committed) if "committed" in p.gpu.available else None,
+ "paged_out_bytes": int(p.gpu.paged_out) if p.gpu.paged_available else None,
+ "gpu_pct": round(p.gpu.gpu_pct, 1) if "engine" in p.gpu.available else None,
+ "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if "engine" in p.gpu.available else None,
"ports": p.detail.ports, "inside": p.inside,
"model_files": [dataclasses.asdict(m) for m in p.detail.models],
"installed_models": [dataclasses.asdict(m) for m in p.detail.candidates],
@@ -45,6 +48,7 @@ def to_dict(s: Snapshot) -> dict:
procs.append(item)
return {
"taken": s.taken, "gpu": s.adapter.name if s.adapter else None, "driver": s.driver, "cuda": s.cuda,
+ "adapter": dataclasses.asdict(s.adapter) if s.adapter else None,
"vram_total": s.vram_total, "vram_used": s.vram_used, "shared_used": s.shared_used,
"attributed": s.attributed, "device": nv, "processes": procs,
}
@@ -54,26 +58,37 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
out = []
nv = s.nv
name = s.adapter.name if s.adapter else "GPU"
- out.append(f"{name} driver {s.driver} CUDA {s.cuda} {nv.driver_model if nv else ''}")
- out.append(f"VRAM {gb(s.vram_used)} / {gb(s.vram_total)} shared (in system RAM) {gb(s.shared_used)}")
+ info = f" driver {s.driver}" if s.driver else ""
+ if s.cuda:
+ info += f" CUDA {s.cuda}"
+ out.append(f"{name}{info} {nv.driver_model if nv else ''}")
+ used = gb(s.vram_used) if s.vram_used is not None else "N/A"
+ shared = gb(s.shared_used) if s.shared_used is not None else "N/A"
+ out.append(f"VRAM {used} / {gb(s.vram_total)} shared (in system RAM) {shared}")
if nv:
rx = nv.pcie_rx_kbs / 1024 if nv.pcie_rx_kbs is not None else None
tx = nv.pcie_tx_kbs / 1024 if nv.pcie_tx_kbs is not None else None
pstate = f"P{nv.pstate}" if nv.pstate is not None else "P N/A"
- out.append(f"GPU {format_metric(nv.util_gpu, unit='%')} mem-bandwidth {format_metric(nv.util_mem, unit='%')} "
+ label = "AMD gfx" if nv.source.startswith("AMD ") else "GPU"
+ clock = (f"gfx {format_metric(nv.gfx_clock)}" if nv.source.startswith("AMD ") else
+ f"SM {format_metric(nv.sm_clock)}")
+ power_kind = f" ({nv.power_kind})" if nv.power_kind else ""
+ out.append(f"{label} {format_metric(nv.util_gpu, unit='%')} mem-bandwidth {format_metric(nv.util_mem, unit='%')} "
f"{format_metric(nv.temp, unit='C')} {format_metric(nv.power_w, '.0f')}/"
- f"{format_metric(nv.power_limit_w, '.0f')} W {pstate} "
- f"SM {format_metric(nv.sm_clock)} MHz "
+ f"{format_metric(nv.power_limit_w, '.0f')} W{power_kind} {pstate} "
+ f"{clock} MHz "
f"video {format_metric(nv.video_clock)} MHz "
f"PCIe rx {format_metric(rx, '.0f')} MB/s "
f"tx {format_metric(tx, '.0f')} MB/s")
out.append("")
out.append(f"{'PID':>7} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")
for p in s.procs[:limit]:
- spill = gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""
- paged = gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""
- out.append(f"{p.pid:>7} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {gb(p.gpu.dedicated):>9} {spill:>8} "
- f"{paged:>8} {p.gpu.gpu_pct:>5.0f} {p.inside}")
+ spill = "N/A" if "shared" not in p.gpu.available else gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""
+ paged = "N/A" if not p.gpu.paged_available else gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""
+ vram = gb(p.gpu.dedicated) if "dedicated" in p.gpu.available else "N/A"
+ gpu_pct = f"{p.gpu.gpu_pct:.0f}" if "engine" in p.gpu.available else "N/A"
+ out.append(f"{p.pid:>7} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {vram:>9} {spill:>8} "
+ f"{paged:>8} {gpu_pct:>5} {p.inside}")
for p in s.procs:
if p.handy:
out.append("")
diff --git a/src/gputaskman/telemetry.py b/src/gputaskman/telemetry.py
new file mode 100644
index 0000000..ee2ce21
--- /dev/null
+++ b/src/gputaskman/telemetry.py
@@ -0,0 +1,38 @@
+"""Optional device-wide telemetry shared by vendor providers.
+
+Percentages and clocks are provider-specific; unsupported metrics are None.
+"""
+from dataclasses import dataclass, field
+
+
+@dataclass
+class DeviceTelemetry:
+ index: int
+ name: str = "?"
+ pci_bus: str = ""
+ driver_model: str = ""
+ mem_total: int | None = None
+ mem_used: int | None = None
+ mem_reserved: int | None = None
+ util_gpu: float | None = None
+ util_mem: float | None = None
+ enc: float | None = None
+ dec: float | None = None
+ temp: float | None = None
+ power_w: float | None = None
+ power_kind: str = ""
+ power_limit_w: float | None = None
+ fan: int | None = None
+ pstate: int | None = None
+ sm_clock: int | None = None
+ sm_clock_max: int | None = None
+ gfx_clock: int | None = None
+ mem_clock: int | None = None
+ video_clock: int | None = None
+ pcie_rx_kbs: int | None = None
+ pcie_tx_kbs: int | None = None
+ pcie_gen: int | None = None
+ pcie_width: int | None = None
+ throttle: list[str] = field(default_factory=list)
+ compute_cap: str = ""
+ source: str = "NVML"
diff --git a/tests/test_core.py b/tests/test_core.py
index 2896160..c66a628 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import contextlib
+import ctypes
import io
import json
import os
@@ -17,6 +18,8 @@ from rich.table import Table
from textual.widgets import Footer, Select, Static
from gputaskman.app import GpuTaskManager, spark
+from gputaskman.amdsmi import AmdSmi
+from gputaskman.adl import Adl, _Metrics
from gputaskman.cli import main
from gputaskman.collector import ProcRow, Snapshot, select_adapter, select_nv_device
from gputaskman.dxgi import Adapter
@@ -26,7 +29,7 @@ from gputaskman.llm import (LlmView, ServerInfo, build_view, is_gpu_device, llam
parse_flags, parse_load_log, probe_server)
from gputaskman.nvml import NvDevice, Nvml
from gputaskman.pdh import GpuCounters, ProcGpu
-from gputaskman.procinfo import (AI_KINDS, MappedModel, ProcDetail, ProcInspector, _is_model_file,
+from gputaskman.procinfo import (AI_KINDS, MappedModel, ProcDetail, ProcInspector, _is_model_file, _match_runtimes,
_models_from_cmdline, classify, installed_models, make_model)
from gputaskman.report import to_dict, to_text
from gputaskman.stores import store_name
@@ -100,6 +103,94 @@ class CliTests(unittest.TestCase):
class DeviceSelectionTests(unittest.TestCase):
+ def test_adl_empty_or_failed_sample_allows_fallback(self):
+ provider = Adl.__new__(Adl)
+ provider.context, provider.index = ctypes.c_void_p(1), 4
+ provider.dll = SimpleNamespace(ADL2_New_QueryPMLogData_Get=lambda *args: 0,
+ ADL2_Main_Control_Destroy=lambda context: None)
+ self.assertIsNone(provider.sample())
+ provider.dll.ADL2_New_QueryPMLogData_Get = lambda *args: -1
+ self.assertIsNone(provider.sample())
+ provider.close()
+ provider.close()
+ self.assertIsNone(provider.sample())
+
+ def test_shared_listen_port_is_not_used_for_server_attribution(self):
+ inspector = ProcInspector()
+ inspector._ports = {1: [9090, 9091], 2: [9090]}
+ with patch.object(inspector, "_refresh_ports"):
+ self.assertEqual(inspector.exclusive_ports(1), [9091])
+ self.assertEqual(inspector.exclusive_ports(2), [])
+
+ def test_adl_uses_only_supported_sensors_and_reports_board_power(self):
+ def read(_context, _index, pointer):
+ sensors = ctypes.cast(pointer, ctypes.POINTER(_Metrics)).contents.sensors
+ for index, value in ((1, 2500), (8, 55), (19, 0), (73, 120)):
+ sensors[index].supported, sensors[index].value = 1, value
+ return 0
+
+ provider = Adl.__new__(Adl)
+ provider.context, provider.index = ctypes.c_void_p(1), 4
+ provider.dll = SimpleNamespace(ADL2_New_QueryPMLogData_Get=read)
+ sample = provider.sample()
+ self.assertEqual((sample.gfx_clock, sample.temp, sample.util_gpu), (2500, 55, 0))
+ self.assertEqual((sample.power_w, sample.power_kind), (120, "board"))
+ self.assertIsNone(sample.mem_clock)
+ self.assertIsNone(sample.util_mem)
+
+ def test_amdsmi_matches_pci_and_handles_partial_metrics(self):
+ amd = Adapter(1, "AMD", 0x1002, 16 * GiB, 0, False, (10, 0, 0))
+ closed = []
+ api = SimpleNamespace(
+ amdsmi_init=lambda: None, amdsmi_shut_down=lambda: closed.append(True),
+ amdsmi_get_processor_handles=lambda: ["other", "target"],
+ amdsmi_get_gpu_device_bdf=lambda h: "0000:0a:00.0" if h == "target" else "0000:0b:00.0",
+ amdsmi_get_gpu_driver_info=lambda h: {"driver_version": "test"},
+ amdsmi_get_gpu_activity=lambda h: {"gfx_activity": 0},
+ amdsmi_get_power_info=lambda h: {"socket_power": 42, "power_limit": "N/A"},
+ AmdSmiTemperatureType=SimpleNamespace(EDGE=1),
+ AmdSmiTemperatureMetric=SimpleNamespace(CURRENT=2),
+ amdsmi_get_temp_metric=lambda *args: 47500,
+ )
+ provider = AmdSmi(amd, [amd], api)
+ sample = provider.sample()
+ self.assertEqual((sample.source, sample.util_gpu, sample.temp, sample.power_w), ("AMD SMI", 0, 47.5, 42))
+ self.assertIsNone(sample.power_limit_w)
+ self.assertIsNone(sample.util_mem)
+ provider.close()
+ provider.close()
+ self.assertEqual(closed, [True])
+ self.assertIsNone(provider.sample())
+
+ def test_amdsmi_rejects_ambiguous_and_nonzero_domain(self):
+ amd = Adapter(1, "AMD", 0x1002, 16 * GiB, 0, False, (10, 0, 0))
+ other = Adapter(2, "AMD", 0x1002, 16 * GiB, 0, False, (10, 0, 0))
+ api = SimpleNamespace(amdsmi_init=lambda: None, amdsmi_shut_down=lambda: None,
+ amdsmi_get_processor_handles=lambda: [1],
+ amdsmi_get_gpu_device_bdf=lambda h: "0001:0a:00.0")
+ self.assertIsNone(AmdSmi(amd, [amd, other], api).sample())
+ provider = AmdSmi(amd, [amd], api)
+ self.assertIsNone(provider.sample())
+ provider.close()
+
+ def test_mixed_vendor_adapters_are_selectable(self):
+ amd = Adapter(2, "AMD GPU", 0x1002, 24 * GiB, 0, False)
+ nvidia = Adapter(1, "NVIDIA GPU", 0x10DE, 8 * GiB, 0, False)
+ self.assertIs(select_adapter([amd, nvidia], 0), nvidia)
+ self.assertIs(select_adapter([amd, nvidia], 1), amd)
+
+ def test_amd_snapshot_does_not_show_cuda_header(self):
+ s = snapshot()
+ s.adapter = Adapter(2, "AMD GPU", 0x1002, 24 * GiB, 0, False)
+ self.assertNotIn("CUDA", to_text(s).splitlines()[0])
+ self.assertEqual(to_dict(s)["adapter"]["vendor_id"], 0x1002)
+
+ def test_hip_runtime_is_compute_evidence(self):
+ self.assertEqual(classify(ProcDetail(1, "worker.exe", runtimes=["HIP"]), {}), "compute")
+ for dll in ("amdhip64.dll", "amdhip64_6.dll"):
+ self.assertIn("HIP", _match_runtimes({dll}))
+ self.assertNotIn("HIP", _match_runtimes({"amd_comgr_2.dll"}))
+
def test_adapter_indices_do_not_clamp_to_another_gpu(self):
gpu = Adapter(1, "NVIDIA GPU", 0x10DE, 16 * GiB, 0, False)
software = Adapter(2, "Software", 0x1414, 0, 0, True)
@@ -110,18 +201,64 @@ class DeviceSelectionTests(unittest.TestCase):
with self.assertRaisesRegex(ValueError, "out of range"):
select_adapter([], 1)
- def test_nvml_matches_unique_adapter_name_before_index(self):
- gpu = Adapter(1, "NVIDIA RTX Spark", 0x10DE, 64 * GiB, 0, False)
+ def test_nvml_matches_pci_address_not_name_or_index(self):
+ gpu = Adapter(1, "NVIDIA RTX Spark", 0x10DE, 64 * GiB, 0, False, (10, 0, 0))
npu = NvDevice(index=0, name="NVIDIA NPU", driver_model="MCDM")
- nv_gpu = NvDevice(index=1, name="NVIDIA RTX SPARK", driver_model="WDDM")
+ nv_gpu = NvDevice(index=1, name="NVIDIA RTX SPARK", pci_bus="00000000:0A:00.0", driver_model="WDDM")
self.assertIs(select_nv_device(gpu, [npu, nv_gpu], 0), nv_gpu)
self.assertIs(select_nv_device(None, [npu, nv_gpu], 0), npu)
self.assertIsNone(select_nv_device(gpu, [npu], 0))
+ self.assertIsNone(select_nv_device(gpu, [nv_gpu, nv_gpu], 0))
intel = Adapter(2, "Intel GPU", 0x8086, 8 * GiB, 0, False)
self.assertIsNone(select_nv_device(intel, [nv_gpu], 0))
class MemoryTests(unittest.TestCase):
+ def test_failed_collection_does_not_reuse_previous_counters(self):
+ counters = GpuCounters.__new__(GpuCounters)
+ counters._query = None
+ with patch("gputaskman.pdh._pdh", SimpleNamespace(PdhCollectQueryData=lambda query: 1)), \
+ patch.object(counters, "_read") as read:
+ self.assertEqual(counters.sample(), ({}, {}))
+ read.assert_not_called()
+
+ def test_counter_samples_track_each_process_field(self):
+ counters = GpuCounters.__new__(GpuCounters)
+ counters._query = None
+ data = {"dedicated": [("pid_42_luid_0x0_0x1_phys_0", 0.0)],
+ "engine": [("pid_42_luid_0x0_0x1_phys_0_eng_0_engtype_Compute", 10.0)]}
+ with patch.object(counters, "_read", side_effect=lambda key: data.get(key, [])), \
+ patch("gputaskman.pdh._pdh", SimpleNamespace(PdhCollectQueryData=lambda query: 0)):
+ processes, _ = counters.sample()
+ gpu = processes[1][42]
+ self.assertEqual(gpu.available, {"dedicated", "engine"})
+ self.assertEqual(gpu.dedicated, 0)
+ self.assertEqual(gpu.gpu_pct, 10)
+ self.assertFalse(gpu.paged_available)
+
+ def test_missing_process_counters_are_not_reported_as_zero(self):
+ s = snapshot()
+ s.procs[0].gpu.available = {"dedicated"}
+ row = to_dict(s)["processes"][0]
+ self.assertEqual(row["vram_bytes"], 8 * GiB)
+ for name in ("shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):
+ self.assertIsNone(row[name])
+ self.assertIn("N/A", to_text(s))
+ s.procs[0].gpu.available.clear()
+ self.assertIsNone(to_dict(s)["attributed"])
+
+ def test_missing_adapter_memory_is_not_reported_as_zero(self):
+ s = snapshot()
+ s.vram_used = None
+ s.shared_used = None
+ self.assertIsNone(to_dict(s)["vram_used"])
+ self.assertIsNone(to_dict(s)["shared_used"])
+ self.assertIn("VRAM N/A", to_text(s))
+ self.assertIn("shared (in system RAM) N/A", to_text(s))
+ output = io.StringIO()
+ Console(file=output, width=120).print(GpuTaskManager().render_device(s))
+ self.assertIn("N/A", output.getvalue())
+
def test_local_counters_are_preferred(self):
self.assertTrue(GpuCounters._COUNTERS["dedicated"][0].endswith(r"\Local Usage"))
self.assertTrue(GpuCounters._COUNTERS["shared"][0].endswith(r"\Non Local Usage"))
diff --git a/tools/build_portable.py b/tools/build_portable.py
index fd70cce..8ef8226 100644
--- a/tools/build_portable.py
+++ b/tools/build_portable.py
@@ -52,7 +52,7 @@ def check_executable(path: Path, arch: str) -> None:
def create_archive(bundle: Path, output: Path, version: str, arch: str) -> Path:
validate_version(version)
check_executable(bundle / "gputaskman.exe", arch)
- forbidden = {"nvml.dll", "nvcuda.dll", "dxgi.dll", "pdh.dll"}
+ forbidden = {"nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll"}
# Editable-install origins contain build-machine paths and are not needed at runtime.
files = sorted(
path for path in bundle.rglob("*")
@@ -107,7 +107,7 @@ def add_notices(bundle: Path) -> None:
"PyInstaller bootloader: see licenses\\pyinstaller (including its exception).\n"
"NVIDIA Python bindings: see licenses\\nvidia-ml-py.\n"
"Runtime package metadata and license texts are included in _internal\\*.dist-info.\n"
- "NVIDIA and Windows DLLs are provided by the installed driver and operating system.\n",
+ "GPU driver and Windows DLLs are provided by the installed driver and operating system.\n",
encoding="utf-8",
)
@@ -139,7 +139,7 @@ def build(output: Path, arch: str, tag: str | None = None) -> Path:
" .\\gputaskman.exe --once\n"
" .\\gputaskman.exe --json\n\n"
"Press q to quit the interactive display.\n"
- "Requires Windows and an NVIDIA GPU with a current WDDM driver.\n"
+ "Requires Windows and an NVIDIA or AMD GPU with a current WDDM driver.\n"
"No Python, uv, CUDA Toolkit, models, or administrator session is required.\n"
"This build is unsigned. No drivers or inference servers are bundled.\n"
"Snapshots can contain local paths and model names; redact before sharing.\n\n"
diff --git a/tools/portable.spec b/tools/portable.spec
index 2a72a0c..68dd7a2 100644
--- a/tools/portable.spec
+++ b/tools/portable.spec
@@ -9,8 +9,8 @@ analysis = Analysis(
datas=collect_data_files("textual") + copy_metadata("gputaskman", recursive=True),
)
-# These belong to Windows or the installed NVIDIA driver, not the portable app.
-driver_dlls = {"nvml.dll", "nvcuda.dll", "dxgi.dll", "pdh.dll"}
+# These belong to Windows or the installed GPU driver, not the portable app.
+driver_dlls = {"nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll"}
analysis.binaries = [
entry for entry in analysis.binaries
if Path(entry[0]).name.lower() not in driver_dlls