A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
feat: adding support for AMD, tested with Vulkan and ROCm/HIP processes (ex. llama) on a Radeon 9060XT 16GB, Windows.
Kdbf61a8a6459
diff --git a/README.md b/README.mdindex ea05653..b560062 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@**An `htop` for what's inside your GPU, not just how full it is.**-Windows + NVIDIA. A terminal UI that connects per-process GPU memory and engine activity+Windows + NVIDIA or AMD. A terminal UI that connects per-process GPU memory and engine activitywith loaded runtimes, model files, and LLM buffer breakdowns.Task Manager says "15 GB used." This tries to answer: *by what? Model weights? KV cache?@@ -36,14 +36,14 @@ or PowerShell in the extracted folder:.\gputaskman.exe --json```-No Python, uv, CUDA Toolkit, model download, or installation is needed. An NVIDIA GPU with a-current Windows WDDM driver is still required. Builds are currently **unsigned**; code signing+No Python, uv, CUDA Toolkit, model download, or installation is needed. A supported GPU with a+current Windows WDDM driver is required. Builds are currently **unsigned**; code signingand an installer are not part of this initial distribution. Each ZIP includes licenses and a`.zip.sha256` companion file is published for integrity checks.## Clone and run-Use **64-bit Windows 10/11**, an **NVIDIA GPU with a current WDDM driver**, and a terminal+Use **64-bit Windows 10/11**, an **NVIDIA or AMD GPU with a current WDDM driver**, and a terminal(Windows Terminal is recommended). Live workload checks have included an RTX 4080 SUPER and anRTX Spark N1X on Windows ARM64. CUDA workloads are supported on ARM64 with a compatible NVIDIAdriver and ARM64 CUDA runtime.@@ -67,7 +67,7 @@ uv run --locked gputaskman**That's it.** uv obtains the pinned Python version if needed, creates `.venv`, and installsthe locked dependencies. The first run needs internet access. There is no CUDA Toolkit,Visual Studio, administrator session, API key, or model download required to monitor the GPU.-NVML ships with the NVIDIA driver.+NVML ships with the NVIDIA driver; AMD ADL is supplied by supported AMD Windows drivers.Already have Python 3.11+ and prefer pip?@@ -91,7 +91,7 @@ skips the consistency check, so do not use it after changing dependencies.- **Device view:** VRAM usage, GPU activity, memory activity, PCIe traffic, temperature, power,video-engine clock, and responsive history charts with an all-charts or single-metric view.- **Process view:** local GPU memory, GPU-accessible system RAM, estimated non-resident commit,- engine activity, and detected runtimes such as CUDA, Vulkan, Direct3D, ggml and ONNX Runtime.+ engine activity, and detected runtimes such as CUDA, HIP, Vulkan, Direct3D, ggml and ONNX Runtime.- **Model inspector:** which model each process holds (LLM, decision, embedding, speech, or diffusion), howit was identified, architecture, quantization, layer count, weights, estimated KV cost, andbuffer sizes from compatible load logs.@@ -140,28 +140,45 @@ uv run --locked gputaskman --help| `q` | Quit |A wide terminal (roughly 140 columns or more) makes the full table easier to read.-All eight history charts are shown by default: GPU, SM clock, VRAM, memory activity,+All eight history charts are shown by default: GPU, SM clock (GFX clock on AMD), VRAM, memory activity,temperature, power, encoder, and decoder. They stack in narrow terminals and share rows asthe window widens. Click the **Charts** dropdown to focus on one metric or return to **All charts**.A blank sample means no reading; zero is drawn at the baseline. Percentages use a 0–100%-scale, temperature uses 0–120°C, and the power and SM clock scales are displayed below the charts.+scale, temperature uses 0–120°C, and the power and clock scales are displayed below the charts.The dashboard and inspector scroll independently in short terminals so the process list stays usable.Model/runtime discovery is throttled, so newly loaded models may take about 15 seconds to appear.-`--gpu N` selects a detected NVIDIA graphics adapter, ordered by dedicated memory; an out-of-range+`--list-gpus` lists selectable hardware adapters with their vendor and LUID. `--gpu N` selects+one of these adapters: NVIDIA adapters are listed first, followed by other vendors, with each+group ordered by dedicated memory. An out-of-rangeindex is rejected rather than silently selecting another device. Compute-only devices such as an-NPU are not graphics adapters. **Single-NVIDIA-GPU systems are the validated configuration**.-On multi-GPU systems, adapter names may be ambiguous, DXGI and NVML ordering may differ, and buffer-logs can span devices; cross-device attribution is not yet reliable.+NPU are not graphics adapters. AMD and NVIDIA hardware coverage is described below; configurations+with multiple GPUs and mixed vendors need further validation.+Device telemetry is matched to WDDM adapters by PCI location, not display name or enumeration order.+If a unique match cannot be established, telemetry is unavailable; buffer logs can still span devices,+so model-buffer attribution across GPUs is not yet reliable. This conservative matching also applies+to NVIDIA and may suppress telemetry when Windows does not expose a usable PCI address.++AMD adapters can be selected and use the existing Windows GPU process counters; HIP runtime DLLs+are recognized as runtime evidence. On supported drivers, the AMD Display Library supplied by the+Windows driver can provide graphics activity, clocks, edge temperature, board or ASIC power, fan+and PCIe lane count. An optional AMD SMI Python installation is a fallback and may provide graphics+activity, edge temperature and socket power. Both require a PCI match to the selected WDDM adapter.+The SMI library and its native dependencies must be available to the Python environment; it is not+bundled in the portable build. The RX 9060 XT HIP workload and ADL telemetry have been observed on+this machine; AMD SMI telemetry and Vulkan workloads remain unvalidated.+A loaded DLL does not prove GPU execution.Unavailable utilization, temperature, power, clock and PCIe readings are displayed as `N/A` intext and the TUI and as `null` in JSON. For example, a missing NVML power limit means that thedriver did not expose that reading; it does not by itself indicate a hardware fault.+Missing per-process memory or engine samples likewise show `N/A`/`null`; the paged estimate is+unavailable when any of its input counters is missing.## How can it know what's inside?It combines evidence rather than reading arbitrary GPU memory. Windows' graphics kernel supplies-per-process memory and engine counters, while NVML supplies device metrics. Process DLLs suggest+per-process memory and engine counters, while NVML or AMD ADL/SMI supplies device metrics. Process DLLs suggestthe runtime, and mapped files or command-line arguments identify candidate models. GGUF metadataand tensor offsets let it estimate weights and KV-cache costs; compatible runtime logs and localserver APIs add allocated buffer sizes and live state. That can explain a llama.cpp process well,@@ -169,7 +186,7 @@ but a runtime DLL or filename alone does not prove which allocation contains whi| Question | Evidence | Confidence / limitation ||---|---|---|-| How full and busy is the card? | WDDM adapter counters and NVML | Measured; driver support varies by metric |+| How full and busy is the card? | WDDM adapter counters and NVML or AMD ADL/SMI | Measured; driver support varies by metric || Which process uses GPU memory? | WDDM process counters | Measured accounting, not a resource ownership graph || Which engines are active? | WDDM engine counters | CUDA can appear on the **3D** engine || Which runtime is loaded? | Mapped DLL names | Detection, not proof that every library is doing GPU work |@@ -351,7 +368,7 @@ file paths, but model and process names still require review.|---|---|| `uv` or `git` is not recognized | Install the prerequisites, then open a new terminal || Windows-only message | Run on native Windows, not Linux, macOS, or WSL |-| Empty or incomplete GPU readings | Check the NVIDIA driver and WDDM 2.0+ support in `dxdiag`; use a normal local desktop session |+| Empty or incomplete GPU readings | Check the GPU driver and WDDM 2.0+ support in `dxdiag`; use a normal local desktop session || Missing DLLs or model details | Protected processes may deny inspection; basic counters can still work without elevation || No exact LLM breakdown | Enable compatible buffer logging, or use the GGUF estimate; API/log access is optional || The screen looks cramped | Widen the terminal or use `--once` / `--json` |@@ -380,7 +397,7 @@ uv run --locked --group release python tools\build_portable.py --arch x64The builder verifies the interpreter and executable architectures, includes Textual resourcesand dependency licenses, and writes the complete portable ZIP and SHA-256 file to `dist`.-It does not bundle Windows/NVIDIA DLLs, inference servers, or models. Keep uv for development;+It does not bundle Windows or GPU driver DLLs, inference servers, or models. Keep uv for development;the executable ZIP is for running the monitor without a Python environment.The **Portable Windows release** GitHub Actions workflow builds on `windows-2022` (x64) and@@ -404,6 +421,7 @@ downloads are bundled in this repository.|---|---|| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration || `nvml.py` | NVIDIA device readings |+| `adl.py`, `amdsmi.py` | Optional AMD device readings || `procinfo.py` | Runtime, mapped-file, and process discovery || `stores.py` | Friendly names from Ollama, LM Studio, and Hugging Face model stores || `gguf.py` | Model metadata, kind, and size estimates |diff --git a/src/gputaskman/adl.py b/src/gputaskman/adl.pynew file mode 100644index 0000000..dc48027--- /dev/null+++ b/src/gputaskman/adl.py@@ -0,0 +1,101 @@+"""Read-only AMD Display Library telemetry supplied by the Windows driver.++ADL2_New_QueryPMLogData_Get is deprecated by AMD; failures leave data unavailable.+Definitions: https://github.com/GPUOpen-LibrariesAndSDKs/display-library/tree/master/include+"""+from __future__ import annotations++import ctypes+import os++from .dxgi import Adapter, VENDOR_AMD+from .telemetry import DeviceTelemetry+++class _AdapterInfo(ctypes.Structure):+ _fields_ = [("size", ctypes.c_int), ("index", ctypes.c_int), ("udid", ctypes.c_char * 256),+ ("bus", ctypes.c_int), ("device", ctypes.c_int), ("function", ctypes.c_int),+ ("vendor", ctypes.c_int), ("name", ctypes.c_char * 256),+ ("display", ctypes.c_char * 256), ("present", ctypes.c_int), ("exists", ctypes.c_int),+ ("driver", ctypes.c_char * 256), ("driver_ext", ctypes.c_char * 256),+ ("pnp", ctypes.c_char * 256), ("os_index", ctypes.c_int)]+++class _Sensor(ctypes.Structure):+ _fields_ = [("supported", ctypes.c_int), ("value", ctypes.c_int)]+++class _Metrics(ctypes.Structure):+ _fields_ = [("size", ctypes.c_int), ("sensors", _Sensor * 256)]+++class Adl:+ def __init__(self, adapter: Adapter | None, adapters: list[Adapter]):+ self.dll = None+ self.context = ctypes.c_void_p()+ self.index = None+ self.driver = ""+ if not adapter or adapter.vendor_id != VENDOR_AMD or adapter.pci_address is None:+ return+ if sum(a.pci_address == adapter.pci_address for a in adapters if not a.software) != 1:+ return+ try:+ # System32 is the AMD driver's installation location, not the working directory.+ self.dll = ctypes.CDLL(os.path.join(os.environ.get("SystemRoot", r"C:\Windows"), "System32", "atiadlxx.dll"))+ crt = ctypes.CDLL("msvcrt")+ crt.malloc.argtypes = [ctypes.c_size_t]+ crt.malloc.restype = ctypes.c_void_p+ self._malloc = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_int)(crt.malloc)+ self.dll.ADL2_Main_Control_Create.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(ctypes.c_void_p)]+ self.dll.ADL2_Main_Control_Destroy.argtypes = [ctypes.c_void_p]+ self.dll.ADL2_Adapter_NumberOfAdapters_Get.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_int)]+ self.dll.ADL2_Adapter_AdapterInfo_Get.argtypes = [ctypes.c_void_p, ctypes.POINTER(_AdapterInfo), ctypes.c_int]+ self.dll.ADL2_New_QueryPMLogData_Get.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.POINTER(_Metrics)]+ if self.dll.ADL2_Main_Control_Create(self._malloc, 1, ctypes.byref(self.context)) != 0:+ return+ count = ctypes.c_int()+ if self.dll.ADL2_Adapter_NumberOfAdapters_Get(self.context, ctypes.byref(count)) != 0 or not 0 < count.value <= 256:+ return+ entries = (_AdapterInfo * count.value)()+ for entry in entries:+ entry.size = ctypes.sizeof(_AdapterInfo)+ if self.dll.ADL2_Adapter_AdapterInfo_Get(self.context, entries, ctypes.sizeof(entries)) != 0:+ return+ matches = {entry.index for entry in entries if (entry.bus, entry.device, entry.function) == adapter.pci_address}+ # ADL may return several logical display handles for the same physical GPU.+ if matches:+ self.index = min(matches)+ except (OSError, AttributeError, ValueError):+ self.close()+ finally:+ if self.index is None:+ self.close()++ def sample(self) -> DeviceTelemetry | None:+ if self.index is None:+ return None+ metrics = _Metrics()+ metrics.size = ctypes.sizeof(metrics)+ try:+ status = self.dll.ADL2_New_QueryPMLogData_Get(self.context, self.index, ctypes.byref(metrics))+ except (OSError, AttributeError):+ return None+ if status != 0 or not any(item.supported for item in metrics.sensors):+ return None+ def sensor(index):+ item = metrics.sensors[index]+ return item.value if item.supported and item.value >= 0 else None+ board_power = sensor(73)+ asic_power = sensor(23)+ return DeviceTelemetry(index=self.index, source="AMD ADL", gfx_clock=sensor(1), mem_clock=sensor(2),+ temp=sensor(8), fan=sensor(15), util_gpu=sensor(19),+ power_w=board_power if board_power is not None else asic_power,+ power_kind="board" if board_power is not None else "ASIC" if asic_power is not None else "",+ pcie_width=sensor(41))++ def close(self):+ self.index = None+ if self.dll is not None and self.context.value:+ context = self.context+ self.context = ctypes.c_void_p()+ self.dll.ADL2_Main_Control_Destroy(context)diff --git a/src/gputaskman/amdsmi.py b/src/gputaskman/amdsmi.pynew file mode 100644index 0000000..a94f1ef--- /dev/null+++ b/src/gputaskman/amdsmi.py@@ -0,0 +1,80 @@+"""Optional read-only AMD SMI telemetry. WDDM collection works without this module."""+from __future__ import annotations++import importlib+import math+import re++from .dxgi import Adapter, VENDOR_AMD+from .telemetry import DeviceTelemetry+++def _number(value):+ return value if isinstance(value, (int, float)) and not isinstance(value, bool) and math.isfinite(value) else None+++def _bdf(value):+ match = re.fullmatch(r"([0-9a-fA-F]+):([0-9a-fA-F]+):([0-9a-fA-F]+)\.([0-9a-fA-F]+)", str(value))+ if not match:+ return None+ domain, bus, device, function = (int(part, 16) for part in match.groups())+ # WDDM's ADAPTERADDRESS has no domain. Refuse nonzero domains rather than guess.+ return (bus, device, function) if domain == 0 else None+++class AmdSmi:+ def __init__(self, adapter: Adapter | None, adapters: list[Adapter], api=None):+ self.api = None+ self.handle = None+ self.driver = ""+ self.initialized = False+ if not adapter or adapter.vendor_id != VENDOR_AMD or adapter.pci_address is None:+ return+ if sum(a.pci_address == adapter.pci_address for a in adapters if not a.software) != 1:+ return+ try:+ self.api = api or importlib.import_module("amdsmi")+ self.api.amdsmi_init()+ self.initialized = True+ matches = [h for h in self.api.amdsmi_get_processor_handles()+ if _bdf(self._call("amdsmi_get_gpu_device_bdf", h)) == adapter.pci_address]+ if len(matches) == 1:+ self.handle = matches[0]+ info = self._call("amdsmi_get_gpu_driver_info", self.handle) or {}+ if isinstance(info, dict):+ self.driver = str(info.get("driver_version") or "")+ except Exception:+ self.close()++ def _call(self, name, *args):+ try:+ return getattr(self.api, name)(*args)+ except Exception:+ return None++ def sample(self) -> DeviceTelemetry | None:+ if self.handle is None:+ return None+ d = DeviceTelemetry(index=0, source="AMD SMI")+ activity = self._call("amdsmi_get_gpu_activity", self.handle)+ if isinstance(activity, dict):+ d.util_gpu = _number(activity.get("gfx_activity"))+ power = self._call("amdsmi_get_power_info", self.handle)+ if isinstance(power, dict):+ d.power_w = _number(power.get("socket_power"))+ d.power_kind = "socket" if d.power_w is not None else ""+ d.power_limit_w = _number(power.get("power_limit"))+ try:+ temp = self._call("amdsmi_get_temp_metric", self.handle,+ self.api.AmdSmiTemperatureType.EDGE, self.api.AmdSmiTemperatureMetric.CURRENT)+ value = _number(temp)+ d.temp = value / 1000 if value is not None else None+ except AttributeError:+ pass+ return d++ def close(self):+ self.handle = None+ if self.initialized:+ self._call("amdsmi_shut_down")+ self.initialized = Falsediff --git a/src/gputaskman/app.py b/src/gputaskman/app.pyindex 41dbfe9..f022019 100644--- a/src/gputaskman/app.py+++ b/src/gputaskman/app.py@@ -42,8 +42,8 @@ EXPLAIN = {"Deeper: edge://gpu, or DevTools > Performance with the GPU track.","3D/game": "Textures, meshes, render targets and descriptor heaps. Windows only sees opaque allocations; ""per-resource names need a graphics debugger (PIX on Windows, NVIDIA Nsight Graphics).",- "video": "Decoded/encoded video frames and codec surfaces (NVDEC/NVENC).",- "compute": "CUDA allocations are opaque to the OS. For more detail trace cudaMalloc with Nsight Systems.",+ "video": "Decoded/encoded video frames and codec surfaces (NVIDIA NVDEC/ENC)",+ "compute": "Compute allocations are opaque to the OS. Runtime-specific profilers can provide more detail (CUDA: cudaMalloc with Nsight Systems).","diffusion": "Model components are identified from launch arguments; GPU allocations cannot be split by component.","AI": "ML runtime allocations (tensors, workspaces). PyTorch: torch.cuda.memory_summary(); ""ONNX Runtime/DirectML pools are opaque from outside.",@@ -55,9 +55,9 @@ HELP_MD = """\| Layer | Source | Works for ||---|---|---|-| **Device** totals, load, bandwidth, PCIe, power | NVML (`nvml.dll`) | NVIDIA |-| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl) | every API incl. **CUDA** |-| **Which runtime** (CUDA, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything |+| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI | availability varies by GPU and driver |+| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl) | workloads visible to WDDM |+| **Which runtime** (CUDA, HIP, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything || **Which model file** | memory-mapped files + command line | llama.cpp, stable-diffusion.cpp, LM Studio, Ollama || **Model name & kind** (LLM, decision, embedding, speech) | GGUF header, Ollama/LM Studio/Hugging Face stores, local server APIs | GGUF family || **Weights / KV cache / compute** split | GGUF header math, llama.cpp/Ollama logs, local server APIs | GGUF/llama.cpp family |@@ -91,7 +91,8 @@ non-resident committed memory, not a measured eviction count or proof of pagefilnot necessarily current residency. A game's textures cannot be enumerated here.**GPU%** = busiest engine (like Task Manager). On GeForce/WDDM, CUDA kernels are usually reported-on the **3D** engine.+on the **3D** engine. AMD device GPU charts show graphics-engine activity, which is not+necessarily equivalent to NVML GPU utilization. Vulkan does not expose another process's HIP internals.Keys: `↑/↓` select · `s` sort · `c` chart · `a` show all · `p` pause · `?` this help · `q` quit@@ -241,9 +242,17 @@ class GpuTaskManager(App):def apply(self, snap: Snapshot) -> None:self.snap = snapnv = snap.nv+ amd = bool(snap.adapter and snap.adapter.vendor_id == 0x1002)+ if getattr(self, "_amd_chart_labels", None) != amd:+ self._amd_chart_labels = amd+ selector = self.query_one("#chart-mode", Select)+ selected = selector.value+ selector.set_options([("All charts", -1), *(("GFX clock" if amd and name == "SM clock" else name, index)+ for index, name in enumerate(self.CHARTS))])+ selector.value = selectedif nv:self.hist_util.append(nv.util_gpu)- self.hist_clock.append(nv.sm_clock)+ self.hist_clock.append(nv.gfx_clock if amd else nv.sm_clock)self.hist_bw.append(nv.util_mem)self.hist_temp.append(nv.temp)self.hist_power.append(nv.power_w)@@ -253,7 +262,8 @@ class GpuTaskManager(App):for history in (self.hist_util, self.hist_clock, self.hist_bw, self.hist_temp,self.hist_power, self.hist_enc, self.hist_dec):history.append(None)- self.hist_vram.append(100 * snap.vram_used / snap.vram_total if snap.vram_total else None)+ self.hist_vram.append(100 * snap.vram_used / snap.vram_total+ if snap.vram_total and snap.vram_used is not None else None)if not self._auto_selected:llm = next((r for r in sorted(snap.procs, key=lambda r: -r.gpu.dedicated) if r.llm), None)if llm:@@ -267,7 +277,8 @@ class GpuTaskManager(App):def render_device(self, s: Snapshot) -> Group:nv = s.nvname = s.adapter.name if s.adapter else (nv.name if nv else "GPU")- head = Text.assemble((name, "bold"), f" driver {s.driver} CUDA {s.cuda} ",+ info = (f" driver {s.driver}" if s.driver else "") + (f" CUDA {s.cuda}" if s.cuda else "")+ head = Text.assemble((name, "bold"), info + " ",(nv.driver_model if nv else "", "bold yellow"), f" {nv.pci_bus if nv else ''}")if self.paused:head.append(" [PAUSED]", style="bold red")@@ -277,15 +288,16 @@ class GpuTaskManager(App):for r in s.procs:by_kind[r.kind] = by_kind.get(r.kind, 0) + r.gpu.dedicatedattributed = sum(by_kind.values())- scale = min(1.0, s.vram_used / attributed) if attributed else 1.0+ scale = min(1.0, s.vram_used / attributed) if attributed and s.vram_used is not None else 1.0order = sorted(by_kind.items(), key=lambda kv: -kv[1])- parts = [(v * scale, KIND_STYLE.get(k, "white")) for k, v in order]- other = max(s.vram_used - attributed * scale, 0)+ parts = [(v * scale, KIND_STYLE.get(k, "white")) for k, v in order] if s.vram_used is not None else []+ other = max(s.vram_used - attributed * scale, 0) if s.vram_used is not None and s.attributed is not None else 0parts.append((other, "grey35"))width = max(min(self.size.width - 40, 90), 20)vram = Text("VRAM ")vram.append_text(stacked_bar(parts, s.vram_total, width))- vram.append(f" {gb(s.vram_used)} / {gb(s.vram_total)}", style="bold")+ used_label = gb(s.vram_used) if s.vram_used is not None else "N/A"+ vram.append(f" {used_label} / {gb(s.vram_total)}", style="bold")legend = Text(" ")for k, v in order[:7]:legend.append("■ ", style=KIND_STYLE.get(k, "white"))@@ -293,8 +305,9 @@ class GpuTaskManager(App):if other > 32 << 20:legend.append("■ ", style="grey35")legend.append(f"unattributed {gb(other)} ")- legend.append(f"free {gb(max(s.vram_total - s.vram_used, 0))}", style="grey62")- if s.shared_used > 64 << 20:+ if s.vram_used is not None:+ legend.append(f"free {gb(max(s.vram_total - s.vram_used, 0))}", style="grey62")+ if s.shared_used is not None and s.shared_used > 64 << 20:legend.append(f" shared(RAM) {gb(s.shared_used)}", style="yellow")if scale < 1.0:legend.append(" [bar normalized; labels are raw]", style="grey62")@@ -305,18 +318,22 @@ class GpuTaskManager(App):tx = nv.pcie_tx_kbs / 1024 if nv.pcie_tx_kbs is not None else Nonepstate = f"P{nv.pstate}" if nv.pstate is not None else "P N/A"m = Text()- m.append(f"GPU {format_metric(nv.util_gpu, unit='%')}", style=f"bold {self.CHART_COLORS['GPU']}")+ label = "AMD gfx" if nv.source.startswith("AMD ") else "GPU"+ m.append(f"{label} {format_metric(nv.util_gpu, unit='%')}", style=f"bold {self.CHART_COLORS['GPU']}")m.append(f" Mem activity {format_metric(nv.util_mem, unit='%')}", style=self.CHART_COLORS["Mem activity"])m.append(f" PCIe↓{format_metric(rx, '.0f'):>5} ↑{format_metric(tx, '.0f'):>4} MB/s")if nv.enc or nv.dec:m.append(f" enc {format_metric(nv.enc, unit='%')} dec {format_metric(nv.dec, unit='%')}")lines.append(m)+ clock_label = (f"gfx {format_metric(nv.gfx_clock)} MHz" if nv.source.startswith("AMD ") else+ f"SM {format_metric(nv.sm_clock)}/{format_metric(nv.sm_clock_max)} MHz")+ power_kind = f" {nv.power_kind}" if nv.power_kind else ""st = Text(f"{format_metric(nv.temp, unit='°C')} {format_metric(nv.power_w, '.0f')}/"- f"{format_metric(nv.power_limit_w, '.0f')} W {pstate} "- f"SM {format_metric(nv.sm_clock)}/{format_metric(nv.sm_clock_max)} MHz "+ f"{format_metric(nv.power_limit_w, '.0f')} W{power_kind} {pstate} "+ f"{clock_label} "f"mem {format_metric(nv.mem_clock)} MHz video {format_metric(nv.video_clock)} MHz "f"PCIe gen{format_metric(nv.pcie_gen)} x{format_metric(nv.pcie_width)} "- f"CC {nv.compute_cap or 'N/A'}", style="grey70")+ + (f"CC {nv.compute_cap or 'N/A'}" if nv.source == "NVML" else nv.source), style="grey70")if nv.throttle:st.append(" throttle: " + ",".join(nv.throttle), style="yellow")lines.append(st)@@ -361,7 +378,10 @@ class GpuTaskManager(App):for index in indices:values, scale, unit, style = self.chart_series(index, self.snap.nv)current = values[-1] if values else None- cell = Text(f"{self.CHARTS[index]:<12} ", style=style)+ name = self.CHARTS[index]+ if getattr(self, "_amd_chart_labels", False) and name == "SM clock":+ name = "GFX clock"+ cell = Text(f"{name:<12} ", style=style)cell.append(f"{format_metric(current, '.0f', unit):>8} ", style="bold")cell.append(spark(values, max(1, cell_width - 22), scale))cells.append(cell)@@ -372,7 +392,7 @@ class GpuTaskManager(App):_, clock_scale, _, _ = self.chart_series(self.CHARTS.index("SM clock"), self.snap.nv)self.query_one("#chart-note", Static).update(f"Recent samples · blank = no reading · power 0–{power_scale:.0f} W · "- f"SM clock 0–{clock_scale:.0f} MHz"+ f"{'GFX' if getattr(self, '_amd_chart_labels', False) else 'SM'} clock 0–{clock_scale:.0f} MHz")def on_resize(self) -> None:@@ -398,17 +418,20 @@ class GpuTaskManager(App):for r in rows:e = r.gpu.engines- spill = Text(gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",+ spill = Text("N/A" if "shared" not in r.gpu.available else gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")- paged = Text(gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")+ paged = Text("N/A" if not r.gpu.paged_available else gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")+ vram = gb(r.gpu.dedicated) if "dedicated" in r.gpu.available else "N/A"+ activity = pct(r.gpu.gpu_pct) if "engine" in r.gpu.available else Text("N/A", justify="right")+ engine = lambda value: pct(value) if "engine" in r.gpu.available else Text("N/A", justify="right")t.add_row(Text(str(r.pid), justify="right"),Text(r.name),Text(r.kind, style=KIND_STYLE.get(r.kind, "")),Text(" ".join(r.detail.runtimes[:3]), style="grey70"),- Text(gb(r.gpu.dedicated), justify="right", style="bold"), spill, paged, pct(r.gpu.gpu_pct),- pct(e.get("3d", 0) + e.get("compute", 0)), pct(e.get("copy", 0)),- pct(e.get("decode", 0) + e.get("encode", 0)),+ Text(vram, justify="right", style="bold"), spill, paged, activity,+ engine(e.get("3d", 0) + e.get("compute", 0)), engine(e.get("copy", 0)),+ engine(e.get("decode", 0) + e.get("encode", 0)),Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),key=str(r.pid),)@@ -429,15 +452,15 @@ class GpuTaskManager(App):panel.border_title = f"{r.name} ({r.pid}) - {r.kind}"out: list = []e = r.gpu.engines- hdr = Text.assemble(("VRAM ", "bold"), (gb(r.gpu.dedicated), "bold"), " spill(RAM) ",- (gb(r.gpu.shared), "bold red" if r.gpu.shared > 256 << 20 else ""),- " paged(est) ", gb(r.gpu.paged_out),- " engines ", " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle")+ hdr = Text.assemble(("VRAM ", "bold"), (gb(r.gpu.dedicated) if "dedicated" in r.gpu.available else "N/A", "bold"), " spill(RAM) ",+ (gb(r.gpu.shared) if "shared" in r.gpu.available else "N/A", "bold red" if r.gpu.shared > 256 << 20 else ""),+ " paged(est) ", gb(r.gpu.paged_out) if r.gpu.paged_available else "N/A",+ " engines ", (" ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle") if "engine" in r.gpu.available else "N/A")if r.detail.runtimes:hdr.append(" runtime ", style="grey62")hdr.append(" ".join(r.detail.runtimes), style="cyan")out.append(hdr)- if r.gpu.paged_out >= 64 << 20:+ if r.gpu.paged_available and r.gpu.paged_out >= 64 << 20:out.append(Text(f"{gb(r.gpu.paged_out)} of {gb(r.gpu.committed)} committed is not accounted for by local/non-local ""usage. Estimated non-residency, not proof of eviction or disk paging.",style="yellow"))diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.pyindex 59df309..bb2fa81 100644--- a/src/gputaskman/cli.py+++ b/src/gputaskman/cli.py@@ -13,6 +13,7 @@ def main() -> None:ap.add_argument("--once", action="store_true", help="print one snapshot and exit (no TUI)")ap.add_argument("--json", action="store_true", help="print one JSON snapshot and exit (implies --once)")ap.add_argument("--gpu", type=int, default=0, help="GPU index (default 0)")+ ap.add_argument("--list-gpus", action="store_true", help="list selectable GPU adapters and exit")ap.add_argument("-i", "--interval", type=float, default=1.0, help="refresh seconds (default 1)")args = ap.parse_args()if args.gpu < 0:@@ -23,6 +24,13 @@ def main() -> None:if sys.platform != "win32":sys.exit("gputaskman currently supports Windows (WDDM) only.")+ if args.list_gpus:+ from .collector import enumerate_adapters, ordered_adapters++ for i, adapter in enumerate(ordered_adapters(enumerate_adapters())):+ print(f"{i}: {adapter.name} ({adapter.vendor}, LUID 0x{adapter.luid & 0xffffffffffffffff:016x})")+ return+if args.gpu:from .collector import enumerate_adapters, select_adapterdiff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.pyindex 216b4ae..1e3829b 100644--- a/src/gputaskman/collector.py+++ b/src/gputaskman/collector.py@@ -8,12 +8,15 @@ from dataclasses import dataclass, fieldimport psutil+from .amdsmi import AmdSmi, _bdf+from .adl import Adlfrom .dxgi import VENDOR_NVIDIA, Adapter, enumerate_adaptersfrom .handy import HandyEvidence, inspect_handyfrom .llm import LlmView, ServerProber, build_view, gb, ollama_log_pathfrom .nvml import Nvml, NvDevicefrom .pdh import GpuCounters, ProcGpufrom .procinfo import AI_KINDS, ProcDetail, ProcInspector, classify+from .telemetry import DeviceTelemetryMiB = 1 << 20@@ -69,23 +72,23 @@ class Snapshot:taken: floatadapter: Adapter | Noneadapters: list[Adapter]- nv: NvDevice | None+ nv: DeviceTelemetry | None # legacy name; may contain telemetry from either providerdriver: strcuda: strvram_total: int- vram_used: int # WDDM adapter-wide dedicated usage (ground truth)- shared_used: int+ vram_used: int | None # WDDM adapter-wide dedicated usage+ shared_used: int | Noneprocs: list[ProcRow] = field(default_factory=list)@property- def attributed(self) -> int:+ def attributed(self) -> int | None:+ if any("dedicated" not in p.gpu.available for p in self.procs):+ return Nonereturn int(sum(p.gpu.dedicated for p in self.procs))def select_adapter(adapters: list[Adapter], gpu_index: int) -> Adapter | None:- hardware = [a for a in adapters if not a.software]- choices = [a for a in hardware if a.vendor_id == VENDOR_NVIDIA] or hardware- choices.sort(key=lambda a: -a.dedicated_vram)+ choices = ordered_adapters(adapters)if gpu_index < 0:raise ValueError("--gpu must be a non-negative index")if gpu_index >= len(choices):@@ -95,15 +98,26 @@ def select_adapter(adapters: list[Adapter], gpu_index: int) -> Adapter | None:return choices[gpu_index]-def select_nv_device(adapter: Adapter | None, devices: list[NvDevice], gpu_index: int) -> NvDevice | None:+def ordered_adapters(adapters: list[Adapter]) -> list[Adapter]:+ """List all hardware adapters; retain NVIDIA-first ordering for existing users."""+ return sorted((a for a in adapters if not a.software),+ key=lambda a: (a.vendor_id != VENDOR_NVIDIA, -a.dedicated_vram, a.luid))+++def select_nv_device(adapter: Adapter | None, devices: list[NvDevice], gpu_index: int,+ adapters: list[Adapter] | None = None) -> NvDevice | None:if adapter:if adapter.vendor_id != VENDOR_NVIDIA:return None- matches = [d for d in devices if d.name.casefold() == adapter.name.casefold()]+ if adapter.pci_address is None:+ return None+ if adapters is not None and sum(a.pci_address == adapter.pci_address for a in adapters) != 1:+ return None+ matches = [d for d in devices if _bdf(d.pci_bus) == adapter.pci_address and d.driver_model != "MCDM"]if len(matches) == 1:return matches[0]- fallback = devices[gpu_index] if gpu_index < len(devices) else None- return None if adapter and fallback and fallback.driver_model == "MCDM" else fallback+ return None+ return devices[gpu_index] if gpu_index < len(devices) else Noneclass Collector:@@ -115,6 +129,8 @@ class Collector:self.adapter = select_adapter(self.adapters, gpu_index)self.counters = GpuCounters()self.nvml = Nvml()+ self.amdsmi = AmdSmi(self.adapter, self.adapters)+ self.adl = Adl(self.adapter, self.adapters)self.inspector = ProcInspector()self.prober = ServerProber()self.scan_budget = scan_budget # None = scan every candidate now (one-shot output)@@ -133,7 +149,7 @@ class Collector:d.parents.append(p.pid)except (psutil.Error, OSError):pass- return [port for ppid in d.parents for port in self.inspector.ports(ppid)]+ return [port for ppid in d.parents for port in self.inspector.exclusive_ports(ppid)]def sample(self) -> Snapshot:per_luid, adapters = self.counters.sample()@@ -141,14 +157,17 @@ class Collector:procs = per_luid.get(luid, {})a = adapters.get(luid)nvs = self.nvml.sample() if self.nvml.ok else []- nv = select_nv_device(self.adapter, nvs, self.gpu_index)+ nv = select_nv_device(self.adapter, nvs, self.gpu_index, self.adapters)+ if self.adapter and self.adapter.vendor_id == 0x1002:+ nv = self.adl.sample() or self.amdsmi.sample()total = (nv.mem_total if nv else 0) or (self.adapter.dedicated_vram if self.adapter else 0)snap = Snapshot(taken=time.time(), adapter=self.adapter, adapters=self.adapters, nv=nv,- driver=self.nvml.driver, cuda=self.nvml.cuda_version, vram_total=total,- vram_used=int(a.dedicated) if a else (nv.mem_used if nv else 0),- shared_used=int(a.shared) if a else 0,+ driver=(self.amdsmi.driver if nv and nv.source == "AMD SMI" else self.nvml.driver if nv and nv.source == "NVML" else ""),+ cuda=self.nvml.cuda_version if nv and nv.source == "NVML" else "", vram_total=total,+ vram_used=int(a.dedicated) if a and a.dedicated is not None else None,+ shared_used=int(a.shared) if a and a.shared is not None else None,)live = [(pid, g) for pid, g in procs.items()@@ -173,9 +192,10 @@ class Collector:for row in snap.procs:if os.path.basename(row.detail.exe).lower() == "handy.exe":row.handy = inspect_handy(row.detail.create_time)- if row.detail.is_diffusion or row.kind not in AI_KINDS or not (row.detail.models or row.kind == "LLM"):+ if (row.detail.is_diffusion or row.kind not in AI_KINDS or not (row.detail.models or row.kind == "LLM")+ or not {"dedicated", "shared"} <= row.gpu.available):continue- own = self.prober.get(row.detail.ports)+ own = self.prober.get(self.inspector.exclusive_ports(row.pid))host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end processlog = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else Nonerow.llm = build_view(row.detail, row.gpu.dedicated, row.gpu.shared, own or host, log,@@ -187,3 +207,5 @@ class Collector:def close(self) -> None:self.counters.close()self.nvml.close()+ self.amdsmi.close()+ self.adl.close()diff --git a/src/gputaskman/dxgi.py b/src/gputaskman/dxgi.pyindex 9af8d2b..93bb675 100644--- a/src/gputaskman/dxgi.py+++ b/src/gputaskman/dxgi.py@@ -32,6 +32,7 @@ class _GUID(ctypes.Structure):_IID_IDXGIFactory1 = _GUID(0x770AAE78, 0xF26F, 0x4DBA, (ctypes.c_ubyte * 8)(0xA8, 0x29, 0x25, 0x3C, 0x83, 0xD1, 0xB3, 0x87))VENDOR_NVIDIA = 0x10DE+VENDOR_AMD = 0x1002VENDORS = {0x10DE: "NVIDIA", 0x1002: "AMD", 0x8086: "Intel", 0x1414: "Microsoft", 0x5143: "Qualcomm"}@@ -43,6 +44,7 @@ class Adapter:dedicated_vram: intshared_system: intsoftware: bool+ pci_address: tuple[int, int, int] | None = None@propertydef vendor(self) -> str:@@ -54,6 +56,38 @@ def _vcall(obj: ctypes.c_void_p, index: int, restype, *argtypes):return ctypes.WINFUNCTYPE(restype, ctypes.c_void_p, *argtypes)(vtbl[index])+class _OpenAdapter(ctypes.Structure):+ _fields_ = [("luid", _LUID), ("handle", wintypes.UINT)]+++class _AdapterAddress(ctypes.Structure):+ _fields_ = [("bus", wintypes.UINT), ("device", wintypes.UINT), ("function", wintypes.UINT)]+++class _QueryAdapter(ctypes.Structure):+ _fields_ = [("handle", wintypes.UINT), ("kind", wintypes.UINT),+ ("data", ctypes.c_void_p), ("size", wintypes.UINT)]+++def adapter_pci_address(luid: int) -> tuple[int, int, int] | None:+ """Obtain the WDDM adapter's bus/device/function; the PCI domain is not exposed."""+ try:+ gdi = ctypes.WinDLL("gdi32")+ opened = _OpenAdapter(_LUID(luid & 0xffffffff, ctypes.c_int32(luid >> 32).value), 0)+ if gdi.D3DKMTOpenAdapterFromLuid(ctypes.byref(opened)) != 0:+ return None+ try:+ address = _AdapterAddress()+ query = _QueryAdapter(opened.handle, 6, ctypes.addressof(address), ctypes.sizeof(address))+ if gdi.D3DKMTQueryAdapterInfo(ctypes.byref(query)) != 0:+ return None+ return address.bus, address.device, address.function+ finally:+ gdi.D3DKMTCloseAdapter(ctypes.byref(wintypes.UINT(opened.handle)))+ except (OSError, AttributeError):+ return None++def enumerate_adapters() -> list[Adapter]:try:dxgi = ctypes.WinDLL("dxgi")@@ -73,7 +107,7 @@ def enumerate_adapters() -> list[Adapter]:try:desc = _DESC1()if _vcall(adapter, 10, ctypes.c_long, ctypes.POINTER(_DESC1))(adapter, ctypes.byref(desc)) == 0:- luid = (desc.AdapterLuid.HighPart << 32) | desc.AdapterLuid.LowPart+ luid = ((desc.AdapterLuid.HighPart & 0xffffffff) << 32) | desc.AdapterLuid.LowPartout.append(Adapter(luid=luid,name=desc.Description,@@ -81,6 +115,7 @@ def enumerate_adapters() -> list[Adapter]:dedicated_vram=desc.DedicatedVideoMemory,shared_system=desc.SharedSystemMemory,software=bool(desc.Flags & 2),+ pci_address=adapter_pci_address(luid),))finally:_vcall(adapter, 2, wintypes.ULONG)(adapter) # Releasediff --git a/src/gputaskman/nvml.py b/src/gputaskman/nvml.pyindex 46e8c52..39d936d 100644--- a/src/gputaskman/nvml.py+++ b/src/gputaskman/nvml.py@@ -2,7 +2,7 @@from __future__ import annotations-from dataclasses import dataclass, field+from .telemetry import DeviceTelemetry as NvDevicetry:import pynvml as N@@ -10,36 +10,6 @@ except ImportError: # pragma: no coverN = None-@dataclass-class NvDevice:- index: int- name: str = "?"- pci_bus: str = ""- driver_model: str = "" # WDDM / TCC / MCDM- mem_total: int = 0- mem_used: int = 0- mem_reserved: int = 0- util_gpu: float | None = None # % of time a kernel was running- util_mem: float | None = None # % of time device memory was being read/written (bandwidth pressure)- enc: float | None = None- dec: float | None = None- temp: int | None = None- power_w: float | None = None- power_limit_w: float | None = None- fan: int | None = None- pstate: int | None = None- sm_clock: int | None = None- sm_clock_max: int | None = None- mem_clock: int | None = None- video_clock: int | None = None- pcie_rx_kbs: int | None = None # host -> GPU- pcie_tx_kbs: int | None = None # GPU -> host- pcie_gen: int | None = None- pcie_width: int | None = None- throttle: list[str] = field(default_factory=list)- compute_cap: str = ""--def format_metric(value: float | int | None, spec: str = "", unit: str = "") -> str:return f"{format(value, spec)}{unit}" if value is not None else "N/A"diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.pyindex 3f16e41..fafbc43 100644--- a/src/gputaskman/pdh.py+++ b/src/gputaskman/pdh.py@@ -63,10 +63,17 @@ class ProcGpu:shared: float = 0.0 # bytes resident in shared (system RAM) GPU memory -- "spill" ("Non Local Usage")committed: float = 0.0 # all GPU memory this process allocated, resident or not ("Total Committed")engines: dict[str, float] = field(default_factory=lambda: defaultdict(float)) # engtype -> util %+ available: set[str] = field(default_factory=lambda: {"dedicated", "shared", "committed", "engine"})++ @property+ def paged_available(self) -> bool:+ return {"dedicated", "shared", "committed"} <= self.available@propertydef paged_out(self) -> float:"""Commit minus local/non-local usage: a non-residency estimate, not an eviction trace."""+ if not self.paged_available:+ return 0.0 # internal sentinel; callers must check paged_available before displayreturn max(self.committed - self.dedicated - self.shared, 0.0)@property@@ -77,8 +84,8 @@ class ProcGpu:@dataclassclass AdapterGpu:- dedicated: float = 0.0- shared: float = 0.0+ dedicated: float | None = None+ shared: float | None = Noneclass GpuCounters:@@ -136,13 +143,14 @@ class GpuCounters:def sample(self) -> tuple[dict[int, dict[int, ProcGpu]], dict[int, AdapterGpu]]:"""Returns ({luid: {pid: ProcGpu}}, {luid: AdapterGpu})."""- _pdh.PdhCollectQueryData(self._query)+ if _pdh.PdhCollectQueryData(self._query) != 0:+ return {}, {}procs: dict[int, dict[int, ProcGpu]] = defaultdict(dict)def proc(luid: int, pid: int) -> ProcGpu:p = procs[luid].get(pid)if p is None:- p = procs[luid][pid] = ProcGpu()+ p = procs[luid][pid] = ProcGpu(available=set())return pfor attr in ("dedicated", "shared", "committed"):@@ -151,10 +159,13 @@ class GpuCounters:if m:p = proc(luid_key(m[2], m[3]), int(m[1]))setattr(p, attr, getattr(p, attr) + val)+ p.available.add(attr)for name, val in self._read("engine"):m = _ENGINE_RE.match(name)if m:- proc(luid_key(m[2], m[3]), int(m[1])).engines[_norm_engine(m[6])] += val+ p = proc(luid_key(m[2], m[3]), int(m[1]))+ p.engines[_norm_engine(m[6])] += val+ p.available.add("engine")adapters: dict[int, AdapterGpu] = defaultdict(AdapterGpu)for attr, key in (("dedicated", "a_dedicated"), ("shared", "a_shared")):@@ -162,7 +173,7 @@ class GpuCounters:m = _ADAPTER_RE.match(name)if m:a = adapters[luid_key(m[1], m[2])]- setattr(a, attr, getattr(a, attr) + val)+ setattr(a, attr, (getattr(a, attr) or 0.0) + val)return procs, adaptersdef close(self) -> None:diff --git a/src/gputaskman/procinfo.py b/src/gputaskman/procinfo.pyindex 6ef0601..bb308b5 100644--- a/src/gputaskman/procinfo.py+++ b/src/gputaskman/procinfo.py@@ -24,6 +24,7 @@ RUNTIMES: list[tuple[str, tuple[str, ...]]] = [("DirectML", ("directml.dll",)),("cuDNN", ("cudnn*.dll",)),("CUDA", ("nvcuda.dll", "nvcuda64.dll", "cudart64_*.dll", "cublas64_*.dll")),+ ("HIP", ("amdhip64.dll", "amdhip64_*.dll")),("Vulkan", ("vulkan-1.dll",)),("D3D12", ("d3d12core.dll",)),("OpenGL", ("opengl32.dll",)),@@ -267,6 +268,12 @@ class ProcInspector:self._refresh_ports()return self._ports.get(pid, [])+ def exclusive_ports(self, pid: int) -> list[int]:+ """Only probe ports with one known listener; localhost may route shared ports to another PID."""+ self._refresh_ports()+ others = {port for other, ports in self._ports.items() if other != pid for port in ports}+ return [port for port in self._ports.get(pid, []) if port not in others]+@staticmethoddef _new(pid: int) -> ProcDetail:d = ProcDetail(pid=pid)@@ -370,7 +377,7 @@ def classify(d: ProcDetail, engines: dict[str, float]) -> str:return "browser"if video or "NVENC" in d.runtimes:return "video"- if "CUDA" in d.runtimes:+ if "CUDA" in d.runtimes or "HIP" in d.runtimes:return "compute"if ("D3D12" in d.runtimes or "Vulkan" in d.runtimes) and engines.get("3d", 0) > 10:return "3D/game"diff --git a/src/gputaskman/report.py b/src/gputaskman/report.pyindex 289d871..7106f84 100644--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@ -17,9 +17,12 @@ def to_dict(s: Snapshot) -> dict:for p in s.procs:item = {"pid": p.pid, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,- "vram_bytes": int(p.gpu.dedicated), "shared_bytes": int(p.gpu.shared),- "committed_bytes": int(p.gpu.committed), "paged_out_bytes": int(p.gpu.paged_out),- "gpu_pct": round(p.gpu.gpu_pct, 1), "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v},+ "vram_bytes": int(p.gpu.dedicated) if "dedicated" in p.gpu.available else None,+ "shared_bytes": int(p.gpu.shared) if "shared" in p.gpu.available else None,+ "committed_bytes": int(p.gpu.committed) if "committed" in p.gpu.available else None,+ "paged_out_bytes": int(p.gpu.paged_out) if p.gpu.paged_available else None,+ "gpu_pct": round(p.gpu.gpu_pct, 1) if "engine" in p.gpu.available else None,+ "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if "engine" in p.gpu.available else None,"ports": p.detail.ports, "inside": p.inside,"model_files": [dataclasses.asdict(m) for m in p.detail.models],"installed_models": [dataclasses.asdict(m) for m in p.detail.candidates],@@ -45,6 +48,7 @@ def to_dict(s: Snapshot) -> dict:procs.append(item)return {"taken": s.taken, "gpu": s.adapter.name if s.adapter else None, "driver": s.driver, "cuda": s.cuda,+ "adapter": dataclasses.asdict(s.adapter) if s.adapter else None,"vram_total": s.vram_total, "vram_used": s.vram_used, "shared_used": s.shared_used,"attributed": s.attributed, "device": nv, "processes": procs,}@@ -54,26 +58,37 @@ def to_text(s: Snapshot, limit: int = 30) -> str:out = []nv = s.nvname = s.adapter.name if s.adapter else "GPU"- out.append(f"{name} driver {s.driver} CUDA {s.cuda} {nv.driver_model if nv else ''}")- out.append(f"VRAM {gb(s.vram_used)} / {gb(s.vram_total)} shared (in system RAM) {gb(s.shared_used)}")+ info = f" driver {s.driver}" if s.driver else ""+ if s.cuda:+ info += f" CUDA {s.cuda}"+ out.append(f"{name}{info} {nv.driver_model if nv else ''}")+ used = gb(s.vram_used) if s.vram_used is not None else "N/A"+ shared = gb(s.shared_used) if s.shared_used is not None else "N/A"+ out.append(f"VRAM {used} / {gb(s.vram_total)} shared (in system RAM) {shared}")if nv:rx = nv.pcie_rx_kbs / 1024 if nv.pcie_rx_kbs is not None else Nonetx = nv.pcie_tx_kbs / 1024 if nv.pcie_tx_kbs is not None else Nonepstate = f"P{nv.pstate}" if nv.pstate is not None else "P N/A"- out.append(f"GPU {format_metric(nv.util_gpu, unit='%')} mem-bandwidth {format_metric(nv.util_mem, unit='%')} "+ label = "AMD gfx" if nv.source.startswith("AMD ") else "GPU"+ clock = (f"gfx {format_metric(nv.gfx_clock)}" if nv.source.startswith("AMD ") else+ f"SM {format_metric(nv.sm_clock)}")+ power_kind = f" ({nv.power_kind})" if nv.power_kind else ""+ out.append(f"{label} {format_metric(nv.util_gpu, unit='%')} mem-bandwidth {format_metric(nv.util_mem, unit='%')} "f"{format_metric(nv.temp, unit='C')} {format_metric(nv.power_w, '.0f')}/"- f"{format_metric(nv.power_limit_w, '.0f')} W {pstate} "- f"SM {format_metric(nv.sm_clock)} MHz "+ f"{format_metric(nv.power_limit_w, '.0f')} W{power_kind} {pstate} "+ f"{clock} MHz "f"video {format_metric(nv.video_clock)} MHz "f"PCIe rx {format_metric(rx, '.0f')} MB/s "f"tx {format_metric(tx, '.0f')} MB/s")out.append("")out.append(f"{'PID':>7} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")for p in s.procs[:limit]:- spill = gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""- paged = gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""- out.append(f"{p.pid:>7} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {gb(p.gpu.dedicated):>9} {spill:>8} "- f"{paged:>8} {p.gpu.gpu_pct:>5.0f} {p.inside}")+ spill = "N/A" if "shared" not in p.gpu.available else gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""+ paged = "N/A" if not p.gpu.paged_available else gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""+ vram = gb(p.gpu.dedicated) if "dedicated" in p.gpu.available else "N/A"+ gpu_pct = f"{p.gpu.gpu_pct:.0f}" if "engine" in p.gpu.available else "N/A"+ out.append(f"{p.pid:>7} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {vram:>9} {spill:>8} "+ f"{paged:>8} {gpu_pct:>5} {p.inside}")for p in s.procs:if p.handy:out.append("")diff --git a/src/gputaskman/telemetry.py b/src/gputaskman/telemetry.pynew file mode 100644index 0000000..ee2ce21--- /dev/null+++ b/src/gputaskman/telemetry.py@@ -0,0 +1,38 @@+"""Optional device-wide telemetry shared by vendor providers.++Percentages and clocks are provider-specific; unsupported metrics are None.+"""+from dataclasses import dataclass, field+++@dataclass+class DeviceTelemetry:+ index: int+ name: str = "?"+ pci_bus: str = ""+ driver_model: str = ""+ mem_total: int | None = None+ mem_used: int | None = None+ mem_reserved: int | None = None+ util_gpu: float | None = None+ util_mem: float | None = None+ enc: float | None = None+ dec: float | None = None+ temp: float | None = None+ power_w: float | None = None+ power_kind: str = ""+ power_limit_w: float | None = None+ fan: int | None = None+ pstate: int | None = None+ sm_clock: int | None = None+ sm_clock_max: int | None = None+ gfx_clock: int | None = None+ mem_clock: int | None = None+ video_clock: int | None = None+ pcie_rx_kbs: int | None = None+ pcie_tx_kbs: int | None = None+ pcie_gen: int | None = None+ pcie_width: int | None = None+ throttle: list[str] = field(default_factory=list)+ compute_cap: str = ""+ source: str = "NVML"diff --git a/tests/test_core.py b/tests/test_core.pyindex 2896160..c66a628 100644--- a/tests/test_core.py+++ b/tests/test_core.py@@ -1,6 +1,7 @@from __future__ import annotationsimport contextlib+import ctypesimport ioimport jsonimport os@@ -17,6 +18,8 @@ from rich.table import Tablefrom textual.widgets import Footer, Select, Staticfrom gputaskman.app import GpuTaskManager, spark+from gputaskman.amdsmi import AmdSmi+from gputaskman.adl import Adl, _Metricsfrom gputaskman.cli import mainfrom gputaskman.collector import ProcRow, Snapshot, select_adapter, select_nv_devicefrom gputaskman.dxgi import Adapter@@ -26,7 +29,7 @@ from gputaskman.llm import (LlmView, ServerInfo, build_view, is_gpu_device, llamparse_flags, parse_load_log, probe_server)from gputaskman.nvml import NvDevice, Nvmlfrom gputaskman.pdh import GpuCounters, ProcGpu-from gputaskman.procinfo import (AI_KINDS, MappedModel, ProcDetail, ProcInspector, _is_model_file,+from gputaskman.procinfo import (AI_KINDS, MappedModel, ProcDetail, ProcInspector, _is_model_file, _match_runtimes,_models_from_cmdline, classify, installed_models, make_model)from gputaskman.report import to_dict, to_textfrom gputaskman.stores import store_name@@ -100,6 +103,94 @@ class CliTests(unittest.TestCase):class DeviceSelectionTests(unittest.TestCase):+ def test_adl_empty_or_failed_sample_allows_fallback(self):+ provider = Adl.__new__(Adl)+ provider.context, provider.index = ctypes.c_void_p(1), 4+ provider.dll = SimpleNamespace(ADL2_New_QueryPMLogData_Get=lambda *args: 0,+ ADL2_Main_Control_Destroy=lambda context: None)+ self.assertIsNone(provider.sample())+ provider.dll.ADL2_New_QueryPMLogData_Get = lambda *args: -1+ self.assertIsNone(provider.sample())+ provider.close()+ provider.close()+ self.assertIsNone(provider.sample())++ def test_shared_listen_port_is_not_used_for_server_attribution(self):+ inspector = ProcInspector()+ inspector._ports = {1: [9090, 9091], 2: [9090]}+ with patch.object(inspector, "_refresh_ports"):+ self.assertEqual(inspector.exclusive_ports(1), [9091])+ self.assertEqual(inspector.exclusive_ports(2), [])++ def test_adl_uses_only_supported_sensors_and_reports_board_power(self):+ def read(_context, _index, pointer):+ sensors = ctypes.cast(pointer, ctypes.POINTER(_Metrics)).contents.sensors+ for index, value in ((1, 2500), (8, 55), (19, 0), (73, 120)):+ sensors[index].supported, sensors[index].value = 1, value+ return 0++ provider = Adl.__new__(Adl)+ provider.context, provider.index = ctypes.c_void_p(1), 4+ provider.dll = SimpleNamespace(ADL2_New_QueryPMLogData_Get=read)+ sample = provider.sample()+ self.assertEqual((sample.gfx_clock, sample.temp, sample.util_gpu), (2500, 55, 0))+ self.assertEqual((sample.power_w, sample.power_kind), (120, "board"))+ self.assertIsNone(sample.mem_clock)+ self.assertIsNone(sample.util_mem)++ def test_amdsmi_matches_pci_and_handles_partial_metrics(self):+ amd = Adapter(1, "AMD", 0x1002, 16 * GiB, 0, False, (10, 0, 0))+ closed = []+ api = SimpleNamespace(+ amdsmi_init=lambda: None, amdsmi_shut_down=lambda: closed.append(True),+ amdsmi_get_processor_handles=lambda: ["other", "target"],+ amdsmi_get_gpu_device_bdf=lambda h: "0000:0a:00.0" if h == "target" else "0000:0b:00.0",+ amdsmi_get_gpu_driver_info=lambda h: {"driver_version": "test"},+ amdsmi_get_gpu_activity=lambda h: {"gfx_activity": 0},+ amdsmi_get_power_info=lambda h: {"socket_power": 42, "power_limit": "N/A"},+ AmdSmiTemperatureType=SimpleNamespace(EDGE=1),+ AmdSmiTemperatureMetric=SimpleNamespace(CURRENT=2),+ amdsmi_get_temp_metric=lambda *args: 47500,+ )+ provider = AmdSmi(amd, [amd], api)+ sample = provider.sample()+ self.assertEqual((sample.source, sample.util_gpu, sample.temp, sample.power_w), ("AMD SMI", 0, 47.5, 42))+ self.assertIsNone(sample.power_limit_w)+ self.assertIsNone(sample.util_mem)+ provider.close()+ provider.close()+ self.assertEqual(closed, [True])+ self.assertIsNone(provider.sample())++ def test_amdsmi_rejects_ambiguous_and_nonzero_domain(self):+ amd = Adapter(1, "AMD", 0x1002, 16 * GiB, 0, False, (10, 0, 0))+ other = Adapter(2, "AMD", 0x1002, 16 * GiB, 0, False, (10, 0, 0))+ api = SimpleNamespace(amdsmi_init=lambda: None, amdsmi_shut_down=lambda: None,+ amdsmi_get_processor_handles=lambda: [1],+ amdsmi_get_gpu_device_bdf=lambda h: "0001:0a:00.0")+ self.assertIsNone(AmdSmi(amd, [amd, other], api).sample())+ provider = AmdSmi(amd, [amd], api)+ self.assertIsNone(provider.sample())+ provider.close()++ def test_mixed_vendor_adapters_are_selectable(self):+ amd = Adapter(2, "AMD GPU", 0x1002, 24 * GiB, 0, False)+ nvidia = Adapter(1, "NVIDIA GPU", 0x10DE, 8 * GiB, 0, False)+ self.assertIs(select_adapter([amd, nvidia], 0), nvidia)+ self.assertIs(select_adapter([amd, nvidia], 1), amd)++ def test_amd_snapshot_does_not_show_cuda_header(self):+ s = snapshot()+ s.adapter = Adapter(2, "AMD GPU", 0x1002, 24 * GiB, 0, False)+ self.assertNotIn("CUDA", to_text(s).splitlines()[0])+ self.assertEqual(to_dict(s)["adapter"]["vendor_id"], 0x1002)++ def test_hip_runtime_is_compute_evidence(self):+ self.assertEqual(classify(ProcDetail(1, "worker.exe", runtimes=["HIP"]), {}), "compute")+ for dll in ("amdhip64.dll", "amdhip64_6.dll"):+ self.assertIn("HIP", _match_runtimes({dll}))+ self.assertNotIn("HIP", _match_runtimes({"amd_comgr_2.dll"}))+def test_adapter_indices_do_not_clamp_to_another_gpu(self):gpu = Adapter(1, "NVIDIA GPU", 0x10DE, 16 * GiB, 0, False)software = Adapter(2, "Software", 0x1414, 0, 0, True)@@ -110,18 +201,64 @@ class DeviceSelectionTests(unittest.TestCase):with self.assertRaisesRegex(ValueError, "out of range"):select_adapter([], 1)- def test_nvml_matches_unique_adapter_name_before_index(self):- gpu = Adapter(1, "NVIDIA RTX Spark", 0x10DE, 64 * GiB, 0, False)+ def test_nvml_matches_pci_address_not_name_or_index(self):+ gpu = Adapter(1, "NVIDIA RTX Spark", 0x10DE, 64 * GiB, 0, False, (10, 0, 0))npu = NvDevice(index=0, name="NVIDIA NPU", driver_model="MCDM")- nv_gpu = NvDevice(index=1, name="NVIDIA RTX SPARK", driver_model="WDDM")+ nv_gpu = NvDevice(index=1, name="NVIDIA RTX SPARK", pci_bus="00000000:0A:00.0", driver_model="WDDM")self.assertIs(select_nv_device(gpu, [npu, nv_gpu], 0), nv_gpu)self.assertIs(select_nv_device(None, [npu, nv_gpu], 0), npu)self.assertIsNone(select_nv_device(gpu, [npu], 0))+ self.assertIsNone(select_nv_device(gpu, [nv_gpu, nv_gpu], 0))intel = Adapter(2, "Intel GPU", 0x8086, 8 * GiB, 0, False)self.assertIsNone(select_nv_device(intel, [nv_gpu], 0))class MemoryTests(unittest.TestCase):+ def test_failed_collection_does_not_reuse_previous_counters(self):+ counters = GpuCounters.__new__(GpuCounters)+ counters._query = None+ with patch("gputaskman.pdh._pdh", SimpleNamespace(PdhCollectQueryData=lambda query: 1)), \+ patch.object(counters, "_read") as read:+ self.assertEqual(counters.sample(), ({}, {}))+ read.assert_not_called()++ def test_counter_samples_track_each_process_field(self):+ counters = GpuCounters.__new__(GpuCounters)+ counters._query = None+ data = {"dedicated": [("pid_42_luid_0x0_0x1_phys_0", 0.0)],+ "engine": [("pid_42_luid_0x0_0x1_phys_0_eng_0_engtype_Compute", 10.0)]}+ with patch.object(counters, "_read", side_effect=lambda key: data.get(key, [])), \+ patch("gputaskman.pdh._pdh", SimpleNamespace(PdhCollectQueryData=lambda query: 0)):+ processes, _ = counters.sample()+ gpu = processes[1][42]+ self.assertEqual(gpu.available, {"dedicated", "engine"})+ self.assertEqual(gpu.dedicated, 0)+ self.assertEqual(gpu.gpu_pct, 10)+ self.assertFalse(gpu.paged_available)++ def test_missing_process_counters_are_not_reported_as_zero(self):+ s = snapshot()+ s.procs[0].gpu.available = {"dedicated"}+ row = to_dict(s)["processes"][0]+ self.assertEqual(row["vram_bytes"], 8 * GiB)+ for name in ("shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):+ self.assertIsNone(row[name])+ self.assertIn("N/A", to_text(s))+ s.procs[0].gpu.available.clear()+ self.assertIsNone(to_dict(s)["attributed"])++ def test_missing_adapter_memory_is_not_reported_as_zero(self):+ s = snapshot()+ s.vram_used = None+ s.shared_used = None+ self.assertIsNone(to_dict(s)["vram_used"])+ self.assertIsNone(to_dict(s)["shared_used"])+ self.assertIn("VRAM N/A", to_text(s))+ self.assertIn("shared (in system RAM) N/A", to_text(s))+ output = io.StringIO()+ Console(file=output, width=120).print(GpuTaskManager().render_device(s))+ self.assertIn("N/A", output.getvalue())+def test_local_counters_are_preferred(self):self.assertTrue(GpuCounters._COUNTERS["dedicated"][0].endswith(r"\Local Usage"))self.assertTrue(GpuCounters._COUNTERS["shared"][0].endswith(r"\Non Local Usage"))diff --git a/tools/build_portable.py b/tools/build_portable.pyindex fd70cce..8ef8226 100644--- a/tools/build_portable.py+++ b/tools/build_portable.py@@ -52,7 +52,7 @@ def check_executable(path: Path, arch: str) -> None:def create_archive(bundle: Path, output: Path, version: str, arch: str) -> Path:validate_version(version)check_executable(bundle / "gputaskman.exe", arch)- forbidden = {"nvml.dll", "nvcuda.dll", "dxgi.dll", "pdh.dll"}+ forbidden = {"nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll"}# Editable-install origins contain build-machine paths and are not needed at runtime.files = sorted(path for path in bundle.rglob("*")@@ -107,7 +107,7 @@ def add_notices(bundle: Path) -> None:"PyInstaller bootloader: see licenses\\pyinstaller (including its exception).\n""NVIDIA Python bindings: see licenses\\nvidia-ml-py.\n""Runtime package metadata and license texts are included in _internal\\*.dist-info.\n"- "NVIDIA and Windows DLLs are provided by the installed driver and operating system.\n",+ "GPU driver and Windows DLLs are provided by the installed driver and operating system.\n",encoding="utf-8",)@@ -139,7 +139,7 @@ def build(output: Path, arch: str, tag: str | None = None) -> Path:" .\\gputaskman.exe --once\n"" .\\gputaskman.exe --json\n\n""Press q to quit the interactive display.\n"- "Requires Windows and an NVIDIA GPU with a current WDDM driver.\n"+ "Requires Windows and an NVIDIA or AMD GPU with a current WDDM driver.\n""No Python, uv, CUDA Toolkit, models, or administrator session is required.\n""This build is unsigned. No drivers or inference servers are bundled.\n""Snapshots can contain local paths and model names; redact before sharing.\n\n"diff --git a/tools/portable.spec b/tools/portable.specindex 2a72a0c..68dd7a2 100644--- a/tools/portable.spec+++ b/tools/portable.spec@@ -9,8 +9,8 @@ analysis = Analysis(datas=collect_data_files("textual") + copy_metadata("gputaskman", recursive=True),)-# These belong to Windows or the installed NVIDIA driver, not the portable app.-driver_dlls = {"nvml.dll", "nvcuda.dll", "dxgi.dll", "pdh.dll"}+# These belong to Windows or the installed GPU driver, not the portable app.+driver_dlls = {"nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll"}analysis.binaries = [entry for entry in analysis.binariesif Path(entry[0]).name.lower() not in driver_dlls