A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
Fix Intel (i915/xe) fdinfo handling found in review
Scott Hanselman97bbce83fcc0
diff --git a/README.md b/README.mdindex 1ecf9f9..eb7e55f 100644--- a/README.md+++ b/README.md@@ -2,7 +2,7 @@**An `htop` for what's inside your GPU, not just how full it is.**-Windows + NVIDIA/AMD, with Linux and WSL support (NVIDIA everywhere, AMD on native Linux; see+Windows + NVIDIA/AMD, with Linux and WSL support (NVIDIA everywhere, AMD and Intel on native Linux; see[Linux and WSL](#linux-and-wsl)) and macOS on Apple Silicon (see [macOS](#macos-apple-silicon)). A terminal UIthat connects per-process GPU memory and engine activity with loaded runtimes, model files, andLLM buffer breakdowns.@@ -118,7 +118,7 @@ skips the consistency check, so do not use it after changing dependencies.Tagged releases include portable Linux builds: `gputaskman-VERSION-linux-x64.tar.gz` and`gputaskman-VERSION-linux-arm64.tar.gz`. They need glibc 2.35 or newer (Ubuntu 22.04, Debian 12,-or newer) and an NVIDIA driver (or, on native Linux, an AMD GPU on the `amdgpu` driver), and work+or newer) and an NVIDIA driver (or, on native Linux, an AMD GPU on the `amdgpu` driver or an Intel GPU on `i915`), and worknatively, in containers, and in WSL:```bash@@ -199,7 +199,8 @@ up without double counting. Otherwise the process shows VRAM and SPILL as `n/a`resident minus shared is not private memory and is never computed.- **Primary node** (`/dev/dri/card*`, typically a compositor): root-owned logind keeps a copy of thatfd, which a normal user can't see, so a single holder can't be proven.-- **Legacy**: older kernels whose amdgpu fdinfo lacks `drm-shared-vram`/`drm-shared-gtt`.+- **Legacy**: older kernels whose amdgpu fdinfo lacks `drm-shared-vram`/`drm-shared-gtt`, and Intel integrated GPUs+ and `xe` (below).- **Incomplete**: some of the process's fds or fdinfo could not be read (GPU% is `n/a` too).- **Partial scan**: the TUI's 50 ms listing budget ran out this refresh, so a process it skipped mighthold the same client (GPU% is `n/a` too). Expected to be rare on a desktop (~9 ms for ~8,000 fds,@@ -232,6 +233,27 @@ That check predates the stricter attribution above (no subtraction of shared bytsingle-holder and complete-scan requirements; per-client timestamps; PID-reuse checks), which iscovered by unit tests only and **not yet validated on AMD hardware**.+**Intel GPUs on native Linux** (the `i915` driver) use the same DRM fdinfo path, with these differences:++| | Source |+|---|---|+| Device temperature, power, GT clock | `hwmon` and `gt/gt0` under `/sys/class/drm/card*`. Power is the rate of the cumulative energy counter between two refreshes, so it is `n/a` on the first |+| Device VRAM total | Size of the local-memory BAR (`resource2`). A card whose window is under 1 GiB (an integrated GPU, or a discrete one without resizable BAR) is treated as integrated and has no total |+| Device VRAM used | The sum of its clients' resident local memory: i915 has no device-wide counter, so buffers shared between processes can be counted twice and processes gputaskman can't inspect are missing (the UI and `--once` say so). `n/a` when any client's local memory is unknown |+| Per-process VRAM and SPILL | `drm-resident-local0` and `drm-resident-system0`, plus their `stolen-` regions, under the same private-client rules as AMD. Both `drm-shared-local0` and `drm-shared-system0` must be present to charge a process, so an integrated GPU (no `local0`) shows `n/a` (`legacy`) |+| Per-process engine load (3d, compute, copy, video) | `drm-engine-render`, `-compute`, `-copy`, `-video`, `-video-enhance` busy time, sampled twice. `n/a` where the driver reports none (before Gen 8). A class with several identical engines (`drm-engine-capacity-*`, such as video on an Arc) is not divided by its capacity, so one busy engine reads 100% |++Device GPU load, memory bandwidth, and the PCIe link are `n/a`: i915 has no device-wide busy counter, and its+sysfs link fields can report the power-on default instead of the live link. The GT clock is read but not shown yet.+`xe` cards are detected, but its fdinfo uses other keys (`drm-cycles-*`, `drm-resident-vram0`/`-gtt`), so its+per-process rows show `n/a` until they are decoded on hardware.++Live check, reported by the contributor and not repeated by the maintainer: Linux 6.12, an Arc A770 (discrete, 16 GiB+BAR) and a UHD 730 (integrated) on `i915`. Both cards were found, named, and selectable with `--gpu`; the Arc showed+per-process VRAM and engine load, and the TUI rendered both. The fdinfo formats of both drivers, taken from the kernel+sources, are covered by unit tests. **Not validated on hardware:** `xe`, Arc cards without resizable BAR, and+multi-GPU systems other than that pair.+**WSL** is different, because Windows owns the GPU. Linux cannot see how much VRAM its ownprocesses use: `nvidia-smi` lists no processes, NVML's per-process samples carry *Windows* PIDs(never looked up in Linux's `/proc`), and Windows charges all Linux GPU memory to the VM worker@@ -303,7 +325,7 @@ Build both `gputaskman` and its `gtm` alias with `nix build path:.` (executablespins nixpkgs. Dependencies come from nixpkgs, independently of `uv.lock`. No pip installation,virtual environment, CUDA Toolkit, or ROCm installation is needed. GPU drivers remain supplied bythe host, never bundled with the package. NVIDIA's nixpkgs Python binding finds NVML through-`/run/opengl-driver/lib`; AMD uses kernel sysfs and DRM fdinfo interfaces directly. On WSL2 the+`/run/opengl-driver/lib`; AMD and Intel use kernel sysfs and DRM fdinfo interfaces directly. On WSL2 thehost driver lives in `/usr/lib/wsl/lib` instead, and the wrappers add that directory to`LD_LIBRARY_PATH` (it is ignored when absent, so NixOS is unaffected).@@ -681,7 +703,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:| GGUF header, model-store manifests | ms, cached | Once per file / every 60 s || Local API probes | ms, local only | llama-server `/v1/models` only when its model path changes || `psutil.open_files()` | 0.6-2.5 s per process | Never |-| AMD on Linux: DRM fdinfo scan (`readlink` every fd, read fdinfo of `/dev/dri` ones) | ~9 ms for ~8,000 fds (Linux desktop) | Every refresh, only when an amdgpu card is shown; listing fds is bounded to 50 ms in the TUI and resumes next refresh, known GPU fds are still read |+| AMD and Intel on Linux: DRM fdinfo scan (`readlink` every fd, read fdinfo of `/dev/dri` ones) | ~9 ms for ~8,000 fds (AMD, Linux desktop; Intel not measured) | Every refresh, only when an amdgpu or Intel card is shown; listing fds is bounded to 50 ms in the TUI and resumes next refresh, known GPU fds are still read || macOS: `ioreg` device stats + Metal clients | ~10 ms | Every refresh || macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above || macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) |@@ -820,7 +842,7 @@ downloads are bundled in this repository.|---|---|| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration || `linux.py` | Linux backend: NVML adapters and per-process usage, container PID matching, WSL detection, `/dev/dxg` holders |-| `drm.py` | AMD on native Linux: amdgpu cards and device readings from sysfs, per-process VRAM/GTT/engine time from DRM fdinfo |+| `drm.py` | AMD and Intel on native Linux: amdgpu and i915 cards and device readings from sysfs, per-process VRAM/GTT/engine time from DRM fdinfo || `wslhost.py` | WSL: interop check, Windows host counter stream, adapter pairing, context matching || `macos.py` | macOS backend: IOKit device stats and per-process GPU time, `footprint` GPU memory poller, libproc memory maps || `nvml.py` | NVIDIA device readings |@@ -838,7 +860,8 @@ downloads are bundled in this repository.- Model identity for non-GGUF runtimes (ONNX Runtime, PyTorch, TensorRT) beyond installed candidates.- Explicit provenance and availability for every counter and estimate.- Reliable multi-GPU matching and per-device model placement.-- Bare-metal Linux validation on NVIDIA hardware; Intel GPUs (`i915`/`xe`) through the same DRM fdinfo path as AMD.+- Bare-metal Linux validation on NVIDIA hardware; Intel `xe` per-process memory and load (`drm-cycles-*`), the Intel GT+ clock in the device panel, and Intel validation beyond the Arc A770 / UHD 730 pair.- macOS: a portable build, Ollama/llama.cpp/MLX validation, and compressed GPU memory (footprint's`swapped`) as the paged column.- WSL: per-process GPU load for Linux rows, and Ollama's systemd journal as a log source.diff --git a/src/gputaskman/app.py b/src/gputaskman/app.pyindex 8127771..34da39b 100644--- a/src/gputaskman/app.py+++ b/src/gputaskman/app.py@@ -22,7 +22,7 @@ from .llm import gbfrom .nvml import NvDevice, format_metricfrom .recovery import LiveCollectorsfrom .report import (DRM_RESIDENT_NOTE, METAL_NOTE, WSL_EMPTY, drm_note, has_engine_counters, has_spill_counters, has_wddm_counters,- inferred_note, vram_label, wsl_note)+ inferred_note, intel_memory_note, vram_label, wsl_note)KIND_STYLE = {"LLM": "bold magenta", "decision": "bold orchid", "decision?": "orchid",@@ -66,8 +66,8 @@ This display-only choice applies to all GPUs and resets to full when the app res| Layer | Source | Works for ||---|---|---|-| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI; amdgpu sysfs on Linux | availability varies by GPU and driver |-| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl); DRM fdinfo for amdgpu on Linux | workloads visible to WDDM or DRM |+| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI; amdgpu or Intel (i915) sysfs on Linux | availability varies by GPU and driver |+| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl); DRM fdinfo for amdgpu or Intel (i915) on Linux | workloads visible to WDDM or DRM || **Which runtime** (CUDA, HIP, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything || **Which model file** | memory-mapped files + command line | llama.cpp, stable-diffusion.cpp, LM Studio, Ollama || **Model name & kind** (LLM, decision, embedding, speech) | GGUF header, Ollama/LM Studio/Hugging Face stores, local server APIs | GGUF family |@@ -91,7 +91,11 @@ GTT (GPU memory in system RAM, shown as SPILL), and gfx/compute/video engine timfirst sample and for ROCm/HIP processes: their compute runs on `/dev/kfd` queues the kernel doesn't time perprocess). VRAM and SPILL are charged to a process only when its DRM clients are provably private: one holder,opened through a render node, and no shared buffers. Otherwise they show `n/a` and the bytes stay unattributed;-the inspector still shows the resident bytes it can reach (shared buffers included, so not added up). Only processes you can inspect are+the inspector still shows the resident bytes it can reach (shared buffers included, so not added up). **Intel GPUs+on Linux** (`i915`) take the same path: sysfs for the device, fdinfo for engine load and, on a discrete Arc, local-memory+VRAM. An integrated GPU shows engine load only (VRAM `n/a`), `xe` cards are detected but their per-process fdinfo isn't+decoded yet (rows show `n/a`), and with no device-wide memory counter the device's VRAM used is a sum over the processes+it can inspect: shared buffers can be counted twice and uninspectable processes are missing. Only processes you can inspect arelisted, and the header says how many it couldn't; run with `sudo` to see other users'. Under **WSL**, Windowscharges all Linux GPU memory to the VM (`vmwp.exe`), so VRAM and load are device-wide only; rowsare Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up share. When WSL@@ -442,6 +446,8 @@ class GpuTaskManager(App):legend.append("\n " + wsl_note(s), style="grey62")if s.backend == "metal":legend.append("\n " + METAL_NOTE, style="grey62")+ if intel_memory_note(s):+ legend.append("\n " + intel_memory_note(s), style="grey62")if s.counter_status:legend.append("\n " + s.counter_status, style="grey62")if s.process_status:diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.pyindex 8ef2cea..ae79e77 100644--- a/src/gputaskman/cli.py+++ b/src/gputaskman/cli.py@@ -46,7 +46,7 @@ def main() -> None:ap.error("--interval must be a finite number greater than zero")if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):- sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu DRM) and macOS (Apple Silicon) only.")+ sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu/Intel DRM) and macOS (Apple Silicon) only.")if args.list_gpus:from .collector import enumerate_adapters, ordered_adaptersdiff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.pyindex afd2f18..1bad53d 100644--- a/src/gputaskman/collector.py+++ b/src/gputaskman/collector.py@@ -115,8 +115,8 @@ class Snapshot:vram_used: int | None # WDDM adapter-wide dedicated usageshared_used: int | Noneprocs: list[ProcRow] = field(default_factory=list)- backend: str = "wddm" # wddm (Windows) | nvml (native Linux, NVIDIA) | drm (native Linux, AMD) | wsl (device-wide only) | metal (macOS)- process_status: str = "" # amdgpu only: what the DRM process scan couldn't cover; empty when it saw everything+ backend: str = "wddm" # wddm (Windows) | nvml (native Linux, NVIDIA) | drm (native Linux, AMD or Intel) | wsl (device-wide only) | metal (macOS)+ process_status: str = "" # drm backend only: what the DRM process scan couldn't cover; empty when it saw everythinghost_status: str = "" # WSL only: Windows host counters via interop -- ok | starting | off (...) | ...wsl_split: str = "" # WSL only: why Linux rows have no (inferred) VRAM, when they don't# False on an APU (amdgpu iGPU): its GTT is ordinary GPU memory, not spill from a full VRAM.diff --git a/src/gputaskman/drm.py b/src/gputaskman/drm.pyindex a1b2c8e..2649812 100644--- a/src/gputaskman/drm.py+++ b/src/gputaskman/drm.py@@ -144,9 +144,15 @@ def parse_fdinfo(text: str) -> dict | None:engines = None # a torn or malformed counter: this client's load is unknown, not 0breakif fields.get("drm-driver") in ("i915", "xe"):+ if not engines:+ # i915 (gen8+) prints a drm-engine-<class> line for each engine class the device has, even at 0 ns;+ # xe reports drm-cycles-* instead and gen7 i915 nothing. No such line means the load is unknown, not 0+ # (AGENTS.md rule 2).+ engines = None# Intel's logical regions are local0 (device memory; the local/stolen pair covers memory that# shrunk out of local into system) and system0 (GPU-placed system memory, the GTT equivalent).- # There is no separate CPU-only region: system0 feeds the shared/spill slot instead.+ # There is no separate CPU-only region: system0 feeds the shared/spill slot instead. These are+ # i915's names; xe calls them vram0/gtt/system, so its memory stays unknown until validated.return {"driver": fields["drm-driver"],"client_id": client_id,@@ -467,25 +473,18 @@ def _vcn_major(device: str) -> int:def _engine_key(name: str, unified_video: bool = False, intel: bool = False) -> str:if intel:lowered = name.lower()- # i915/xe engine class-instance names: rcs0 render/3D, ccs0 compute,- # bcs0 blitter (copies), vcs0 video decode, vecs0 video encode.- if lowered.startswith("rcs"):+ # i915's fdinfo names whole engine classes: render, copy, video, video-enhance, compute. The command-streamer+ # names (rcs, ccs, bcs, vcs, vecs, optionally with an instance number) are how xe and the engine tables spell+ # the same classes. Video-enhance (VECS) is post-processing, not encode, so it counts as video.+ if lowered == "render" or lowered.startswith("rcs"):return "3d"- if lowered.startswith("ccs"):+ if lowered == "compute" or lowered.startswith("ccs"):return "compute"- if lowered.startswith("bcs"):+ if lowered == "copy" or lowered.startswith("bcs"):return "copy"- if lowered.startswith("vcs"):- return "video"- if lowered.startswith("vecs"):- return "encode"- if lowered == "compute":- return "compute"- if lowered == "copy":- return "copy"- if lowered.startswith("video"):- return "video" # video / video-enhance: decode and encode can't be told apart- return "other" # any other class instance: busy time still counts toward GPU%+ if lowered.startswith(("video", "vcs", "vecs")):+ return "video" # decode and encode can't be told apart+ return "other" # any other class: busy time still counts toward GPU%lowered = name.lower()if unified_video and ("enc" in lowered or "dec" in lowered):return "video" # one ring for both: decode and encode can't be told apart@@ -569,8 +568,8 @@ class DrmCounters:def _stamp_intel_memory(self, clients: dict[tuple, list]) -> None:"""Intel cards expose no device-wide memory sysfs: used local memory is the sum of the card'sclients' resident local bytes, counted once per client even when several processes share it- (buffers shared between clients can double-count; the status note says so). With no readable- client the sum is None -- unknown, not 0."""+ (buffers shared between clients can double-count, and clients that can't be inspected are missing;+ report.INTEL_MEMORY_NOTE says so). With no readable client the sum is None -- unknown, not 0."""for card in self.cards:if not card.intel:continuediff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.pyindex adf54fe..621176f 100644--- a/src/gputaskman/pdh.py+++ b/src/gputaskman/pdh.py@@ -84,11 +84,11 @@ class ProcGpu:# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.inferred: float | None = Noneinferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)- # amdgpu (DRM fdinfo): resident bytes in every buffer this process can reach, shared ones included. A reference+ # DRM fdinfo (amdgpu, Intel): resident bytes in every buffer this process can reach, shared ones included. A reference# for display only; the same buffer can appear under several processes, so these are never summed. None: unknown.resident_vram: float | None = Noneresident_gtt: float | None = None- drm_reason: str = "" # amdgpu: why dedicated/shared are n/a (drm.REASON_*); empty when they are measured+ drm_reason: str = "" # DRM fdinfo: why dedicated/shared are n/a (drm.REASON_*); empty when they are measuredkfd: bool = False # amdgpu: holds /dev/kfd (ROCm/HIP), whose compute isn't timed per process@propertydiff --git a/src/gputaskman/report.py b/src/gputaskman/report.pyindex f0f0062..4820cd8 100644--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@ -35,7 +35,8 @@ def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:def has_spill_counters(s: Snapshot, row: ProcRow) -> bool:- """Per-process GPU memory in system RAM: WDDM's non-local usage, or amdgpu's GTT on Linux."""+ """Per-process GPU memory in system RAM: WDDM's non-local usage, or a DRM client's GTT (amdgpu) or+ system0 (Intel) bytes on Linux."""return (has_wddm_counters(s, row) or (s.backend == "drm" and not row.host)) and "shared" in row.gpu.available@@ -59,8 +60,9 @@ ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newerDRM_NOTES = {- "legacy": ("VRAM n/a: this kernel's amdgpu fdinfo has incomplete or unparseable drm-shared-* stats, so resident "- "memory may include buffers imported from other processes and would be counted once per importer."),+ "legacy": ("VRAM n/a: this driver's fdinfo has missing or unparseable drm-shared-* stats (older amdgpu kernels, "+ "Intel integrated GPUs, xe), so resident memory may include buffers imported from other processes "+ "and would be counted once per importer."),"shared_client": ("VRAM and GPU% n/a: this process holds a DRM client other processes hold too (fd passing, ""fork). fdinfo counts the client once and doesn't say which of them uses it."),"shared_buffers": ("VRAM n/a: this process's DRM client shares buffers with other clients (dma-buf, e.g. "@@ -87,6 +89,15 @@ def inferred_note(g) -> str:METAL_NOTE = ("macOS: unified memory, so GPU memory is part of system RAM. Rows show each process's graphics ""footprint; other users' processes (WindowServer, daemons) show GPU% only, since reading their ""memory needs root.")+INTEL_MEMORY_NOTE = ("Intel VRAM used is the sum of local memory resident in the processes gputaskman can inspect "+ "(the driver has no device-wide counter): buffers shared between processes can be counted "+ "twice, and processes it can't inspect are missing.")+++def intel_memory_note(s: Snapshot) -> str:+ """Intel's device VRAM used is a sum over visible DRM clients, not a device counter."""+ return INTEL_MEMORY_NOTE if (s.backend == "drm" and s.nv and s.nv.source == "Intel sysfs"+ and s.vram_used is not None) else ""def to_dict(s: Snapshot) -> dict:@@ -169,6 +180,8 @@ def to_text(s: Snapshot, limit: int = 30) -> str:out.append(memory)if s.backend == "wsl":out.append(wsl_note(s))+ if intel_memory_note(s):+ out.append(intel_memory_note(s))if s.counter_status:out.append(s.counter_status)if nv and s.backend != "metal":diff --git a/tests/test_drm.py b/tests/test_drm.pyindex 717b4ff..2abe286 100644--- a/tests/test_drm.py+++ b/tests/test_drm.py@@ -9,6 +9,7 @@ from pathlib import Pathfrom gputaskman import drmGiB = 1 << 30+MiB = 1 << 20FDINFO = """pos:\t0flags:\t02100002@@ -42,6 +43,115 @@ drm-engine-enc:\t1200 nsdrm-engine-capacity-gfx:\t1"""+# The Intel samples are laid out the way the kernels print them (i915_drm_client.c; the xe_drm_client.c documentation+# sample; sizes in the units drm_print_memory_stats uses). They are built from that code, not captured from hardware.+I915_FDINFO = """pos:\t0+flags:\t02100002+mnt_id:\t28+ino:\t415+drm-driver:\ti915+drm-pdev:\t0000:03:00.0+drm-client-id:\t42+drm-total-system0:\t8 MiB+drm-shared-system0:\t0+drm-active-system0:\t0+drm-resident-system0:\t6 MiB+drm-purgeable-system0:\t0+drm-total-local0:\t2048 MiB+drm-shared-local0:\t0+drm-active-local0:\t16 MiB+drm-resident-local0:\t2048 MiB+drm-purgeable-local0:\t0+drm-total-stolen-local0:\t0+drm-shared-stolen-local0:\t0+drm-active-stolen-local0:\t0+drm-resident-stolen-local0:\t64 KiB+drm-purgeable-stolen-local0:\t0+drm-engine-render:\t5308400172 ns+drm-engine-copy:\t1200 ns+drm-engine-video:\t0 ns+drm-engine-capacity-video:\t2+drm-engine-video-enhance:\t0 ns+drm-engine-capacity-video-enhance:\t2+drm-engine-compute:\t750000000 ns+drm-engine-capacity-compute:\t4+"""++# An integrated GPU has no local0 region, only system0 and its stolen carve-out.+I915_IGPU_FDINFO = """pos:\t0+flags:\t02100002+mnt_id:\t28+ino:\t415+drm-driver:\ti915+drm-pdev:\t0000:00:02.0+drm-client-id:\t7+drm-total-system0:\t64 MiB+drm-shared-system0:\t0+drm-active-system0:\t0+drm-resident-system0:\t60 MiB+drm-purgeable-system0:\t0+drm-total-stolen-system0:\t0+drm-shared-stolen-system0:\t0+drm-active-stolen-system0:\t0+drm-resident-stolen-system0:\t0+drm-purgeable-stolen-system0:\t0+drm-engine-render:\t1200000000 ns+drm-engine-copy:\t0 ns+drm-engine-video:\t300000000 ns+drm-engine-video-enhance:\t0 ns+"""++# i915 before Gen8 prints the memory stats but no drm-engine lines at all.+I915_GEN7_FDINFO = """drm-driver:\ti915+drm-pdev:\t0000:00:02.0+drm-client-id:\t5+drm-total-system0:\t4 MiB+drm-shared-system0:\t0+drm-active-system0:\t0+drm-resident-system0:\t4 MiB+drm-purgeable-system0:\t0+"""++# xe: memory regions are named system/gtt/vram0/stolen and the load is drm-cycles-*, not drm-engine-<class> ns.+XE_FDINFO = """pos:\t0+flags:\t0100002+mnt_id:\t26+ino:\t685+drm-driver:\txe+drm-client-id:\t3+drm-pdev:\t0000:03:00.0+drm-total-system:\t0+drm-shared-system:\t0+drm-active-system:\t0+drm-resident-system:\t0+drm-purgeable-system:\t0+drm-total-gtt:\t192 KiB+drm-shared-gtt:\t0+drm-active-gtt:\t0+drm-resident-gtt:\t192 KiB+drm-total-vram0:\t23992 KiB+drm-shared-vram0:\t16 MiB+drm-active-vram0:\t0+drm-resident-vram0:\t23992 KiB+drm-total-stolen:\t0+drm-shared-stolen:\t0+drm-active-stolen:\t0+drm-resident-stolen:\t0+drm-cycles-rcs:\t28257900+drm-total-cycles-rcs:\t7655183225+drm-cycles-bcs:\t0+drm-total-cycles-bcs:\t7655183225+drm-cycles-vcs:\t0+drm-total-cycles-vcs:\t7655183225+drm-engine-capacity-vcs:\t2+drm-cycles-vecs:\t0+drm-total-cycles-vecs:\t7655183225+drm-engine-capacity-vecs:\t2+drm-cycles-ccs:\t0+drm-total-cycles-ccs:\t7655183225+drm-engine-capacity-ccs:\t4+"""+def _make_card(root: Path, name, pdev, vendor, device, vram, driver="amdgpu", gtt=0, apu=False) -> Path:dev = root / "drm" / name / "device"@@ -96,6 +206,26 @@ def _fdinfo(client, pdev, gfx=0, vram=0, gtt=0, shared_vram=0, shared_gtt=0, sha)+def _i915_fdinfo(client, pdev, render=0, video=0, local_mib=0, system_mib=0, shared_local_mib=0, engines=True) -> str:+ """A discrete i915 client's fdinfo: memory by region (sizes in MiB) and, unless engines is False, the engine+ classes the way i915 prints them."""+ text = (+ "pos:\t0\n"+ "drm-driver:\ti915\n"+ f"drm-client-id:\t{client}\n"+ f"drm-pdev:\t{pdev}\n"+ f"drm-shared-system0:\t0\n"+ f"drm-resident-system0:\t{system_mib} MiB\n"+ f"drm-shared-local0:\t{shared_local_mib} MiB\n"+ f"drm-resident-local0:\t{local_mib} MiB\n"+ )+ if engines:+ text += (f"drm-engine-render:\t{render} ns\n"+ "drm-engine-copy:\t0 ns\n"+ f"drm-engine-video:\t{video} ns\n")+ return text++class Clock:"""A monotonic_ns stand-in the test advances by hand: every fdinfo read in one refresh sees the same time."""@@ -792,9 +922,19 @@ class EngineAndSelectionTests(unittest.TestCase):self.assertEqual(drm._engine_key("ccs0", intel=True), "compute")self.assertEqual(drm._engine_key("bcs0", intel=True), "copy")self.assertEqual(drm._engine_key("vcs0", intel=True), "video")- self.assertEqual(drm._engine_key("vecs0", intel=True), "encode")+ self.assertEqual(drm._engine_key("vecs0", intel=True), "video") # video-enhance is post-processing, not encodeself.assertEqual(drm._engine_key("rcs0", intel=False), "other") # Intel-only prefix on AMD+ def test_amd_selectable_next_to_nvidia_on_linux(self):+ from gputaskman import collector+ from gputaskman.dxgi import Adapter++ nvidia = Adapter(1, "NVIDIA GPU", 0x10DE, 8 * GiB, 0, False)+ amd = Adapter(2, "AMD Radeon 610M", 0x1002, 2 * GiB, 0, False)+ with patch.object(collector, "BACKEND", "nvml"):+ self.assertIs(collector.select_adapter([amd, nvidia], 0), nvidia)+ self.assertIs(collector.select_adapter([amd, nvidia], 1), amd)+def test_intel_selectable_next_to_nvidia_on_linux(self):from gputaskman import collectorfrom gputaskman.dxgi import Adapter@@ -827,9 +967,18 @@ class IntelCardTests(unittest.TestCase):self.assertEqual(drm._engine_key("ccs0", intel=True), "compute")self.assertEqual(drm._engine_key("bcs0", intel=True), "copy")self.assertEqual(drm._engine_key("vcs0", intel=True), "video")- self.assertEqual(drm._engine_key("vecs0", intel=True), "encode")+ self.assertEqual(drm._engine_key("vecs0", intel=True), "video")self.assertEqual(drm._engine_key("rcs0", intel=False), "other")+ def test_intel_engine_classes_as_i915_prints_them(self):+ # i915 names whole classes (i915_drm_client.c uabi_class_names), not command streamers.+ for name, expected in (("render", "3d"), ("copy", "copy"), ("video", "video"),+ ("video-enhance", "video"), ("compute", "compute")):+ with self.subTest(name=name):+ self.assertEqual(drm._engine_key(name, intel=True), expected)+ self.assertEqual(drm._engine_key("something-new", intel=True), "other")+ self.assertEqual(drm._engine_key("render", intel=False), "other") # not an AMD engine name+def test_intel_card_is_recognized_and_amd_unchanged(self):counters = self.make([self.INTEL])self.assertTrue(counters.ok)@@ -880,5 +1029,160 @@ class IntelCardTests(unittest.TestCase):self.assertIsNone(out[self.IGPU.key].shared)+class IntelFdinfoTests(unittest.TestCase):+ """fdinfo laid out the way the Intel kernels print it (the fixtures at the top of this file)."""++ def test_i915_discrete_client(self):+ info = drm.parse_fdinfo(I915_FDINFO)+ self.assertEqual((info["driver"], info["client_id"], info["pdev"]), ("i915", 42, "0000:03:00.0"))+ self.assertEqual(info["vram"], 2048 * MiB + 64 * 1024) # local0 plus the stolen-local0 it shrank into+ self.assertEqual(info["gtt"], 6 * MiB) # system0+ self.assertEqual(info["shared"], 0) # both shared regions are reported, and empty+ # Only the drm-engine-<class> counters are engines; the capacity lines are not.+ self.assertEqual(info["engines"], {"render": 5308400172, "copy": 1200, "video": 0,+ "video-enhance": 0, "compute": 750_000_000})++ def test_i915_integrated_client(self):+ info = drm.parse_fdinfo(I915_IGPU_FDINFO)+ self.assertIsNone(info["vram"]) # no local0 on an integrated GPU: unknown, not 0+ self.assertEqual(info["gtt"], 60 * MiB) # system0 plus an empty stolen-system0+ self.assertIsNone(info["shared"]) # no drm-shared-local0, so the shared-buffer gate stays closed+ self.assertEqual(set(info["engines"]), {"render", "copy", "video", "video-enhance"})++ def test_i915_without_engine_lines_has_unknown_load(self):+ info = drm.parse_fdinfo(I915_GEN7_FDINFO)+ self.assertIsNone(info["engines"]) # unknown, not an empty (zero) load+ self.assertEqual(info["gtt"], 4 * MiB) # the memory is still read++ def test_xe_client_reports_neither_memory_nor_load(self):+ info = drm.parse_fdinfo(XE_FDINFO)+ self.assertEqual((info["driver"], info["client_id"], info["pdev"]), ("xe", 3, "0000:03:00.0"))+ # xe calls its regions system/gtt/vram0/stolen and reports drm-cycles-*; none of that is decoded yet.+ for field in ("vram", "gtt", "shared", "engines"):+ with self.subTest(field=field):+ self.assertIsNone(info[field])+++class IntelScanTests(unittest.TestCase):+ """DrmCounters.sample() over Intel clients in a fake /proc. Each test builds its own card, because a sample+ stamps the card's used memory."""++ PDEV = "0000:03:00.0"++ def _setup(self, tmp, driver="i915"):+ proc = Path(tmp) / "proc"+ proc.mkdir()+ card = drm.DrmCard(key=drm.pci_key(self.PDEV), pdev=self.PDEV, sysfs=str(Path(tmp) / "card1" / "device"),+ driver=driver, vendor_id=0x8086, device_id=0x56A0, name="Intel Arc A770",+ vram_total=16 * GiB, integrated=False, intel=True)+ self.clock = Clock()+ return card, proc, drm.DrmCounters(found=[card], proc=str(proc), clock=self.clock)++ def test_render_and_video_load_are_measured_between_two_samples(self):+ with tempfile.TemporaryDirectory() as tmp:+ card, proc, counters = self._setup(tmp)+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, local_mib=256, system_mib=8))+ counters.start()+ self.clock.tick()+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, render=500_000_000, video=250_000_000,+ local_mib=256, system_mib=8))+ g = counters.sample()[0][card.key][100]+ self.assertIn("engine", g.available)+ self.assertEqual(dict(g.engines), {"3d": 50.0, "copy": 0.0, "video": 25.0}) # none fell into "other"+ self.assertEqual(g.gpu_pct, 50.0)+ self.assertEqual((g.drm_reason, g.dedicated, g.shared), ("", 256 * MiB, 8 * MiB))++ def test_xe_client_is_unknown_not_idle_and_not_empty(self):+ with tempfile.TemporaryDirectory() as tmp:+ card, proc, counters = self._setup(tmp, driver="xe")+ _write_fd(proc, 100, 5, XE_FDINFO)+ counters.start()+ self.clock.tick()+ procs, adapters = counters.sample()+ g = procs[card.key][100]+ self.assertEqual(g.drm_reason, drm.REASON_LEGACY)+ self.assertNotIn("engine", g.available) # drm-cycles-* is not decoded: unknown, not 0%+ self.assertNotIn("dedicated", g.available)+ self.assertIsNone(g.resident_vram)+ self.assertIsNone(adapters[card.key].dedicated) # and so is the card's used memory++ def test_unknown_engine_load_does_not_hide_measured_memory(self):+ with tempfile.TemporaryDirectory() as tmp:+ card, proc, counters = self._setup(tmp)+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, local_mib=64, engines=False))+ counters.start()+ self.clock.tick()+ g = counters.sample()[0][card.key][100]+ self.assertNotIn("engine", g.available) # no drm-engine lines: the load is unknown+ self.assertIn("dedicated", g.available) # and that has no bearing on the memory+ self.assertEqual(g.dedicated, 64 * MiB)++ def test_integrated_gpu_load_is_measured_while_its_vram_is_n_a(self):+ pdev = "0000:00:02.0"+ with tempfile.TemporaryDirectory() as tmp:+ proc = Path(tmp) / "proc"+ proc.mkdir()+ card = drm.DrmCard(key=drm.pci_key(pdev), pdev=pdev, sysfs=str(Path(tmp) / "card0" / "device"),+ driver="i915", vendor_id=0x8086, device_id=0xA782, name="Intel UHD Graphics",+ vram_total=0, integrated=True, intel=True)+ clock = Clock()+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=clock)+ _write_fd(proc, 100, 5, I915_IGPU_FDINFO)+ counters.start()+ clock.tick()+ _write_fd(proc, 100, 5, I915_IGPU_FDINFO.replace("render:\t1200000000", "render:\t1700000000"))+ procs, adapters = counters.sample()+ g = procs[card.key][100]+ self.assertEqual(g.drm_reason, drm.REASON_LEGACY) # shared buffers can't be ruled out+ self.assertNotIn("dedicated", g.available)+ self.assertIn("engine", g.available)+ self.assertEqual(g.engines["3d"], 50.0)+ self.assertIsNone(adapters[card.key].dedicated)++ def test_device_used_sums_each_client_once_and_is_unknown_when_incomplete(self):+ with tempfile.TemporaryDirectory() as tmp:+ card, proc, counters = self._setup(tmp)+ self.assertIsNone(counters.sample()[1][card.key].dedicated) # no client visible: unknown, not 0+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, local_mib=100))+ _write_fd(proc, 200, 5, _i915_fdinfo(1, self.PDEV, local_mib=100)) # the same client, held by a second process+ _write_fd(proc, 300, 5, _i915_fdinfo(2, self.PDEV, local_mib=50))+ adapters = counters.sample()[1]+ self.assertEqual(adapters[card.key].dedicated, 150 * MiB)+ self.assertIsNone(adapters[card.key].shared) # Intel has no device-wide system-memory counter+ _write_fd(proc, 400, 5, f"drm-driver:\ti915\ndrm-client-id:\t3\ndrm-pdev:\t{self.PDEV}\n"+ "drm-resident-local0:\tlots\n")+ self.assertIsNone(counters.sample()[1][card.key].dedicated) # one client unreadable: the sum is unknown+++class IntelNoteTests(unittest.TestCase):+ """Intel's device VRAM-used is a sum over the clients gputaskman can inspect, and the output says so."""++ def _snapshot(self, source, vram_used=4 * GiB):+ from gputaskman.collector import Snapshot+ from gputaskman.telemetry import DeviceTelemetry++ return Snapshot(taken=1, adapter=None, adapters=[], nv=DeviceTelemetry(index=0, source=source), driver="",+ cuda="", vram_total=16 * GiB, vram_used=vram_used, shared_used=None, backend="drm",+ procs=[])++ def test_text_carries_the_note_for_intel_only(self):+ from gputaskman.report import INTEL_MEMORY_NOTE, to_text++ self.assertIn(INTEL_MEMORY_NOTE, to_text(self._snapshot("Intel sysfs")))+ self.assertNotIn(INTEL_MEMORY_NOTE, to_text(self._snapshot("AMD sysfs"))) # amdgpu has a device counter+ self.assertNotIn(INTEL_MEMORY_NOTE, to_text(self._snapshot("Intel sysfs", vram_used=None))) # N/A, no caveat++ def test_device_panel_carries_the_note_for_intel_only(self):+ from gputaskman.app import GpuTaskManager+ from gputaskman.report import INTEL_MEMORY_NOTE++ def panel(snapshot):+ return "\n".join(item.plain for item in GpuTaskManager().render_device(snapshot).renderables)++ self.assertIn(INTEL_MEMORY_NOTE, panel(self._snapshot("Intel sysfs")))+ self.assertNotIn(INTEL_MEMORY_NOTE, panel(self._snapshot("AMD sysfs")))+ self.assertNotIn(INTEL_MEMORY_NOTE, panel(self._snapshot("Intel sysfs", vram_used=None)))++if __name__ == "__main__":unittest.main()