Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Fix Intel (i915/xe) fdinfo handling found in review

Scott Hanselman97bbce83fcc0
diff --git a/README.md b/README.md
index 1ecf9f9..eb7e55f 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
**An `htop` for what's inside your GPU, not just how full it is.**
-Windows + NVIDIA/AMD, with Linux and WSL support (NVIDIA everywhere, AMD on native Linux; see
+Windows + NVIDIA/AMD, with Linux and WSL support (NVIDIA everywhere, AMD and Intel on native Linux; see
[Linux and WSL](#linux-and-wsl)) and macOS on Apple Silicon (see [macOS](#macos-apple-silicon)). A terminal UI
that connects per-process GPU memory and engine activity with loaded runtimes, model files, and
LLM buffer breakdowns.
@@ -118,7 +118,7 @@ skips the consistency check, so do not use it after changing dependencies.
Tagged releases include portable Linux builds: `gputaskman-VERSION-linux-x64.tar.gz` and
`gputaskman-VERSION-linux-arm64.tar.gz`. They need glibc 2.35 or newer (Ubuntu 22.04, Debian 12,
-or newer) and an NVIDIA driver (or, on native Linux, an AMD GPU on the `amdgpu` driver), and work
+or newer) and an NVIDIA driver (or, on native Linux, an AMD GPU on the `amdgpu` driver or an Intel GPU on `i915`), and work
natively, in containers, and in WSL:
```bash
@@ -199,7 +199,8 @@ up without double counting. Otherwise the process shows VRAM and SPILL as `n/a`
resident minus shared is not private memory and is never computed.
- **Primary node** (`/dev/dri/card*`, typically a compositor): root-owned logind keeps a copy of that
fd, which a normal user can't see, so a single holder can't be proven.
-- **Legacy**: older kernels whose amdgpu fdinfo lacks `drm-shared-vram`/`drm-shared-gtt`.
+- **Legacy**: older kernels whose amdgpu fdinfo lacks `drm-shared-vram`/`drm-shared-gtt`, and Intel integrated GPUs
+ and `xe` (below).
- **Incomplete**: some of the process's fds or fdinfo could not be read (GPU% is `n/a` too).
- **Partial scan**: the TUI's 50 ms listing budget ran out this refresh, so a process it skipped might
hold the same client (GPU% is `n/a` too). Expected to be rare on a desktop (~9 ms for ~8,000 fds,
@@ -232,6 +233,27 @@ That check predates the stricter attribution above (no subtraction of shared byt
single-holder and complete-scan requirements; per-client timestamps; PID-reuse checks), which is
covered by unit tests only and **not yet validated on AMD hardware**.
+**Intel GPUs on native Linux** (the `i915` driver) use the same DRM fdinfo path, with these differences:
+
+| | Source |
+|---|---|
+| Device temperature, power, GT clock | `hwmon` and `gt/gt0` under `/sys/class/drm/card*`. Power is the rate of the cumulative energy counter between two refreshes, so it is `n/a` on the first |
+| Device VRAM total | Size of the local-memory BAR (`resource2`). A card whose window is under 1 GiB (an integrated GPU, or a discrete one without resizable BAR) is treated as integrated and has no total |
+| Device VRAM used | The sum of its clients' resident local memory: i915 has no device-wide counter, so buffers shared between processes can be counted twice and processes gputaskman can't inspect are missing (the UI and `--once` say so). `n/a` when any client's local memory is unknown |
+| Per-process VRAM and SPILL | `drm-resident-local0` and `drm-resident-system0`, plus their `stolen-` regions, under the same private-client rules as AMD. Both `drm-shared-local0` and `drm-shared-system0` must be present to charge a process, so an integrated GPU (no `local0`) shows `n/a` (`legacy`) |
+| Per-process engine load (3d, compute, copy, video) | `drm-engine-render`, `-compute`, `-copy`, `-video`, `-video-enhance` busy time, sampled twice. `n/a` where the driver reports none (before Gen 8). A class with several identical engines (`drm-engine-capacity-*`, such as video on an Arc) is not divided by its capacity, so one busy engine reads 100% |
+
+Device GPU load, memory bandwidth, and the PCIe link are `n/a`: i915 has no device-wide busy counter, and its
+sysfs link fields can report the power-on default instead of the live link. The GT clock is read but not shown yet.
+`xe` cards are detected, but its fdinfo uses other keys (`drm-cycles-*`, `drm-resident-vram0`/`-gtt`), so its
+per-process rows show `n/a` until they are decoded on hardware.
+
+Live check, reported by the contributor and not repeated by the maintainer: Linux 6.12, an Arc A770 (discrete, 16 GiB
+BAR) and a UHD 730 (integrated) on `i915`. Both cards were found, named, and selectable with `--gpu`; the Arc showed
+per-process VRAM and engine load, and the TUI rendered both. The fdinfo formats of both drivers, taken from the kernel
+sources, are covered by unit tests. **Not validated on hardware:** `xe`, Arc cards without resizable BAR, and
+multi-GPU systems other than that pair.
+
**WSL** is different, because Windows owns the GPU. Linux cannot see how much VRAM its own
processes use: `nvidia-smi` lists no processes, NVML's per-process samples carry *Windows* PIDs
(never looked up in Linux's `/proc`), and Windows charges all Linux GPU memory to the VM worker
@@ -303,7 +325,7 @@ Build both `gputaskman` and its `gtm` alias with `nix build path:.` (executables
pins nixpkgs. Dependencies come from nixpkgs, independently of `uv.lock`. No pip installation,
virtual environment, CUDA Toolkit, or ROCm installation is needed. GPU drivers remain supplied by
the host, never bundled with the package. NVIDIA's nixpkgs Python binding finds NVML through
-`/run/opengl-driver/lib`; AMD uses kernel sysfs and DRM fdinfo interfaces directly. On WSL2 the
+`/run/opengl-driver/lib`; AMD and Intel use kernel sysfs and DRM fdinfo interfaces directly. On WSL2 the
host driver lives in `/usr/lib/wsl/lib` instead, and the wrappers add that directory to
`LD_LIBRARY_PATH` (it is ignored when absent, so NixOS is unaffected).
@@ -681,7 +703,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:
| GGUF header, model-store manifests | ms, cached | Once per file / every 60 s |
| Local API probes | ms, local only | llama-server `/v1/models` only when its model path changes |
| `psutil.open_files()` | 0.6-2.5 s per process | Never |
-| AMD on Linux: DRM fdinfo scan (`readlink` every fd, read fdinfo of `/dev/dri` ones) | ~9 ms for ~8,000 fds (Linux desktop) | Every refresh, only when an amdgpu card is shown; listing fds is bounded to 50 ms in the TUI and resumes next refresh, known GPU fds are still read |
+| AMD and Intel on Linux: DRM fdinfo scan (`readlink` every fd, read fdinfo of `/dev/dri` ones) | ~9 ms for ~8,000 fds (AMD, Linux desktop; Intel not measured) | Every refresh, only when an amdgpu or Intel card is shown; listing fds is bounded to 50 ms in the TUI and resumes next refresh, known GPU fds are still read |
| macOS: `ioreg` device stats + Metal clients | ~10 ms | Every refresh |
| macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above |
| macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) |
@@ -820,7 +842,7 @@ downloads are bundled in this repository.
|---|---|
| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration |
| `linux.py` | Linux backend: NVML adapters and per-process usage, container PID matching, WSL detection, `/dev/dxg` holders |
-| `drm.py` | AMD on native Linux: amdgpu cards and device readings from sysfs, per-process VRAM/GTT/engine time from DRM fdinfo |
+| `drm.py` | AMD and Intel on native Linux: amdgpu and i915 cards and device readings from sysfs, per-process VRAM/GTT/engine time from DRM fdinfo |
| `wslhost.py` | WSL: interop check, Windows host counter stream, adapter pairing, context matching |
| `macos.py` | macOS backend: IOKit device stats and per-process GPU time, `footprint` GPU memory poller, libproc memory maps |
| `nvml.py` | NVIDIA device readings |
@@ -838,7 +860,8 @@ downloads are bundled in this repository.
- Model identity for non-GGUF runtimes (ONNX Runtime, PyTorch, TensorRT) beyond installed candidates.
- Explicit provenance and availability for every counter and estimate.
- Reliable multi-GPU matching and per-device model placement.
-- Bare-metal Linux validation on NVIDIA hardware; Intel GPUs (`i915`/`xe`) through the same DRM fdinfo path as AMD.
+- Bare-metal Linux validation on NVIDIA hardware; Intel `xe` per-process memory and load (`drm-cycles-*`), the Intel GT
+ clock in the device panel, and Intel validation beyond the Arc A770 / UHD 730 pair.
- macOS: a portable build, Ollama/llama.cpp/MLX validation, and compressed GPU memory (footprint's
`swapped`) as the paged column.
- WSL: per-process GPU load for Linux rows, and Ollama's systemd journal as a log source.
diff --git a/src/gputaskman/app.py b/src/gputaskman/app.py
index 8127771..34da39b 100644
--- a/src/gputaskman/app.py
+++ b/src/gputaskman/app.py
@@ -22,7 +22,7 @@ from .llm import gb
from .nvml import NvDevice, format_metric
from .recovery import LiveCollectors
from .report import (DRM_RESIDENT_NOTE, METAL_NOTE, WSL_EMPTY, drm_note, has_engine_counters, has_spill_counters, has_wddm_counters,
- inferred_note, vram_label, wsl_note)
+ inferred_note, intel_memory_note, vram_label, wsl_note)
KIND_STYLE = {
"LLM": "bold magenta", "decision": "bold orchid", "decision?": "orchid",
@@ -66,8 +66,8 @@ This display-only choice applies to all GPUs and resets to full when the app res
| Layer | Source | Works for |
|---|---|---|
-| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI; amdgpu sysfs on Linux | availability varies by GPU and driver |
-| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl); DRM fdinfo for amdgpu on Linux | workloads visible to WDDM or DRM |
+| **Device** telemetry | NVML, AMD ADL, or optional AMD SMI; amdgpu or Intel (i915) sysfs on Linux | availability varies by GPU and driver |
+| **Per-process** VRAM, spill to RAM, per-engine load | Windows WDDM counters (dxgkrnl); DRM fdinfo for amdgpu or Intel (i915) on Linux | workloads visible to WDDM or DRM |
| **Which runtime** (CUDA, HIP, Vulkan, D3D12, ggml, ONNX, PyTorch...) | DLLs mapped in the process | everything |
| **Which model file** | memory-mapped files + command line | llama.cpp, stable-diffusion.cpp, LM Studio, Ollama |
| **Model name & kind** (LLM, decision, embedding, speech) | GGUF header, Ollama/LM Studio/Hugging Face stores, local server APIs | GGUF family |
@@ -91,7 +91,11 @@ GTT (GPU memory in system RAM, shown as SPILL), and gfx/compute/video engine tim
first sample and for ROCm/HIP processes: their compute runs on `/dev/kfd` queues the kernel doesn't time per
process). VRAM and SPILL are charged to a process only when its DRM clients are provably private: one holder,
opened through a render node, and no shared buffers. Otherwise they show `n/a` and the bytes stay unattributed;
-the inspector still shows the resident bytes it can reach (shared buffers included, so not added up). Only processes you can inspect are
+the inspector still shows the resident bytes it can reach (shared buffers included, so not added up). **Intel GPUs
+on Linux** (`i915`) take the same path: sysfs for the device, fdinfo for engine load and, on a discrete Arc, local-memory
+VRAM. An integrated GPU shows engine load only (VRAM `n/a`), `xe` cards are detected but their per-process fdinfo isn't
+decoded yet (rows show `n/a`), and with no device-wide memory counter the device's VRAM used is a sum over the processes
+it can inspect: shared buffers can be counted twice and uninspectable processes are missing. Only processes you can inspect are
listed, and the header says how many it couldn't; run with `sudo` to see other users'. Under **WSL**, Windows
charges all Linux GPU memory to the VM (`vmwp.exe`), so VRAM and load are device-wide only; rows
are Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up share. When WSL
@@ -442,6 +446,8 @@ class GpuTaskManager(App):
legend.append("\n " + wsl_note(s), style="grey62")
if s.backend == "metal":
legend.append("\n " + METAL_NOTE, style="grey62")
+ if intel_memory_note(s):
+ legend.append("\n " + intel_memory_note(s), style="grey62")
if s.counter_status:
legend.append("\n " + s.counter_status, style="grey62")
if s.process_status:
diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.py
index 8ef2cea..ae79e77 100644
--- a/src/gputaskman/cli.py
+++ b/src/gputaskman/cli.py
@@ -46,7 +46,7 @@ def main() -> None:
ap.error("--interval must be a finite number greater than zero")
if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):
- sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu DRM) and macOS (Apple Silicon) only.")
+ sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu/Intel DRM) and macOS (Apple Silicon) only.")
if args.list_gpus:
from .collector import enumerate_adapters, ordered_adapters
diff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.py
index afd2f18..1bad53d 100644
--- a/src/gputaskman/collector.py
+++ b/src/gputaskman/collector.py
@@ -115,8 +115,8 @@ class Snapshot:
vram_used: int | None # WDDM adapter-wide dedicated usage
shared_used: int | None
procs: list[ProcRow] = field(default_factory=list)
- backend: str = "wddm" # wddm (Windows) | nvml (native Linux, NVIDIA) | drm (native Linux, AMD) | wsl (device-wide only) | metal (macOS)
- process_status: str = "" # amdgpu only: what the DRM process scan couldn't cover; empty when it saw everything
+ backend: str = "wddm" # wddm (Windows) | nvml (native Linux, NVIDIA) | drm (native Linux, AMD or Intel) | wsl (device-wide only) | metal (macOS)
+ process_status: str = "" # drm backend only: what the DRM process scan couldn't cover; empty when it saw everything
host_status: str = "" # WSL only: Windows host counters via interop -- ok | starting | off (...) | ...
wsl_split: str = "" # WSL only: why Linux rows have no (inferred) VRAM, when they don't
# False on an APU (amdgpu iGPU): its GTT is ordinary GPU memory, not spill from a full VRAM.
diff --git a/src/gputaskman/drm.py b/src/gputaskman/drm.py
index a1b2c8e..2649812 100644
--- a/src/gputaskman/drm.py
+++ b/src/gputaskman/drm.py
@@ -144,9 +144,15 @@ def parse_fdinfo(text: str) -> dict | None:
engines = None # a torn or malformed counter: this client's load is unknown, not 0
break
if fields.get("drm-driver") in ("i915", "xe"):
+ if not engines:
+ # i915 (gen8+) prints a drm-engine-<class> line for each engine class the device has, even at 0 ns;
+ # xe reports drm-cycles-* instead and gen7 i915 nothing. No such line means the load is unknown, not 0
+ # (AGENTS.md rule 2).
+ engines = None
# Intel's logical regions are local0 (device memory; the local/stolen pair covers memory that
# shrunk out of local into system) and system0 (GPU-placed system memory, the GTT equivalent).
- # There is no separate CPU-only region: system0 feeds the shared/spill slot instead.
+ # There is no separate CPU-only region: system0 feeds the shared/spill slot instead. These are
+ # i915's names; xe calls them vram0/gtt/system, so its memory stays unknown until validated.
return {
"driver": fields["drm-driver"],
"client_id": client_id,
@@ -467,25 +473,18 @@ def _vcn_major(device: str) -> int:
def _engine_key(name: str, unified_video: bool = False, intel: bool = False) -> str:
if intel:
lowered = name.lower()
- # i915/xe engine class-instance names: rcs0 render/3D, ccs0 compute,
- # bcs0 blitter (copies), vcs0 video decode, vecs0 video encode.
- if lowered.startswith("rcs"):
+ # i915's fdinfo names whole engine classes: render, copy, video, video-enhance, compute. The command-streamer
+ # names (rcs, ccs, bcs, vcs, vecs, optionally with an instance number) are how xe and the engine tables spell
+ # the same classes. Video-enhance (VECS) is post-processing, not encode, so it counts as video.
+ if lowered == "render" or lowered.startswith("rcs"):
return "3d"
- if lowered.startswith("ccs"):
+ if lowered == "compute" or lowered.startswith("ccs"):
return "compute"
- if lowered.startswith("bcs"):
+ if lowered == "copy" or lowered.startswith("bcs"):
return "copy"
- if lowered.startswith("vcs"):
- return "video"
- if lowered.startswith("vecs"):
- return "encode"
- if lowered == "compute":
- return "compute"
- if lowered == "copy":
- return "copy"
- if lowered.startswith("video"):
- return "video" # video / video-enhance: decode and encode can't be told apart
- return "other" # any other class instance: busy time still counts toward GPU%
+ if lowered.startswith(("video", "vcs", "vecs")):
+ return "video" # decode and encode can't be told apart
+ return "other" # any other class: busy time still counts toward GPU%
lowered = name.lower()
if unified_video and ("enc" in lowered or "dec" in lowered):
return "video" # one ring for both: decode and encode can't be told apart
@@ -569,8 +568,8 @@ class DrmCounters:
def _stamp_intel_memory(self, clients: dict[tuple, list]) -> None:
"""Intel cards expose no device-wide memory sysfs: used local memory is the sum of the card's
clients' resident local bytes, counted once per client even when several processes share it
- (buffers shared between clients can double-count; the status note says so). With no readable
- client the sum is None -- unknown, not 0."""
+ (buffers shared between clients can double-count, and clients that can't be inspected are missing;
+ report.INTEL_MEMORY_NOTE says so). With no readable client the sum is None -- unknown, not 0."""
for card in self.cards:
if not card.intel:
continue
diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.py
index adf54fe..621176f 100644
--- a/src/gputaskman/pdh.py
+++ b/src/gputaskman/pdh.py
@@ -84,11 +84,11 @@ class ProcGpu:
# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.
inferred: float | None = None
inferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)
- # amdgpu (DRM fdinfo): resident bytes in every buffer this process can reach, shared ones included. A reference
+ # DRM fdinfo (amdgpu, Intel): resident bytes in every buffer this process can reach, shared ones included. A reference
# for display only; the same buffer can appear under several processes, so these are never summed. None: unknown.
resident_vram: float | None = None
resident_gtt: float | None = None
- drm_reason: str = "" # amdgpu: why dedicated/shared are n/a (drm.REASON_*); empty when they are measured
+ drm_reason: str = "" # DRM fdinfo: why dedicated/shared are n/a (drm.REASON_*); empty when they are measured
kfd: bool = False # amdgpu: holds /dev/kfd (ROCm/HIP), whose compute isn't timed per process
@property
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index f0f0062..4820cd8 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -35,7 +35,8 @@ def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:
def has_spill_counters(s: Snapshot, row: ProcRow) -> bool:
- """Per-process GPU memory in system RAM: WDDM's non-local usage, or amdgpu's GTT on Linux."""
+ """Per-process GPU memory in system RAM: WDDM's non-local usage, or a DRM client's GTT (amdgpu) or
+ system0 (Intel) bytes on Linux."""
return (has_wddm_counters(s, row) or (s.backend == "drm" and not row.host)) and "shared" in row.gpu.available
@@ -59,8 +60,9 @@ ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer
DRM_NOTES = {
- "legacy": ("VRAM n/a: this kernel's amdgpu fdinfo has incomplete or unparseable drm-shared-* stats, so resident "
- "memory may include buffers imported from other processes and would be counted once per importer."),
+ "legacy": ("VRAM n/a: this driver's fdinfo has missing or unparseable drm-shared-* stats (older amdgpu kernels, "
+ "Intel integrated GPUs, xe), so resident memory may include buffers imported from other processes "
+ "and would be counted once per importer."),
"shared_client": ("VRAM and GPU% n/a: this process holds a DRM client other processes hold too (fd passing, "
"fork). fdinfo counts the client once and doesn't say which of them uses it."),
"shared_buffers": ("VRAM n/a: this process's DRM client shares buffers with other clients (dma-buf, e.g. "
@@ -87,6 +89,15 @@ def inferred_note(g) -> str:
METAL_NOTE = ("macOS: unified memory, so GPU memory is part of system RAM. Rows show each process's graphics "
"footprint; other users' processes (WindowServer, daemons) show GPU% only, since reading their "
"memory needs root.")
+INTEL_MEMORY_NOTE = ("Intel VRAM used is the sum of local memory resident in the processes gputaskman can inspect "
+ "(the driver has no device-wide counter): buffers shared between processes can be counted "
+ "twice, and processes it can't inspect are missing.")
+
+
+def intel_memory_note(s: Snapshot) -> str:
+ """Intel's device VRAM used is a sum over visible DRM clients, not a device counter."""
+ return INTEL_MEMORY_NOTE if (s.backend == "drm" and s.nv and s.nv.source == "Intel sysfs"
+ and s.vram_used is not None) else ""
def to_dict(s: Snapshot) -> dict:
@@ -169,6 +180,8 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
out.append(memory)
if s.backend == "wsl":
out.append(wsl_note(s))
+ if intel_memory_note(s):
+ out.append(intel_memory_note(s))
if s.counter_status:
out.append(s.counter_status)
if nv and s.backend != "metal":
diff --git a/tests/test_drm.py b/tests/test_drm.py
index 717b4ff..2abe286 100644
--- a/tests/test_drm.py
+++ b/tests/test_drm.py
@@ -9,6 +9,7 @@ from pathlib import Path
from gputaskman import drm
GiB = 1 << 30
+MiB = 1 << 20
FDINFO = """pos:\t0
flags:\t02100002
@@ -42,6 +43,115 @@ drm-engine-enc:\t1200 ns
drm-engine-capacity-gfx:\t1
"""
+# The Intel samples are laid out the way the kernels print them (i915_drm_client.c; the xe_drm_client.c documentation
+# sample; sizes in the units drm_print_memory_stats uses). They are built from that code, not captured from hardware.
+I915_FDINFO = """pos:\t0
+flags:\t02100002
+mnt_id:\t28
+ino:\t415
+drm-driver:\ti915
+drm-pdev:\t0000:03:00.0
+drm-client-id:\t42
+drm-total-system0:\t8 MiB
+drm-shared-system0:\t0
+drm-active-system0:\t0
+drm-resident-system0:\t6 MiB
+drm-purgeable-system0:\t0
+drm-total-local0:\t2048 MiB
+drm-shared-local0:\t0
+drm-active-local0:\t16 MiB
+drm-resident-local0:\t2048 MiB
+drm-purgeable-local0:\t0
+drm-total-stolen-local0:\t0
+drm-shared-stolen-local0:\t0
+drm-active-stolen-local0:\t0
+drm-resident-stolen-local0:\t64 KiB
+drm-purgeable-stolen-local0:\t0
+drm-engine-render:\t5308400172 ns
+drm-engine-copy:\t1200 ns
+drm-engine-video:\t0 ns
+drm-engine-capacity-video:\t2
+drm-engine-video-enhance:\t0 ns
+drm-engine-capacity-video-enhance:\t2
+drm-engine-compute:\t750000000 ns
+drm-engine-capacity-compute:\t4
+"""
+
+# An integrated GPU has no local0 region, only system0 and its stolen carve-out.
+I915_IGPU_FDINFO = """pos:\t0
+flags:\t02100002
+mnt_id:\t28
+ino:\t415
+drm-driver:\ti915
+drm-pdev:\t0000:00:02.0
+drm-client-id:\t7
+drm-total-system0:\t64 MiB
+drm-shared-system0:\t0
+drm-active-system0:\t0
+drm-resident-system0:\t60 MiB
+drm-purgeable-system0:\t0
+drm-total-stolen-system0:\t0
+drm-shared-stolen-system0:\t0
+drm-active-stolen-system0:\t0
+drm-resident-stolen-system0:\t0
+drm-purgeable-stolen-system0:\t0
+drm-engine-render:\t1200000000 ns
+drm-engine-copy:\t0 ns
+drm-engine-video:\t300000000 ns
+drm-engine-video-enhance:\t0 ns
+"""
+
+# i915 before Gen8 prints the memory stats but no drm-engine lines at all.
+I915_GEN7_FDINFO = """drm-driver:\ti915
+drm-pdev:\t0000:00:02.0
+drm-client-id:\t5
+drm-total-system0:\t4 MiB
+drm-shared-system0:\t0
+drm-active-system0:\t0
+drm-resident-system0:\t4 MiB
+drm-purgeable-system0:\t0
+"""
+
+# xe: memory regions are named system/gtt/vram0/stolen and the load is drm-cycles-*, not drm-engine-<class> ns.
+XE_FDINFO = """pos:\t0
+flags:\t0100002
+mnt_id:\t26
+ino:\t685
+drm-driver:\txe
+drm-client-id:\t3
+drm-pdev:\t0000:03:00.0
+drm-total-system:\t0
+drm-shared-system:\t0
+drm-active-system:\t0
+drm-resident-system:\t0
+drm-purgeable-system:\t0
+drm-total-gtt:\t192 KiB
+drm-shared-gtt:\t0
+drm-active-gtt:\t0
+drm-resident-gtt:\t192 KiB
+drm-total-vram0:\t23992 KiB
+drm-shared-vram0:\t16 MiB
+drm-active-vram0:\t0
+drm-resident-vram0:\t23992 KiB
+drm-total-stolen:\t0
+drm-shared-stolen:\t0
+drm-active-stolen:\t0
+drm-resident-stolen:\t0
+drm-cycles-rcs:\t28257900
+drm-total-cycles-rcs:\t7655183225
+drm-cycles-bcs:\t0
+drm-total-cycles-bcs:\t7655183225
+drm-cycles-vcs:\t0
+drm-total-cycles-vcs:\t7655183225
+drm-engine-capacity-vcs:\t2
+drm-cycles-vecs:\t0
+drm-total-cycles-vecs:\t7655183225
+drm-engine-capacity-vecs:\t2
+drm-cycles-ccs:\t0
+drm-total-cycles-ccs:\t7655183225
+drm-engine-capacity-ccs:\t4
+"""
+
def _make_card(root: Path, name, pdev, vendor, device, vram, driver="amdgpu", gtt=0, apu=False) -> Path:
dev = root / "drm" / name / "device"
@@ -96,6 +206,26 @@ def _fdinfo(client, pdev, gfx=0, vram=0, gtt=0, shared_vram=0, shared_gtt=0, sha
)
+def _i915_fdinfo(client, pdev, render=0, video=0, local_mib=0, system_mib=0, shared_local_mib=0, engines=True) -> str:
+ """A discrete i915 client's fdinfo: memory by region (sizes in MiB) and, unless engines is False, the engine
+ classes the way i915 prints them."""
+ text = (
+ "pos:\t0\n"
+ "drm-driver:\ti915\n"
+ f"drm-client-id:\t{client}\n"
+ f"drm-pdev:\t{pdev}\n"
+ f"drm-shared-system0:\t0\n"
+ f"drm-resident-system0:\t{system_mib} MiB\n"
+ f"drm-shared-local0:\t{shared_local_mib} MiB\n"
+ f"drm-resident-local0:\t{local_mib} MiB\n"
+ )
+ if engines:
+ text += (f"drm-engine-render:\t{render} ns\n"
+ "drm-engine-copy:\t0 ns\n"
+ f"drm-engine-video:\t{video} ns\n")
+ return text
+
+
class Clock:
"""A monotonic_ns stand-in the test advances by hand: every fdinfo read in one refresh sees the same time."""
@@ -792,9 +922,19 @@ class EngineAndSelectionTests(unittest.TestCase):
self.assertEqual(drm._engine_key("ccs0", intel=True), "compute")
self.assertEqual(drm._engine_key("bcs0", intel=True), "copy")
self.assertEqual(drm._engine_key("vcs0", intel=True), "video")
- self.assertEqual(drm._engine_key("vecs0", intel=True), "encode")
+ self.assertEqual(drm._engine_key("vecs0", intel=True), "video") # video-enhance is post-processing, not encode
self.assertEqual(drm._engine_key("rcs0", intel=False), "other") # Intel-only prefix on AMD
+ def test_amd_selectable_next_to_nvidia_on_linux(self):
+ from gputaskman import collector
+ from gputaskman.dxgi import Adapter
+
+ nvidia = Adapter(1, "NVIDIA GPU", 0x10DE, 8 * GiB, 0, False)
+ amd = Adapter(2, "AMD Radeon 610M", 0x1002, 2 * GiB, 0, False)
+ with patch.object(collector, "BACKEND", "nvml"):
+ self.assertIs(collector.select_adapter([amd, nvidia], 0), nvidia)
+ self.assertIs(collector.select_adapter([amd, nvidia], 1), amd)
+
def test_intel_selectable_next_to_nvidia_on_linux(self):
from gputaskman import collector
from gputaskman.dxgi import Adapter
@@ -827,9 +967,18 @@ class IntelCardTests(unittest.TestCase):
self.assertEqual(drm._engine_key("ccs0", intel=True), "compute")
self.assertEqual(drm._engine_key("bcs0", intel=True), "copy")
self.assertEqual(drm._engine_key("vcs0", intel=True), "video")
- self.assertEqual(drm._engine_key("vecs0", intel=True), "encode")
+ self.assertEqual(drm._engine_key("vecs0", intel=True), "video")
self.assertEqual(drm._engine_key("rcs0", intel=False), "other")
+ def test_intel_engine_classes_as_i915_prints_them(self):
+ # i915 names whole classes (i915_drm_client.c uabi_class_names), not command streamers.
+ for name, expected in (("render", "3d"), ("copy", "copy"), ("video", "video"),
+ ("video-enhance", "video"), ("compute", "compute")):
+ with self.subTest(name=name):
+ self.assertEqual(drm._engine_key(name, intel=True), expected)
+ self.assertEqual(drm._engine_key("something-new", intel=True), "other")
+ self.assertEqual(drm._engine_key("render", intel=False), "other") # not an AMD engine name
+
def test_intel_card_is_recognized_and_amd_unchanged(self):
counters = self.make([self.INTEL])
self.assertTrue(counters.ok)
@@ -880,5 +1029,160 @@ class IntelCardTests(unittest.TestCase):
self.assertIsNone(out[self.IGPU.key].shared)
+class IntelFdinfoTests(unittest.TestCase):
+ """fdinfo laid out the way the Intel kernels print it (the fixtures at the top of this file)."""
+
+ def test_i915_discrete_client(self):
+ info = drm.parse_fdinfo(I915_FDINFO)
+ self.assertEqual((info["driver"], info["client_id"], info["pdev"]), ("i915", 42, "0000:03:00.0"))
+ self.assertEqual(info["vram"], 2048 * MiB + 64 * 1024) # local0 plus the stolen-local0 it shrank into
+ self.assertEqual(info["gtt"], 6 * MiB) # system0
+ self.assertEqual(info["shared"], 0) # both shared regions are reported, and empty
+ # Only the drm-engine-<class> counters are engines; the capacity lines are not.
+ self.assertEqual(info["engines"], {"render": 5308400172, "copy": 1200, "video": 0,
+ "video-enhance": 0, "compute": 750_000_000})
+
+ def test_i915_integrated_client(self):
+ info = drm.parse_fdinfo(I915_IGPU_FDINFO)
+ self.assertIsNone(info["vram"]) # no local0 on an integrated GPU: unknown, not 0
+ self.assertEqual(info["gtt"], 60 * MiB) # system0 plus an empty stolen-system0
+ self.assertIsNone(info["shared"]) # no drm-shared-local0, so the shared-buffer gate stays closed
+ self.assertEqual(set(info["engines"]), {"render", "copy", "video", "video-enhance"})
+
+ def test_i915_without_engine_lines_has_unknown_load(self):
+ info = drm.parse_fdinfo(I915_GEN7_FDINFO)
+ self.assertIsNone(info["engines"]) # unknown, not an empty (zero) load
+ self.assertEqual(info["gtt"], 4 * MiB) # the memory is still read
+
+ def test_xe_client_reports_neither_memory_nor_load(self):
+ info = drm.parse_fdinfo(XE_FDINFO)
+ self.assertEqual((info["driver"], info["client_id"], info["pdev"]), ("xe", 3, "0000:03:00.0"))
+ # xe calls its regions system/gtt/vram0/stolen and reports drm-cycles-*; none of that is decoded yet.
+ for field in ("vram", "gtt", "shared", "engines"):
+ with self.subTest(field=field):
+ self.assertIsNone(info[field])
+
+
+class IntelScanTests(unittest.TestCase):
+ """DrmCounters.sample() over Intel clients in a fake /proc. Each test builds its own card, because a sample
+ stamps the card's used memory."""
+
+ PDEV = "0000:03:00.0"
+
+ def _setup(self, tmp, driver="i915"):
+ proc = Path(tmp) / "proc"
+ proc.mkdir()
+ card = drm.DrmCard(key=drm.pci_key(self.PDEV), pdev=self.PDEV, sysfs=str(Path(tmp) / "card1" / "device"),
+ driver=driver, vendor_id=0x8086, device_id=0x56A0, name="Intel Arc A770",
+ vram_total=16 * GiB, integrated=False, intel=True)
+ self.clock = Clock()
+ return card, proc, drm.DrmCounters(found=[card], proc=str(proc), clock=self.clock)
+
+ def test_render_and_video_load_are_measured_between_two_samples(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, counters = self._setup(tmp)
+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, local_mib=256, system_mib=8))
+ counters.start()
+ self.clock.tick()
+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, render=500_000_000, video=250_000_000,
+ local_mib=256, system_mib=8))
+ g = counters.sample()[0][card.key][100]
+ self.assertIn("engine", g.available)
+ self.assertEqual(dict(g.engines), {"3d": 50.0, "copy": 0.0, "video": 25.0}) # none fell into "other"
+ self.assertEqual(g.gpu_pct, 50.0)
+ self.assertEqual((g.drm_reason, g.dedicated, g.shared), ("", 256 * MiB, 8 * MiB))
+
+ def test_xe_client_is_unknown_not_idle_and_not_empty(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, counters = self._setup(tmp, driver="xe")
+ _write_fd(proc, 100, 5, XE_FDINFO)
+ counters.start()
+ self.clock.tick()
+ procs, adapters = counters.sample()
+ g = procs[card.key][100]
+ self.assertEqual(g.drm_reason, drm.REASON_LEGACY)
+ self.assertNotIn("engine", g.available) # drm-cycles-* is not decoded: unknown, not 0%
+ self.assertNotIn("dedicated", g.available)
+ self.assertIsNone(g.resident_vram)
+ self.assertIsNone(adapters[card.key].dedicated) # and so is the card's used memory
+
+ def test_unknown_engine_load_does_not_hide_measured_memory(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, counters = self._setup(tmp)
+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, local_mib=64, engines=False))
+ counters.start()
+ self.clock.tick()
+ g = counters.sample()[0][card.key][100]
+ self.assertNotIn("engine", g.available) # no drm-engine lines: the load is unknown
+ self.assertIn("dedicated", g.available) # and that has no bearing on the memory
+ self.assertEqual(g.dedicated, 64 * MiB)
+
+ def test_integrated_gpu_load_is_measured_while_its_vram_is_n_a(self):
+ pdev = "0000:00:02.0"
+ with tempfile.TemporaryDirectory() as tmp:
+ proc = Path(tmp) / "proc"
+ proc.mkdir()
+ card = drm.DrmCard(key=drm.pci_key(pdev), pdev=pdev, sysfs=str(Path(tmp) / "card0" / "device"),
+ driver="i915", vendor_id=0x8086, device_id=0xA782, name="Intel UHD Graphics",
+ vram_total=0, integrated=True, intel=True)
+ clock = Clock()
+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=clock)
+ _write_fd(proc, 100, 5, I915_IGPU_FDINFO)
+ counters.start()
+ clock.tick()
+ _write_fd(proc, 100, 5, I915_IGPU_FDINFO.replace("render:\t1200000000", "render:\t1700000000"))
+ procs, adapters = counters.sample()
+ g = procs[card.key][100]
+ self.assertEqual(g.drm_reason, drm.REASON_LEGACY) # shared buffers can't be ruled out
+ self.assertNotIn("dedicated", g.available)
+ self.assertIn("engine", g.available)
+ self.assertEqual(g.engines["3d"], 50.0)
+ self.assertIsNone(adapters[card.key].dedicated)
+
+ def test_device_used_sums_each_client_once_and_is_unknown_when_incomplete(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, counters = self._setup(tmp)
+ self.assertIsNone(counters.sample()[1][card.key].dedicated) # no client visible: unknown, not 0
+ _write_fd(proc, 100, 5, _i915_fdinfo(1, self.PDEV, local_mib=100))
+ _write_fd(proc, 200, 5, _i915_fdinfo(1, self.PDEV, local_mib=100)) # the same client, held by a second process
+ _write_fd(proc, 300, 5, _i915_fdinfo(2, self.PDEV, local_mib=50))
+ adapters = counters.sample()[1]
+ self.assertEqual(adapters[card.key].dedicated, 150 * MiB)
+ self.assertIsNone(adapters[card.key].shared) # Intel has no device-wide system-memory counter
+ _write_fd(proc, 400, 5, f"drm-driver:\ti915\ndrm-client-id:\t3\ndrm-pdev:\t{self.PDEV}\n"
+ "drm-resident-local0:\tlots\n")
+ self.assertIsNone(counters.sample()[1][card.key].dedicated) # one client unreadable: the sum is unknown
+
+
+class IntelNoteTests(unittest.TestCase):
+ """Intel's device VRAM-used is a sum over the clients gputaskman can inspect, and the output says so."""
+
+ def _snapshot(self, source, vram_used=4 * GiB):
+ from gputaskman.collector import Snapshot
+ from gputaskman.telemetry import DeviceTelemetry
+
+ return Snapshot(taken=1, adapter=None, adapters=[], nv=DeviceTelemetry(index=0, source=source), driver="",
+ cuda="", vram_total=16 * GiB, vram_used=vram_used, shared_used=None, backend="drm",
+ procs=[])
+
+ def test_text_carries_the_note_for_intel_only(self):
+ from gputaskman.report import INTEL_MEMORY_NOTE, to_text
+
+ self.assertIn(INTEL_MEMORY_NOTE, to_text(self._snapshot("Intel sysfs")))
+ self.assertNotIn(INTEL_MEMORY_NOTE, to_text(self._snapshot("AMD sysfs"))) # amdgpu has a device counter
+ self.assertNotIn(INTEL_MEMORY_NOTE, to_text(self._snapshot("Intel sysfs", vram_used=None))) # N/A, no caveat
+
+ def test_device_panel_carries_the_note_for_intel_only(self):
+ from gputaskman.app import GpuTaskManager
+ from gputaskman.report import INTEL_MEMORY_NOTE
+
+ def panel(snapshot):
+ return "\n".join(item.plain for item in GpuTaskManager().render_device(snapshot).renderables)
+
+ self.assertIn(INTEL_MEMORY_NOTE, panel(self._snapshot("Intel sysfs")))
+ self.assertNotIn(INTEL_MEMORY_NOTE, panel(self._snapshot("AMD sysfs")))
+ self.assertNotIn(INTEL_MEMORY_NOTE, panel(self._snapshot("Intel sysfs", vram_used=None)))
+
+
if __name__ == "__main__":
unittest.main()