Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Charge amdgpu memory only to provably private DRM clients

Scott Hanselman632916c42c91
diff --git a/AGENTS.md b/AGENTS.md
index e1f252b..f0b4121 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -19,7 +19,9 @@ these rules:
A measured zero stays `0`. Engine availability is independent of VRAM availability.
3. **Windows per-process memory uses `Local Usage` / `Non Local Usage`**, not `Dedicated Usage`,
which counts shared surfaces against `dwm.exe`, `csrss.exe`, and the owning app. The adapter
- counter is the reference total.
+ counter is the reference total. On Linux amdgpu (`drm.py`), a process gets bytes only when all its
+ DRM clients are provably private (one holder, render node, every `drm-shared-*` is 0); never
+ compute resident minus shared, since the two count by different placements.
4. **Model evidence has a ladder:** `api` > `mapped` > `cmdline` > `installed`. App settings mean
*selected*. Log events are *timestamped history*, not current residency. Model-store folders only
supply friendly names. Name-based hints (`kind_hint`, the `decision?` category) never change the
diff --git a/README.md b/README.md
index d00b50a..d80f2a6 100644
--- a/README.md
+++ b/README.md
@@ -165,16 +165,28 @@ the device), like WDDM's non-local usage. On an APU (an integrated GPU, detected
reporting a VRAM vendor, a heuristic validated on one APU) GTT is the GPU's normal memory and "VRAM"
is only a RAM carve-out, so GTT is still shown but never flagged as spill, and an LLM's memory is its
VRAM plus GTT. The SDMA engine (`dma`) fills the COPY column. Paged stays blank: there is no commit
-counter. A DRM client
-held by several processes (fd passing, fork; 1Password and its crash handler on the test desktop) has one
-set of counters and nothing says which process uses it, so each is listed with VRAM and GPU% `n/a` and the
-bytes stay in *unattributed*. A *buffer* shared
-between clients (a window a compositor imports, for example) is resident in each of them, and fdinfo
-does not say which one owns it. VRAM therefore shows each process's private bytes (resident minus
-shared), which add up without double counting; shared buffers stay in *unattributed* and are listed
-per process, owner unknown, in the inspector and JSON (`vram_shared_bytes`); shared GTT is left out
-of SPILL the same way. Older kernels whose amdgpu fdinfo has no `drm-shared-*` keys can't make that
-split, so per-process VRAM shows `n/a` there instead of double counting. amdgpu has no device-wide
+counter. VRAM and SPILL are charged to a process only when nothing else can hold its bytes: each of
+its DRM clients has exactly one visible holder, was opened through a render node (`/dev/dri/renderD*`),
+and reports zero bytes in every `drm-shared-*` region. Then its resident VRAM and GTT are its own and add
+up without double counting. Otherwise the process shows VRAM and SPILL as `n/a` (`null` in JSON, with
+`vram_unattributed_reason`) and its bytes stay in *unattributed*:
+
+- **Shared client** (fd passing, fork; 1Password and its crash handler on the test desktop): one set of
+ counters, and nothing says which process uses it. GPU% is `n/a` too.
+- **Shared buffers** (a window a compositor imports, video frames): `drm-shared-*` counts a buffer by
+ where it *prefers* to live, resident or not, while `drm-resident-*` counts where buffers *are*, so
+ resident minus shared is not private memory and is never computed.
+- **Primary node** (`/dev/dri/card*`, typically a compositor): root-owned logind keeps a copy of that
+ fd, which a normal user can't see, so a single holder can't be proven.
+- **Legacy**: older kernels whose amdgpu fdinfo lacks `drm-shared-vram`/`drm-shared-gtt`.
+- **Incomplete**: some of the process's fds or fdinfo could not be read (GPU% is `n/a` too).
+- **Partial scan**: the TUI's 50 ms listing budget ran out this refresh, so a process it skipped might
+ hold the same client (GPU% is `n/a` too). Expected to be rare on a desktop (~9 ms for ~8,000 fds,
+ not yet re-measured with this rule); one-shot output always scans fully.
+
+The inspector and JSON (`vram_resident_bytes`, `gtt_resident_bytes`) still show every buffer the
+process can reach, shared ones included; the same buffer can appear under several processes, so these
+are a reference and are never summed. Unparseable or missing values are unknown (`null`), not 0. amdgpu has no device-wide
video engine counter, so the device's encode/decode load stays N/A; per-process video load is
measured. On VCN 4 and later (RDNA3 and newer, including the RX 9070) decode and encode share one
ring, so it is reported as `video` rather than split. Only processes you can inspect are listed; the header and JSON (`process_status`) say how many could not be, and `sudo` includes other users'. `--gpu 1`
@@ -187,14 +199,17 @@ This path is native-Linux only; under WSL the GPU is `/dev/dxg`, with no DRM dri
Live check, bare-metal Arch-based Linux (kernel 7.2.15, `amdgpu`), Radeon RX 9070-series (Navi 48)
plus a Radeon 610M iGPU: both cards were found, named, and read; VAAPI HEVC encode and decode with `ffmpeg`
each showed as a `video` row at ~95-100% video load (on the RX 9070 both were charged to amdgpu's
-unified `enc` ring, hence `video`); desktop processes' private VRAM summed
+unified `enc` ring, hence `video`); desktop processes' VRAM summed
to less than the card's used VRAM, with the compositor's imported windows listed as shared.
llama.cpp's ROCm server image (`ghcr.io/ggml-org/llama.cpp:server-rocm`, Qwen3-4B-Instruct-2507
Q4_K_M, `-ngl 99 -c 8192`, next to a running game) showed as an `LLM` row with `llama.cpp+ggml+HIP`
-and 3.95 GB of private VRAM, matching the 2.49 GB of weights plus 1.21 GB of f16 KV cache plus compute
+and 3.95 GB of VRAM, matching the 2.49 GB of weights plus 1.21 GB of f16 KV cache plus compute
buffers, and the 4.11 GB KFD reports for the process in `/sys/class/kfd/kfd/proc/<pid>/vram_*`.
While it generated at 103 tok/s the device read 99% gfx and 339 W, but fdinfo had no engine lines
for it, which is why ROCm rows show engines as `n/a`. PyTorch-ROCm has not been run on this path.
+That check predates the stricter attribution above (no subtraction of shared bytes; render-node,
+single-holder and complete-scan requirements; per-client timestamps; PID-reuse checks), which is
+covered by unit tests only and **not yet validated on AMD hardware**.
**WSL** is different, because Windows owns the GPU. Linux cannot see how much VRAM its own
processes use: `nvidia-smi` lists no processes, NVML's per-process samples carry *Windows* PIDs
diff --git a/src/gputaskman/app.py b/src/gputaskman/app.py
index 5e0712e..69e566a 100644
--- a/src/gputaskman/app.py
+++ b/src/gputaskman/app.py
@@ -20,7 +20,7 @@ from textual.worker import get_current_worker
from .collector import BACKEND, Collector, ProcRow, Snapshot
from .llm import gb
from .nvml import NvDevice, format_metric
-from .report import (DRM_SHARED_CLIENT_NOTE, DRM_UNSPLIT_NOTE, METAL_NOTE, WSL_EMPTY, has_engine_counters, has_spill_counters, has_wddm_counters,
+from .report import (DRM_RESIDENT_NOTE, METAL_NOTE, WSL_EMPTY, drm_note, has_engine_counters, has_spill_counters, has_wddm_counters,
inferred_note, vram_label, wsl_note)
KIND_STYLE = {
@@ -83,7 +83,9 @@ candidates remain unproven; engine activity is measured per process, not per mod
(`amdgpu`) use sysfs for the device and DRM fdinfo (`/proc/<pid>/fdinfo`) per process: VRAM,
GTT (GPU memory in system RAM, shown as SPILL), and gfx/compute/video engine time (`n/a` for a new client's
first sample and for ROCm/HIP processes: their compute runs on `/dev/kfd` queues the kernel doesn't time per
-process). A client several processes share shows `n/a` for each of them. Only processes you can inspect are
+process). VRAM and SPILL are charged to a process only when its DRM clients are provably private: one holder,
+opened through a render node, and no shared buffers. Otherwise they show `n/a` and the bytes stay unattributed;
+the inspector still shows the resident bytes it can reach (shared buffers included, so not added up). Only processes you can inspect are
listed, and the header says how many it couldn't; run with `sudo` to see other users'. Under **WSL**, Windows
charges all Linux GPU memory to the VM (`vmwp.exe`), so VRAM and load are device-wide only; rows
are Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up share. When WSL
@@ -484,7 +486,7 @@ class GpuTaskManager(App):
style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") and s.shared_is_spill
else "yellow")
paged = Text(gb(r.gpu.paged_out) if wddm and r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")
- if wddm and "shared" not in r.gpu.available:
+ if (wddm or (s.backend == "drm" and not r.host)) and "shared" not in r.gpu.available:
spill = Text("n/a", justify="right", style="grey62")
if wddm and not r.gpu.paged_available:
paged = Text("n/a", justify="right", style="grey62")
@@ -540,15 +542,15 @@ class GpuTaskManager(App):
elif s.wsl_split:
out.append(Text(f"No per-process VRAM: {s.wsl_split}.", style="grey62"))
elif s.backend == "drm":
- out.append(Text(DRM_SHARED_CLIENT_NOTE if r.gpu.shared_client else DRM_UNSPLIT_NOTE, style="grey62"))
+ out.append(Text(drm_note(r.gpu), style="grey62"))
elif s.backend == "metal":
out.append(Text("GPU memory n/a: footprint can't read other users' processes without root "
"(or hasn't measured this one yet). GPU% comes from Metal's per-process GPU time.",
style="grey62"))
- if s.backend == "drm" and r.gpu.shared_vram >= 1 << 20:
- out.append(Text(f"+ {gb(r.gpu.shared_vram)} in VRAM buffers shared with other processes (e.g. windows a "
- "compositor imports), not counted above: the driver doesn't say which process owns a shared "
- "buffer, so they stay in 'unattributed'.", style="grey62"))
+ if s.backend == "drm" and not r.host and (r.gpu.resident_vram is not None or r.gpu.resident_gtt is not None):
+ def size(v): return gb(v) if v is not None else "n/a"
+ out.append(Text(f"Resident VRAM {size(r.gpu.resident_vram)}, GTT {size(r.gpu.resident_gtt)}. "
+ + DRM_RESIDENT_NOTE, style="grey62"))
if s.backend == "drm" and r.gpu.kfd:
out.append(Text("Engines n/a: this process uses ROCm/HIP (/dev/kfd), whose compute queues the kernel doesn't "
"time per process. The device's AMD gfx load includes its work.", style="grey62"))
diff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.py
index 51fb614..041adcc 100644
--- a/src/gputaskman/collector.py
+++ b/src/gputaskman/collector.py
@@ -195,8 +195,10 @@ class Collector:
self.drm_card = next((c for c in drm.cards if c.key == self.adapter.luid), None)
if drm is not None and self.drm_card is None:
self.counters.drm = None # an NVIDIA GPU is shown: don't walk every process's fds for nothing
- elif drm is not None and scan_budget is None:
- drm.scan_budget = None # one-shot output sees every process
+ elif drm is not None:
+ drm.start() # engine baseline over every process
+ if scan_budget is None:
+ drm.scan_budget = None # one-shot output sees every process
self.inspector = ProcInspector()
self.prober = ServerProber()
self.scan_budget = scan_budget # None = scan every candidate now (one-shot output)
diff --git a/src/gputaskman/diff.py b/src/gputaskman/diff.py
index 31892f3..0c2062f 100644
--- a/src/gputaskman/diff.py
+++ b/src/gputaskman/diff.py
@@ -13,7 +13,8 @@ from .llm import gb
MiB = 1 << 20
NOISE = 16 * MiB # smaller per-process moves are allocator and compositor churn, not a model loading
-MEASURED = ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes")
+MEASURED = ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes",
+ "vram_resident_bytes", "gtt_resident_bytes") # resident: amdgpu, shared buffers included (never summed)
DEVICE = (("vram_used", "dedicated used"), ("shared_used", "shared used"),
("attributed", "attributed to processes"), ("unattributed", "unattributed"))
SEGMENT_ORDER = ("weights", "kv", "compute", "runtime", "unknown")
diff --git a/src/gputaskman/drm.py b/src/gputaskman/drm.py
index 517f1cb..261ec55 100644
--- a/src/gputaskman/drm.py
+++ b/src/gputaskman/drm.py
@@ -72,18 +72,18 @@ def _int(path: str, base: int = 10) -> int | None:
return None
-def parse_size(value: str) -> int:
- """'2060 KiB' -> bytes; a bare number is bytes; anything unparseable is 0."""
+def parse_size(value: str | None) -> int | None:
+ """'2060 KiB' -> bytes; a bare number is bytes; anything unparseable is None (unknown, not 0)."""
parts = (value or "").strip().split()
if not parts:
- return 0
+ return None
try:
number = int(parts[0])
except ValueError:
- return 0
+ return None
unit = parts[1].lower() if len(parts) > 1 else ""
multiplier = _SIZE_UNITS.get(unit)
- return number * multiplier if multiplier is not None else 0
+ return number * multiplier if multiplier is not None else None
def parse_fdinfo(text: str) -> dict | None:
@@ -100,31 +100,34 @@ def parse_fdinfo(text: str) -> dict | None:
except ValueError:
client_id = None
- def memory(resident: str, legacy: str) -> int:
- return parse_size(fields.get(resident, fields.get(legacy, "0")))
+ def memory(resident: str, legacy: str) -> int | None:
+ return parse_size(fields.get(resident, fields.get(legacy)))
- def shared(name: str) -> int | None:
- return parse_size(fields[name]) if name in fields else None
+ # drm-shared-<region> counts buffers with more than one handle by their *preferred* region, resident or not,
+ # while drm-resident-<region> counts by where buffers are now, so the two can't be subtracted. The total is
+ # still exact evidence of whether this client shares any buffer at all. None when a region is missing
+ # (older amdgpu, before the generic DRM memory stats) or unparseable: unknown, not 0.
+ shared_keys = [k for k in ("drm-shared-vram", "drm-shared-gtt") if k in fields]
+ shared_values = [parse_size(fields[k]) for k in shared_keys]
+ # Only VRAM and GTT matter here: drm-shared-cpu counts system-memory buffers this tool doesn't charge.
+ shared = sum(shared_values) if len(shared_keys) == 2 and None not in shared_values else None
- engines: dict[str, int] = {}
+ engines: dict[str, int] | None = {}
for key, value in fields.items():
if not key.startswith("drm-engine-") or key.startswith("drm-engine-capacity-"):
continue
- token = value.split()[0] if value.split() else "0"
try:
- engines[key[len("drm-engine-"):]] = int(token)
- except ValueError:
- engines[key[len("drm-engine-"):]] = 0
+ engines[key[len("drm-engine-"):]] = int(value.split()[0])
+ except (ValueError, IndexError):
+ engines = None # a torn or malformed counter: this client's load is unknown, not 0
+ break
return {
"driver": fields["drm-driver"],
"client_id": client_id,
"pdev": fields.get("drm-pdev", ""),
"vram": memory("drm-resident-vram", "drm-memory-vram"),
"gtt": memory("drm-resident-gtt", "drm-memory-gtt"),
- # Buffers shared with other clients, also counted by every importer. None on older amdgpu
- # (before the generic DRM memory stats), whose fdinfo can't split private from shared.
- "shared_vram": shared("drm-shared-vram"),
- "shared_gtt": shared("drm-shared-gtt"),
+ "shared": shared,
"cpu": memory("drm-resident-cpu", "drm-memory-cpu"),
"engines": engines,
}
@@ -333,8 +336,10 @@ def read_device(card: DrmCard, index: int) -> DeviceTelemetry:
if hwmon:
dev.temp = _hwmon_temp(hwmon)
power = _microwatts(os.path.join(hwmon, "power1_average"))
- if power is None:
- power = _microwatts(os.path.join(hwmon, "power1_input"))
+ if not power: # some parts expose power1_average but leave it at 0
+ input_power = _microwatts(os.path.join(hwmon, "power1_input"))
+ if input_power is not None:
+ power = input_power
dev.power_w = power
if power is not None:
dev.power_kind = _read(os.path.join(hwmon, "power1_label")) or "" # amdgpu: "PPT" (package power)
@@ -384,8 +389,23 @@ def _engine_key(name: str, unified_video: bool = False) -> str:
return "other" # vpe, ...: busy time still counts toward GPU%, like WDDM's unnamed engines
+_DEFERRED = object() # _scan_pid: the scan budget ran out partway through this process's fds
+_PRUNE_NS = 60 * 1_000_000_000 # forget engine baselines for clients not seen for a minute
+
+# Why a row's DRM memory is n/a instead of measured (ProcGpu.drm_reason)
+REASON_LEGACY = "legacy" # older amdgpu: no drm-shared-vram/gtt, so private and shared bytes can't be told apart
+REASON_SHARED_CLIENT = "shared_client" # one DRM client held by several processes (fd passing, fork)
+REASON_SHARED_BUFFERS = "shared_buffers" # the client holds buffers shared with other clients (dma-buf)
+REASON_PRIMARY = "primary" # opened via /dev/dri/card*, whose root-held copies (logind) a normal user can't see
+REASON_INCOMPLETE = "incomplete" # some of its fds or fdinfo couldn't be read
+REASON_PARTIAL_SCAN = "partial_scan" # the budget ran out before every process was listed: a holder may be missed
+
+
class DrmCounters:
- """Same interface as pdh.GpuCounters: sample() -> ({adapter key: {pid: ProcGpu}}, {adapter key: AdapterGpu})."""
+ """Same interface as pdh.GpuCounters: sample() -> ({adapter key: {pid: ProcGpu}}, {adapter key: AdapterGpu}).
+
+ Call start() once the card is selected: engine load is a rate, so it takes the first (unbudgeted) baseline.
+ """
def __init__(self, found: list[DrmCard] | None = None, proc: str = "/proc",
clock=time.monotonic_ns, timer=time.perf_counter) -> None:
@@ -393,16 +413,22 @@ class DrmCounters:
self.ok = bool(self.cards)
self._proc = proc
self._clock = clock
- self._prev: dict[tuple, dict[str, int]] = {}
- self._prev_ts: int | None = None
self._timer = timer
- self._known: dict[int, tuple[list[str], bool]] = {}
+ self._prev: dict[tuple, tuple[dict[str, int], int]] = {} # client -> (high-water engine ns, read time)
+ # pid -> (start time, [(fd name, link target)], holds /dev/kfd, every fd and link was readable)
+ self._known: dict[int, tuple[str, list[tuple[str, str]], bool, bool]] = {}
self._cursor = 0
self.status = "" # what the last scan couldn't cover, for the UI and JSON; empty when it saw everything
- self.scan_budget: float | None = None # the baseline below covers every process
- if self.ok:
- self.sample() # engine load is a rate: take the baseline now, like the WDDM counters do
- self.scan_budget = SCAN_BUDGET
+ self.partial = False # the last scan ran out of budget, so a DRM client's holders may not all be known
+ self.scan_budget: float | None = SCAN_BUDGET
+
+ def start(self) -> None:
+ """Take the engine baseline over every process, like the WDDM counters do at startup."""
+ if not self.ok:
+ return
+ budget, self.scan_budget = self.scan_budget, None
+ self.sample()
+ self.scan_budget = budget
def _adapters(self) -> dict[int, AdapterGpu]:
out: dict[int, AdapterGpu] = {}
@@ -415,142 +441,223 @@ class DrmCounters:
)
return out
- def _scan_pid(self, pid: int) -> tuple[list[str], bool] | None:
- """(this process's /dev/dri fd names, whether it holds /dev/kfd); None when its fds can't be inspected."""
- dri: list[str] = []
+ def _start_time(self, pid: int) -> str | None:
+ """Field 22 of /proc/<pid>/stat: tells a reused pid from the process cached under it."""
+ try:
+ with open(os.path.join(self._proc, str(pid), "stat"), encoding="utf-8", errors="replace") as f:
+ fields = f.read().rsplit(")", 1)[1].split()
+ return fields[19]
+ except (OSError, IndexError):
+ return None
+
+ def _scan_pid(self, pid: int, deadline: float | None):
+ """(start time, [(fd name, target)] for /dev/dri, holds /dev/kfd, complete); None when its fds can't be
+ listed; _DEFERRED when the budget ran out partway through."""
+ start = self._start_time(pid)
+ dri: list[tuple[str, str]] = []
holds_kfd = False
+ complete = True
try:
fds = os.scandir(os.path.join(self._proc, str(pid), "fd"))
except PermissionError:
return None
except OSError:
- return dri, holds_kfd # gone since /proc was listed
+ return start, dri, holds_kfd, complete # gone since /proc was listed
with fds:
- for fd in fds:
+ for n, fd in enumerate(fds, 1):
+ if deadline is not None and n % 64 == 0 and self._timer() >= deadline:
+ return _DEFERRED
try:
target = os.readlink(fd.path)
- except OSError:
+ except PermissionError:
+ complete = False
continue
+ except OSError:
+ continue # closed since the listing
if target == "/dev/kfd":
holds_kfd = True
elif target.startswith("/dev/dri/"):
- dri.append(fd.name)
- return dri, holds_kfd
+ dri.append((fd.name, target))
+ return start, dri, holds_kfd, complete
- def _clients(self) -> tuple[dict[tuple, tuple[set[int], dict]], set[int]]:
- """One entry per DRM client: (the pids holding it, fdinfo); plus the pids holding /dev/kfd.
+ def _clients(self):
+ """One entry per DRM client: [pids holding it, fdinfo, via a render node, read time]; plus the pids holding
+ /dev/kfd and the pids whose fds or fdinfo couldn't all be read.
- Listing a process's fds is the costly part, so it is bounded by scan_budget, resuming where the last refresh
- stopped. Past the budget a process keeps its last known fds, whose fdinfo is still read: rows and counters
- stay current, and only a GPU client opened since the last pass can show up a refresh or two late.
+ Listing a process's fds is the costly part, so it is bounded by scan_budget: processes not seen before go
+ first, then the rest resume where the last refresh stopped. Past the budget a process keeps its last known
+ fds (dropped if its pid was reused), whose fdinfo is still read: rows and counters stay current, and only an
+ fd handed to an already-running process shows up a refresh or two late.
"""
by_pdev = {c.pdev: c.key for c in self.cards}
- clients: dict[tuple, tuple[set[int], dict]] = {}
+ clients: dict[tuple, list] = {}
kfd: set[int] = set()
+ incomplete: set[int] = set()
self.status = ""
+ self.partial = False
try:
with os.scandir(self._proc) as entries:
pids = sorted(int(e.name) for e in entries if e.name.isdigit())
except OSError:
self.status = f"{self._proc} is not readable"
- return clients, kfd
+ return clients, kfd, incomplete
start = bisect_left(pids, self._cursor)
+ ordered = pids[start:] + pids[:start]
+ ordered = [p for p in ordered if p not in self._known] + [p for p in ordered if p in self._known]
deadline = None if self.scan_budget is None else self._timer() + self.scan_budget
- known: dict[int, tuple[list[str], bool]] = {}
- denied = skipped = 0
- self._cursor = 0
- for pid in pids[start:] + pids[:start]:
- if deadline is not None and (skipped or self._timer() >= deadline):
- if not skipped:
- self._cursor = pid
- skipped += 1
- entry = self._known.get(pid)
- else:
- entry = self._scan_pid(pid)
- denied += entry is None
- if entry is None:
- continue
+ known: dict[int, tuple] = {}
+ denied = deferred = 0
+ out_of_time = False
+ cursor = None # first known process not rescanned: where the rotation resumes (new ones go first anyway)
+ scanned = 0
+ for pid in ordered:
+ entry = None
+ if not out_of_time and (deadline is None or self._timer() < deadline):
+ # The first process of a refresh is always listed in full, so one huge fd table can't stall the rotation.
+ entry = self._scan_pid(pid, deadline if scanned else None)
+ scanned += 1
+ if entry is None:
+ denied += 1
+ continue
+ if entry is _DEFERRED:
+ entry = None
+ if entry is None: # not scanned this refresh: reuse the cached fds if this is still the same process
+ out_of_time = True
+ if cursor is None and pid in self._known:
+ cursor = pid
+ deferred += 1
+ cached = self._known.get(pid)
+ if cached is None or cached[0] != self._start_time(pid):
+ continue
+ entry = cached
known[pid] = entry
- if entry[1]:
+ _start, dri, holds_kfd, complete = entry
+ if holds_kfd:
kfd.add(pid)
- for name in entry[0]:
+ if not complete:
+ incomplete.add(pid)
+ for name, target in dri:
try:
with open(os.path.join(self._proc, str(pid), "fdinfo", name),
encoding="utf-8", errors="replace") as f:
- info = parse_fdinfo(f.read())
- except OSError:
+ text = f.read()
+ except PermissionError:
+ incomplete.add(pid)
continue
+ except OSError:
+ continue # closed since the listing
+ read_at = self._clock()
+ info = parse_fdinfo(text)
if not info or info["driver"] not in DRIVERS or info["pdev"] not in by_pdev:
continue
client_id = info["client_id"]
key = (info["pdev"], client_id) if client_id is not None else (info["pdev"], pid, name)
- clients.setdefault(key, (set(), info))[0].add(pid)
+ client = clients.setdefault(key, [set(), info, True, read_at])
+ client[0].add(pid)
+ client[2] = client[2] and target.startswith("/dev/dri/renderD")
self._known = known
+ self.partial = out_of_time
+ if not out_of_time:
+ self._cursor = 0
+ elif cursor is not None:
+ self._cursor = cursor # else only new processes were deferred: the known ones keep their place
notes = []
- if skipped:
- notes.append(f"scan budget reached, {skipped} processes keep their last known GPU fds this refresh")
+ if out_of_time:
+ notes.append(f"scan budget reached, {deferred} processes not rescanned this refresh (last known GPU fds kept)")
if denied:
notes.append(f"{denied} processes not inspectable (other users' or kernel threads; sudo includes them)")
+ if incomplete:
+ notes.append(f"{len(incomplete)} processes whose fds or fdinfo couldn't all be read")
self.status = "; ".join(notes)
- return clients, kfd
+ return clients, kfd, incomplete
def sample(self) -> tuple[dict[int, dict[int, ProcGpu]], dict[int, AdapterGpu]]:
- now = self._clock()
by_pdev = {c.pdev: c.key for c in self.cards}
unified = {c.pdev for c in self.cards if c.unified_video}
procs: dict[int, dict[int, ProcGpu]] = {c.key: {} for c in self.cards}
adapters = self._adapters()
- clients, kfd = self._clients()
- delta = now - self._prev_ts if self._prev_ts is not None else 0
- previous: dict[tuple, dict[str, int]] = {}
- for client_key, (owners, info) in clients.items():
- key = by_pdev[info["pdev"]]
- previous[client_key] = dict(info["engines"])
- rows = []
- for pid in owners:
- # No commit counter in fdinfo, so Paged stays unavailable rather than 0. ROCm/HIP (a /dev/kfd
- # holder) submits compute through user-mode KFD queues the kernel doesn't time per process, so
- # its engine load is unknown, not 0 (AGENTS.md rule 2); its VRAM is still in fdinfo.
- available = {"dedicated", "shared"} if pid in kfd else {"dedicated", "shared", "engine"}
- proc = procs[key].setdefault(pid, ProcGpu(attributed=True, available=available))
- proc.kfd = pid in kfd
- rows.append(proc)
- if len(owners) > 1:
- # One client held by several processes (fd passing, fork): nothing says which of them uses it,
- # so each is listed without counters instead of charging one by guess (rules 1 and 5). Its bytes
- # stay in the adapter's unattributed part, like a shared buffer's, so the other rows still add up.
- for proc in rows:
- proc.attributed = False
- proc.shared_client = True
- proc.available -= {"shared", "engine"}
+ clients, kfd, incomplete = self._clients()
+
+ by_pid: dict[tuple[int, int], list[tuple[tuple, list]]] = {}
+ for client_key, client in clients.items():
+ for pid in client[0]:
+ by_pid.setdefault((by_pdev[client[1]["pdev"]], pid), []).append((client_key, client))
+
+ now = self._clock()
+ current: dict[tuple, tuple[dict[str, int], int]] = {}
+ for client_key, (owners, info, _render, read_at) in clients.items():
+ old = self._prev.get(client_key)
+ if info["engines"] is None:
+ if old is not None:
+ current[client_key] = old
continue
- proc = rows[0]
- # A buffer shared between clients is resident in each of them and fdinfo doesn't say which owns it.
- # Only private bytes are charged (AGENTS.md rule 1); shared ones are reported apart, owner unknown.
- if info["shared_vram"] is None:
- # Older amdgpu: resident bytes include every imported buffer, so summing them would count
- # shared buffers once per importer. Unknown rather than double counted (rules 1 and 9).
+ high = dict(old[0]) if old else {}
+ for name, ns in info["engines"].items():
+ high[name] = max(ns, high.get(name, 0)) # the DRM spec allows a counter to step back briefly
+ current[client_key] = (high, read_at)
+
+ for (key, pid), held in by_pid.items():
+ # No commit counter in fdinfo, so Paged stays unavailable rather than 0. ROCm/HIP (a /dev/kfd holder)
+ # submits compute through user-mode KFD queues the kernel doesn't time per process, so its engine load is
+ # unknown, not 0 (AGENTS.md rule 2); its VRAM is still in fdinfo.
+ proc = ProcGpu(attributed=True, available={"dedicated", "shared", "engine"})
+ proc.kfd = pid in kfd
+ infos = [c[1] for _k, c in held]
+ vram = [i["vram"] for i in infos]
+ gtt = [i["gtt"] for i in infos]
+ # Every buffer this process can reach, shared ones included: a reference, never summed (rule 1).
+ complete = pid not in incomplete # otherwise any sum would be a partial total that looks whole
+ proc.resident_vram = float(sum(vram)) if complete and None not in vram else None
+ proc.resident_gtt = float(sum(gtt)) if complete and None not in gtt else None
+
+ # Bytes are charged to this process only when nothing else can hold them: one visible holder, reached
+ # through a render node, no shared buffers at all, and every value known. Anything else leaves the whole
+ # process's bytes in the adapter's unattributed part instead of charging one by guess (rules 1 and 5).
+ reason = ""
+ if pid in incomplete:
+ reason = REASON_INCOMPLETE
+ elif any(len(c[0]) > 1 for _k, c in held):
+ reason = REASON_SHARED_CLIENT
+ elif any(i["shared"] is None for i in infos):
+ reason = REASON_LEGACY
+ elif any(i["shared"] for i in infos):
+ reason = REASON_SHARED_BUFFERS
+ elif not all(c[2] for _k, c in held):
+ reason = REASON_PRIMARY
+ elif proc.resident_vram is None or proc.resident_gtt is None:
+ reason = REASON_LEGACY
+ elif self.partial:
+ reason = REASON_PARTIAL_SCAN
+ proc.drm_reason = reason
+ if reason:
proc.attributed = False
proc.available -= {"dedicated", "shared"}
else:
- shared = float(info["shared_vram"])
- proc.dedicated += max(float(info["vram"]) - shared, 0.0)
- proc.shared_vram += shared
- proc.shared += max(float(info["gtt"]) - float(info["shared_gtt"] or 0), 0.0)
- old = self._prev.get(client_key)
- if old is None or delta <= 0:
- proc.available.discard("engine") # a rate needs two samples: unknown, not 0 (rule 2)
- continue
+ proc.dedicated = proc.resident_vram
+ proc.shared = proc.resident_gtt
+
engines: dict[str, float] = {}
- for name, ns in info["engines"].items():
- # amdgpu omits an engine until it has run, so a missing earlier value really is 0.
- pct = min(100.0, max(0.0, 100.0 * (ns - old.get(name, 0)) / delta))
- engine = _engine_key(name, info["pdev"] in unified)
- engines[engine] = engines.get(engine, 0.0) + pct
+ for client_key, (owners, info, _render, read_at) in held:
+ old = self._prev.get(client_key)
+ if (proc.kfd or not complete or self.partial or len(owners) > 1 or info["engines"] is None
+ or old is None or read_at - old[1] <= 0):
+ engines = {}
+ proc.available.discard("engine") # unknown, not 0 (rule 2)
+ break
+ delta = read_at - old[1]
+ for name, ns in current[client_key][0].items():
+ # amdgpu omits an engine until it has run, so a missing earlier value really is 0.
+ pct = min(100.0, max(0.0, 100.0 * (ns - old[0].get(name, 0)) / delta))
+ engine = _engine_key(name, info["pdev"] in unified)
+ engines[engine] = min(100.0, engines.get(engine, 0.0) + pct)
for engine, pct in engines.items():
- proc.engines[engine] = min(100.0, proc.engines[engine] + pct)
- self._prev = previous
- self._prev_ts = now
+ proc.engines[engine] = pct
+ procs[key][pid] = proc
+
+ for client_key, value in self._prev.items(): # keep baselines of clients a deferred scan missed, briefly
+ if client_key not in current and now - value[1] < _PRUNE_NS:
+ current[client_key] = value
+ self._prev = current
return procs, adapters
def close(self) -> None:
diff --git a/src/gputaskman/linux.py b/src/gputaskman/linux.py
index a719e83..ef25cee 100644
--- a/src/gputaskman/linux.py
+++ b/src/gputaskman/linux.py
@@ -232,7 +232,7 @@ class LinuxCounters:
if not self.wsl:
from .drm import DrmCounters
- drm = DrmCounters()
+ drm = DrmCounters() # cheap: only lists cards; the Collector starts it if an AMD GPU is selected
self.drm = drm if drm.ok else None
if N is None:
return
diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.py
index 392bf84..adf54fe 100644
--- a/src/gputaskman/pdh.py
+++ b/src/gputaskman/pdh.py
@@ -84,11 +84,12 @@ class ProcGpu:
# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.
inferred: float | None = None
inferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)
- # amdgpu (DRM fdinfo): resident VRAM in buffers shared with other DRM clients, kept out of `dedicated`. fdinfo
- # can't say which client owns a shared buffer, so it is shown apart and left in the adapter's unattributed part.
- shared_vram: float = 0.0
+ # amdgpu (DRM fdinfo): resident bytes in every buffer this process can reach, shared ones included. A reference
+ # for display only; the same buffer can appear under several processes, so these are never summed. None: unknown.
+ resident_vram: float | None = None
+ resident_gtt: float | None = None
+ drm_reason: str = "" # amdgpu: why dedicated/shared are n/a (drm.REASON_*); empty when they are measured
kfd: bool = False # amdgpu: holds /dev/kfd (ROCm/HIP), whose compute isn't timed per process
- shared_client: bool = False # amdgpu: holds a DRM client other processes hold too, so its counters have no one owner
@property
def shown_vram(self) -> float | None:
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index 5ea9117..c3eb54a 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -58,13 +58,26 @@ ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer
"gputaskman runs.")
-DRM_UNSPLIT_NOTE = ("VRAM n/a: this kernel's amdgpu fdinfo has no drm-shared-* keys, so resident memory includes "
- "every buffer imported from other processes and would be counted once per importer. Newer "
- "kernels report shared buffers separately.")
-
-
-DRM_SHARED_CLIENT_NOTE = ("VRAM and GPU% n/a: this process holds a DRM client other processes hold too (fd passing, "
- "fork). fdinfo counts the client once and doesn't say which of them uses it.")
+DRM_NOTES = {
+ "legacy": ("VRAM n/a: this kernel's amdgpu fdinfo has no drm-shared-* keys, so resident memory may include "
+ "buffers imported from other processes and would be counted once per importer."),
+ "shared_client": ("VRAM and GPU% n/a: this process holds a DRM client other processes hold too (fd passing, "
+ "fork). fdinfo counts the client once and doesn't say which of them uses it."),
+ "shared_buffers": ("VRAM n/a: this process's DRM client shares buffers with other clients (dma-buf, e.g. "
+ "windows and video frames). fdinfo counts shared buffers by where they prefer to live, not "
+ "where they are, so its private part can't be measured."),
+ "primary": ("VRAM n/a: this process opened the GPU through /dev/dri/card*, which root-owned services such as "
+ "logind may also hold where a normal user can't see them; run as root to check."),
+ "incomplete": "VRAM and GPU% n/a: some of this process's file descriptors or fdinfo couldn't be read.",
+ "partial_scan": ("VRAM and GPU% n/a this refresh: the scan budget ran out before every process's fds were "
+ "listed, so another holder of this DRM client can't be ruled out."),
+}
+DRM_RESIDENT_NOTE = ("Resident: every buffer the process can reach, shared ones included. The same buffer can "
+ "appear under several processes, so these are not added up.")
+
+
+def drm_note(g: ProcGpu) -> str:
+ return DRM_NOTES.get(g.drm_reason, "")
def inferred_note(g) -> str:
@@ -87,7 +100,11 @@ def to_dict(s: Snapshot) -> dict:
"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,
"vram_inferred_by": p.gpu.inferred_by or None,
"vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed and "dedicated" in p.gpu.available else None,
- "vram_shared_bytes": int(p.gpu.shared_vram) if s.backend == "drm" and not p.host else None,
+ "vram_resident_bytes": (int(p.gpu.resident_vram) if s.backend == "drm" and not p.host
+ and p.gpu.resident_vram is not None else None),
+ "gtt_resident_bytes": (int(p.gpu.resident_gtt) if s.backend == "drm" and not p.host
+ and p.gpu.resident_gtt is not None else None),
+ "vram_unattributed_reason": p.gpu.drm_reason or None if s.backend == "drm" and not p.host else None,
"shared_bytes": int(p.gpu.shared) if has_spill_counters(s, p) else None,
"committed_bytes": int(p.gpu.committed) if wddm and "committed" in p.gpu.available else None,
"paged_out_bytes": int(p.gpu.paged_out) if wddm and p.gpu.paged_available else None,
@@ -120,7 +137,8 @@ def to_dict(s: Snapshot) -> dict:
"taken": s.taken, "gpu": s.adapter.name if s.adapter else None, "driver": s.driver, "cuda": s.cuda,
"adapter": dataclasses.asdict(s.adapter) if s.adapter else None,
"vram_total": s.vram_total, "vram_used": s.vram_used, "shared_used": s.shared_used,
- "attributed": s.attributed, "backend": s.backend, "host_status": s.host_status or None,
+ "attributed": s.attributed, "backend": s.backend, "shared_is_spill": s.shared_is_spill,
+ "host_status": s.host_status or None,
"process_status": s.process_status or None,
"wsl_split": s.wsl_split or None, "device": nv, "processes": procs,
}
@@ -173,7 +191,7 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
wddm = has_wddm_counters(s, p)
spill = gb(p.gpu.shared) if has_spill_counters(s, p) and p.gpu.shared >= 1 << 20 else ""
paged = gb(p.gpu.paged_out) if wddm and p.gpu.paged_out >= 1 << 20 else ""
- if wddm and "shared" not in p.gpu.available:
+ if (wddm or (s.backend == "drm" and not p.host)) and "shared" not in p.gpu.available:
spill = "N/A"
if wddm and not p.gpu.paged_available:
paged = "N/A"
@@ -185,10 +203,10 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
out.append(INFERRED_NOTE)
if any(p.gpu.inferred_by == "order" for p in shown):
out.append(ORDER_NOTE)
- if s.backend == "drm" and any(not p.host and not p.gpu.attributed and not p.gpu.shared_client for p in shown):
- out.append(DRM_UNSPLIT_NOTE)
- if s.backend == "drm" and any(not p.host and p.gpu.shared_client for p in shown):
- out.append(DRM_SHARED_CLIENT_NOTE)
+ if s.backend == "drm":
+ for note in dict.fromkeys(drm_note(p.gpu) for p in shown if not p.host):
+ if note:
+ out.append(note)
if s.process_status:
out.append(f"DRM process scan: {s.process_status}.")
if s.backend == "wsl" and s.wsl_split and any(not p.host and not p.gpu.attributed and p.gpu.inferred is None
diff --git a/tests/test_drm.py b/tests/test_drm.py
index 923896c..1a4baaa 100644
--- a/tests/test_drm.py
+++ b/tests/test_drm.py
@@ -79,21 +79,36 @@ def _write_fd(proc: Path, pid, fd, text, target="/dev/dri/renderD128") -> None:
os.symlink(target, link)
-def _fdinfo(client, pdev, gfx=0, vram=0, gtt=0, shared_vram=0) -> str:
+def _fdinfo(client, pdev, gfx=0, vram=0, gtt=0, shared_vram=0, shared_gtt=0) -> str:
return (
"pos:\t0\n"
"drm-driver:\tamdgpu\n"
f"drm-client-id:\t{client}\n"
f"drm-pdev:\t{pdev}\n"
- f"drm-memory-vram:\t{vram} KiB\n"
f"drm-shared-vram:\t{shared_vram} KiB\n"
- f"drm-memory-gtt: \t{gtt} KiB\n"
- "drm-memory-cpu: \t0 KiB\n"
+ f"drm-resident-vram:\t{vram} KiB\n"
+ f"drm-shared-gtt:\t{shared_gtt} KiB\n"
+ f"drm-resident-gtt:\t{gtt} KiB\n"
+ "drm-shared-cpu:\t0\n"
+ "drm-resident-cpu:\t0\n"
f"drm-engine-gfx:\t{gfx} ns\n"
f"drm-engine-capacity-gfx:\t1\n"
)
+class Clock:
+ """A monotonic_ns stand-in the test advances by hand: every fdinfo read in one refresh sees the same time."""
+
+ def __init__(self) -> None:
+ self.t = 0
+
+ def __call__(self) -> int:
+ return self.t
+
+ def tick(self, seconds: float = 1.0) -> None:
+ self.t += int(seconds * 1_000_000_000)
+
+
class ParseTests(unittest.TestCase):
def test_parse_size(self):
self.assertEqual(drm.parse_size("2060 KiB"), 2060 * 1024)
@@ -103,8 +118,9 @@ class ParseTests(unittest.TestCase):
self.assertEqual(drm.parse_size("0"), 0)
self.assertEqual(drm.parse_size("512"), 512)
self.assertEqual(drm.parse_size(" 4 KiB "), 4096)
- self.assertEqual(drm.parse_size("bogus"), 0)
- self.assertEqual(drm.parse_size("12 bananas"), 0)
+ self.assertIsNone(drm.parse_size("bogus")) # unknown, not 0
+ self.assertIsNone(drm.parse_size("12 bananas"))
+ self.assertIsNone(drm.parse_size(None))
def test_parse_fdinfo_example(self):
info = drm.parse_fdinfo(FDINFO)
@@ -133,19 +149,33 @@ class ParseTests(unittest.TestCase):
self.assertEqual(info["cpu"], 0)
self.assertEqual(info["engines"], {})
self.assertIsNone(info["client_id"])
- self.assertIsNone(info["shared_vram"]) # no drm-shared-* keys: unknown, not 0
+ self.assertIsNone(info["shared"]) # no drm-shared-* keys: unknown, not 0
def test_parse_fdinfo_prefers_resident_keys(self):
info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-resident-vram:\t8 KiB\ndrm-memory-vram:\t9 KiB\n"
"drm-shared-vram:\t0\ndrm-shared-gtt:\t4 KiB\n")
self.assertEqual(info["vram"], 8 * 1024)
- self.assertEqual(info["shared_vram"], 0)
- self.assertEqual(info["shared_gtt"], 4 * 1024)
+ self.assertEqual(info["shared"], 4 * 1024)
+
+ def test_parse_fdinfo_shared_needs_vram_and_gtt(self):
+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-memory-vram:\t300 MiB\ndrm-shared-vram:\t0\n")
+ self.assertIsNone(info["shared"]) # no drm-shared-gtt: whether GTT buffers are shared is unknown
+
+ def test_parse_fdinfo_shared_cpu_is_ignored(self):
+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-shared-vram:\t0\ndrm-shared-gtt:\t0\n"
+ "drm-shared-cpu:\t64 KiB\n")
+ self.assertEqual(info["shared"], 0) # a shared system-memory buffer doesn't make VRAM/GTT unprivate
+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-shared-vram:\t0\ndrm-shared-gtt:\t0\n"
+ "drm-shared-cpu:\tlots\n")
+ self.assertEqual(info["shared"], 0)
- def test_parse_fdinfo_shared_vram(self):
- info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-memory-vram:\t300 MiB\ndrm-shared-vram:\t200 MiB\n")
- self.assertEqual(info["vram"], 300 * 1024**2)
- self.assertEqual(info["shared_vram"], 200 * 1024**2) # part of vram, not on top of it
+ def test_parse_fdinfo_unparseable_is_unknown(self):
+ info = drm.parse_fdinfo("drm-driver:\tamdgpu\ndrm-resident-vram:\tlots\ndrm-shared-vram:\t0\n"
+ "drm-shared-gtt:\t?\ndrm-engine-gfx:\ttorn\n")
+ self.assertIsNone(info["vram"])
+ self.assertIsNone(info["gtt"]) # missing entirely
+ self.assertIsNone(info["shared"])
+ self.assertIsNone(info["engines"])
PCI_IDS = """# fake
@@ -261,13 +291,15 @@ class DrmCountersTests(unittest.TestCase):
card = _card(dev, pdev)
proc = tmp / "proc"
- _write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_000_000_000, vram=3, gtt=2, shared_vram=2))
+ _write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_000_000_000, vram=3, gtt=2))
_write_fd(proc, 300, 12, _fdinfo(27, "0000:ff:00.0", gfx=9_999_999_999, vram=4))
# DRM-looking fdinfo behind a non-DRM fd: the /proc/<pid>/fd target rules it out.
_write_fd(proc, 400, 13, _fdinfo(28, pdev, gfx=9_999_999_999, vram=4), target="socket:[123]")
- times = iter([0, 1_000_000_000, 2_000_000_000]) # the constructor takes the baseline at 0
- counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))
+ clock = Clock()
+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=clock)
+ counters.start()
+ clock.tick()
self.assertTrue(counters.ok)
procs, adapters = counters.sample()
@@ -275,24 +307,34 @@ class DrmCountersTests(unittest.TestCase):
self.assertIn(100, procs[card.key])
self.assertNotIn(300, procs[card.key]) # unknown pdev
self.assertNotIn(400, procs[card.key]) # non-DRM fd
- self.assertEqual(procs[card.key][100].dedicated, 1024) # private only: 3 KiB resident - 2 KiB shared
- self.assertEqual(procs[card.key][100].shared_vram, 2048)
- self.assertFalse(procs[card.key][100].paged_available) # no commit counter: unavailable, not 0
- self.assertEqual(procs[card.key][100].shared, 2048)
- self.assertEqual(procs[card.key][100].committed, 0)
- self.assertEqual(procs[card.key][100].engines.get("3d", 0.0), 0.0) # idle since the baseline
+ g = procs[card.key][100]
+ self.assertEqual(g.dedicated, 3 * 1024) # private client, no shared buffers: resident is its own
+ self.assertEqual(g.resident_vram, 3 * 1024)
+ self.assertEqual(g.shared, 2048)
+ self.assertEqual(g.drm_reason, "")
+ self.assertFalse(g.paged_available) # no commit counter: unavailable, not 0
+ self.assertEqual(g.committed, 0)
+ self.assertEqual(g.engines.get("3d", 0.0), 0.0) # idle since the baseline
self.assertEqual(adapters[card.key].dedicated, 1 * GiB)
self.assertEqual(adapters[card.key].shared, 64 * 1024)
(dev / "mem_info_vram_used").write_text(str(2 * GiB))
_write_fd(proc, 100, 10, _fdinfo(26, pdev, gfx=1_500_000_000, vram=1, gtt=2))
-
+ clock.tick()
procs, adapters = counters.sample()
self.assertEqual(procs[card.key][100].engines["3d"], 50.0)
self.assertIn("engine", procs[card.key][100].available)
self.assertEqual(adapters[card.key].dedicated, 2 * GiB)
counters.close()
+ def test_constructor_does_not_scan(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ dev = Path(tmp) / "drm" / "card1" / "device"
+ dev.mkdir(parents=True)
+ with patch.object(drm.DrmCounters, "sample") as sample:
+ drm.DrmCounters(found=[_card(dev, "0000:03:00.0")], proc=str(Path(tmp) / "proc"))
+ sample.assert_not_called() # an NVIDIA GPU may be selected: no fd walk until start()
+
def test_rocm_process_engine_load_is_unknown(self):
# A HIP process (llama.cpp-ROCm) holds /dev/kfd next to its render node: its VRAM is in fdinfo, but
# its compute runs on KFD user queues with no per-process busy time, so engines are unavailable.
@@ -306,10 +348,13 @@ class DrmCountersTests(unittest.TestCase):
_write_fd(proc, 100, 3, "pos:\t0\nflags:\t02\n", target="/dev/kfd")
_write_fd(proc, 100, 7, _fdinfo(500, pdev, vram=3891436))
_write_fd(proc, 200, 7, _fdinfo(501, pdev, gfx=5, vram=4)) # a graphics client, for contrast
- times = iter([0, 1_000_000_000])
- counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))
+ clock = Clock()
+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=clock)
+ counters.start()
+ clock.tick()
procs, _ = counters.sample()
self.assertEqual(procs[card.key][100].dedicated, 3891436 * 1024)
+ self.assertTrue(procs[card.key][100].kfd)
self.assertNotIn("engine", procs[card.key][100].available)
self.assertIn("engine", procs[card.key][200].available)
@@ -325,31 +370,79 @@ class DrmEdgeCaseTests(unittest.TestCase):
card = _card(dev, pdev)
card.unified_video = unified_video
proc = tmp / "proc"
- for pid, fd, text in fds:
- _write_fd(proc, pid, fd, text.replace("PDEV", pdev))
- times = iter([0, 1_000_000_000, 2_000_000_000])
- return card, proc, drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times))
+ for fd_spec in fds:
+ pid, fd, text, *target = fd_spec
+ _write_fd(proc, pid, fd, text.replace("PDEV", pdev), *target)
+ self.clock = Clock()
+ counters = drm.DrmCounters(found=[card], proc=str(proc), clock=self.clock)
+ counters.start()
+ self.clock.tick()
+ return card, proc, counters
def test_old_kernel_without_shared_keys_is_not_summed(self):
old = "drm-driver:\tamdgpu\ndrm-client-id:\t5\ndrm-pdev:\tPDEV\ndrm-memory-vram:\t4096 KiB\n"
with tempfile.TemporaryDirectory() as tmp:
card, _, counters = self._counters(Path(tmp), [(100, 5, old)])
g = counters.sample()[0][card.key][100]
- self.assertFalse(g.attributed) # resident includes imports: would double count
+ self.assertFalse(g.attributed) # resident may include imports: would double count
+ self.assertEqual(g.drm_reason, drm.REASON_LEGACY)
self.assertEqual(g.dedicated, 0)
self.assertNotIn("dedicated", g.available)
+ self.assertNotIn("shared", g.available)
+ self.assertEqual(g.resident_vram, 4096 * 1024) # still shown as a reference
+
+ def test_any_shared_buffer_leaves_the_process_unattributed(self):
+ # drm-shared-* counts by preferred placement and drm-resident-* by current placement, so resident minus
+ # shared is not this process's private memory. Any sharing at all: n/a, never a subtraction.
+ for extra in ({"shared_vram": 2}, {"shared_gtt": 4}):
+ with self.subTest(extra), tempfile.TemporaryDirectory() as tmp:
+ card, _, counters = self._counters(Path(tmp), [(100, 5, _fdinfo(7, "PDEV", vram=3, gtt=10, **extra))])
+ g = counters.sample()[0][card.key][100]
+ self.assertEqual(g.drm_reason, drm.REASON_SHARED_BUFFERS)
+ self.assertIsNone(g.shown_vram)
+ self.assertNotIn("shared", g.available)
+ self.assertEqual(g.resident_vram, 3 * 1024)
+ self.assertEqual(g.resident_gtt, 10 * 1024)
+ self.assertIn("engine", g.available) # sharing buffers doesn't share the client's engine time
+
+ def test_primary_node_is_not_charged(self):
+ # logind (root) holds a copy of a compositor's /dev/dri/card* fd that a normal user can't see.
+ with tempfile.TemporaryDirectory() as tmp:
+ card, _, counters = self._counters(Path(tmp), [(100, 5, _fdinfo(7, "PDEV", vram=3), "/dev/dri/card1")])
+ g = counters.sample()[0][card.key][100]
+ self.assertEqual(g.drm_reason, drm.REASON_PRIMARY)
+ self.assertIsNone(g.shown_vram)
- def test_shared_gtt_left_out_of_spill(self):
- text = _fdinfo(7, "PDEV", gtt=10) + "drm-shared-gtt:\t4 KiB\n"
+ def test_unparseable_memory_is_unknown(self):
+ text = _fdinfo(7, "PDEV", vram=3).replace("drm-resident-vram:\t3 KiB", "drm-resident-vram:\tlots")
with tempfile.TemporaryDirectory() as tmp:
card, _, counters = self._counters(Path(tmp), [(100, 5, text)])
- self.assertEqual(counters.sample()[0][card.key][100].shared, 6 * 1024)
+ g = counters.sample()[0][card.key][100]
+ self.assertIsNone(g.resident_vram)
+ self.assertIsNone(g.shown_vram)
+
+ def test_torn_engine_counter_is_unknown(self):
+ text = _fdinfo(7, "PDEV").replace("drm-engine-gfx:\t0 ns", "drm-engine-gfx:\t")
+ with tempfile.TemporaryDirectory() as tmp:
+ card, _, counters = self._counters(Path(tmp), [(100, 5, text)])
+ self.assertNotIn("engine", counters.sample()[0][card.key][100].available)
+
+ def test_engine_counter_stepping_back_is_not_counted_twice(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, counters = self._counters(Path(tmp), [(100, 5, _fdinfo(7, "PDEV", gfx=1_000_000_000))])
+ path = proc / "100" / "fdinfo" / "5"
+ path.write_text(_fdinfo(7, card.pdev, gfx=900_000_000)) # the DRM spec allows a brief step back
+ self.assertEqual(counters.sample()[0][card.key][100].engines.get("3d", 0.0), 0.0)
+ self.clock.tick()
+ path.write_text(_fdinfo(7, card.pdev, gfx=1_500_000_000))
+ self.assertEqual(counters.sample()[0][card.key][100].engines["3d"], 50.0) # from the high-water mark
def test_unified_vcn_reports_video(self):
with tempfile.TemporaryDirectory() as tmp:
- text = "drm-driver:\tamdgpu\ndrm-client-id:\t9\ndrm-pdev:\tPDEV\ndrm-shared-vram:\t0\ndrm-engine-enc:\t{} ns\n"
- card, proc, counters = self._counters(Path(tmp), [(100, 5, text.format(0))], unified_video=True)
- (proc / "100" / "fdinfo" / "5").write_text(text.format(1_000_000_000).replace("PDEV", card.pdev))
+ card, proc, counters = self._counters(Path(tmp), [(100, 5, _fdinfo(9, "PDEV").replace(
+ "drm-engine-gfx:\t0 ns", "drm-engine-enc:\t0 ns"))], unified_video=True)
+ (proc / "100" / "fdinfo" / "5").write_text(_fdinfo(9, card.pdev).replace(
+ "drm-engine-gfx:\t0 ns", "drm-engine-enc:\t1000000000 ns"))
engines = counters.sample()[0][card.key][100].engines
self.assertEqual(engines["video"], 100.0)
self.assertNotIn("encode", engines)
@@ -362,6 +455,18 @@ class DrmEdgeCaseTests(unittest.TestCase):
self.assertIsNone(a.shared)
self.assertIsNone(drm.read_device(card, 0).mem_used)
+ def test_power_average_of_zero_falls_back_to_input(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ dev = Path(tmp) / "device"
+ hwmon = dev / "hwmon" / "hwmon0"
+ hwmon.mkdir(parents=True)
+ (hwmon / "power1_average").write_text("0\n")
+ (hwmon / "power1_input").write_text("42000000\n")
+ self.assertEqual(drm.read_device(_card(dev, "0000:03:00.0"), 0).power_w, 42.0)
+ (hwmon / "power1_average").unlink()
+ (hwmon / "power1_input").write_text("0\n")
+ self.assertEqual(drm.read_device(_card(dev, "0000:03:00.0"), 0).power_w, 0.0) # measured 0 stays 0
+
def test_vcn_major_from_ip_discovery(self):
with tempfile.TemporaryDirectory() as tmp:
uvd = Path(tmp) / "ip_discovery" / "die" / "0" / "UVD" / "0"
@@ -380,32 +485,38 @@ class DrmScanTests(unittest.TestCase):
card = _card(dev, self.PDEV)
proc = tmp / "proc"
proc.mkdir()
- return card, proc, {} if timer is None else {"timer": timer}
+ self.clock = Clock()
+ kw = {"clock": self.clock}
+ if timer is not None:
+ kw["timer"] = timer
+ return card, proc, kw
def test_new_client_engine_load_is_unknown_until_two_samples(self):
with tempfile.TemporaryDirectory() as tmp:
card, proc, kw = self._setup(Path(tmp))
_write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, gfx=0, vram=4096))
- times = iter([0, 1_000_000_000, 2_000_000_000, 3_000_000_000])
- counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times), **kw)
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start()
+ self.clock.tick()
_write_fd(proc, 200, 5, _fdinfo(2, self.PDEV, gfx=500_000_000, vram=4096))
procs = counters.sample()[0][card.key]
self.assertIn("engine", procs[100].available) # in the baseline: a real, measured idle
self.assertNotIn("engine", procs[200].available) # first seen now: unknown, not 0
self.assertEqual(procs[200].dedicated, 4096 * 1024) # memory doesn't need two samples
self.assertFalse(procs[200].kfd)
+ self.clock.tick()
_write_fd(proc, 200, 5, _fdinfo(2, self.PDEV, gfx=1_000_000_000, vram=4096))
self.assertEqual(counters.sample()[0][card.key][200].engines["3d"], 50.0)
def test_engine_first_reported_counts_from_zero(self):
# amdgpu omits an engine until it has run, so its whole counter is new work.
- text = "drm-driver:\tamdgpu\ndrm-client-id:\t1\ndrm-pdev:\t{}\ndrm-shared-vram:\t0\n{}"
with tempfile.TemporaryDirectory() as tmp:
card, proc, kw = self._setup(Path(tmp))
- _write_fd(proc, 100, 5, text.format(self.PDEV, ""))
- times = iter([0, 1_000_000_000])
- counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times), **kw)
- _write_fd(proc, 100, 5, text.format(self.PDEV, "drm-engine-gfx:\t250000000 ns\n"))
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV).replace("drm-engine-gfx:\t0 ns\n", ""))
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start()
+ self.clock.tick()
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, gfx=250_000_000))
self.assertEqual(counters.sample()[0][card.key][100].engines["3d"], 25.0)
def test_client_held_by_several_processes_is_charged_to_none(self):
@@ -414,17 +525,34 @@ class DrmScanTests(unittest.TestCase):
_write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, gfx=0, vram=4096))
_write_fd(proc, 200, 9, _fdinfo(1, self.PDEV, gfx=0, vram=4096))
_write_fd(proc, 300, 5, _fdinfo(2, self.PDEV, gfx=0, vram=8192))
- times = iter([0, 1_000_000_000])
- counters = drm.DrmCounters(found=[card], proc=str(proc), clock=lambda: next(times), **kw)
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start()
+ self.clock.tick()
procs = counters.sample()[0][card.key]
for pid in (100, 200):
- self.assertTrue(procs[pid].shared_client)
+ self.assertEqual(procs[pid].drm_reason, drm.REASON_SHARED_CLIENT)
self.assertIsNone(procs[pid].shown_vram)
self.assertNotIn("engine", procs[pid].available)
self.assertEqual(procs[300].shown_vram, 8192 * 1024)
# The other rows still add up: the shared client's bytes are left out, not the whole total.
self.assertEqual(sum(g.dedicated for g in procs.values()), 8192 * 1024)
- self.assertTrue(all("dedicated" in g.available for g in procs.values()))
+
+ def test_private_and_co_owned_clients_leave_the_whole_process_unattributed(self):
+ # pid 100 alone holds client 1 and shares client 2 with pid 200: charging client 1 to it would depend on
+ # which other processes happened to be visible, so neither process gets bytes.
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, kw = self._setup(Path(tmp))
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, vram=1024))
+ _write_fd(proc, 100, 6, _fdinfo(2, self.PDEV, vram=2048))
+ _write_fd(proc, 200, 5, _fdinfo(2, self.PDEV, vram=2048))
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start()
+ procs = counters.sample()[0][card.key]
+ for pid in (100, 200):
+ self.assertEqual(procs[pid].drm_reason, drm.REASON_SHARED_CLIENT)
+ self.assertEqual(procs[pid].dedicated, 0)
+ self.assertNotIn("dedicated", procs[pid].available)
+ self.assertEqual(procs[100].resident_vram, 3072 * 1024)
def test_uninspectable_processes_are_reported(self):
with tempfile.TemporaryDirectory() as tmp:
@@ -445,6 +573,54 @@ class DrmScanTests(unittest.TestCase):
counters.sample()
self.assertEqual(counters.status, "") # readable now
+ def test_unreadable_fd_after_listing_is_incomplete(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, kw = self._setup(Path(tmp))
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, vram=4096))
+ _write_fd(proc, 100, 6, _fdinfo(2, self.PDEV, vram=4096))
+ real = os.readlink
+
+ def readlink(path):
+ if str(path).endswith(os.path.join("100", "fd", "6")):
+ raise PermissionError(path)
+ return real(path)
+
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ with patch.object(drm.os, "readlink", readlink):
+ g = counters.sample()[0][card.key][100]
+ self.assertEqual(g.drm_reason, drm.REASON_INCOMPLETE) # client 2 might be shared: don't charge 1
+ self.assertIsNone(g.shown_vram)
+
+ def test_unreadable_fdinfo_is_incomplete(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, kw = self._setup(Path(tmp))
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, vram=4096))
+ _write_fd(proc, 100, 6, _fdinfo(2, self.PDEV, vram=4096))
+ real = open
+
+ def guarded(path, *a, **k):
+ if str(path).endswith(os.path.join("fdinfo", "6")):
+ raise PermissionError(path)
+ return real(path, *a, **k)
+
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ with patch("builtins.open", guarded):
+ g = counters.sample()[0][card.key][100]
+ self.assertEqual(g.drm_reason, drm.REASON_INCOMPLETE)
+
+ def test_reused_pid_drops_cached_fds(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, kw = self._setup(Path(tmp))
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, vram=4096))
+ (proc / "100" / "stat").write_text("100 (old) S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1000 0\n")
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.sample()
+ self.assertIn(100, counters._known)
+ (proc / "100" / "stat").write_text("100 (new) S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 2000 0\n")
+ counters.scan_budget = 0.0 # nothing is rescanned: only the cache could keep pid 100
+ self.assertNotIn(100, counters.sample()[0][card.key])
+ self.assertNotIn(100, counters._known)
+
def test_scan_budget_keeps_rows_and_resumes(self):
with tempfile.TemporaryDirectory() as tmp:
ticks = [0.0]
@@ -456,23 +632,149 @@ class DrmScanTests(unittest.TestCase):
card, proc, kw = self._setup(Path(tmp), timer)
for pid in (100, 200, 300, 400):
_write_fd(proc, pid, 5, _fdinfo(pid, self.PDEV, gfx=0, vram=4096))
- counters = drm.DrmCounters(found=[card], proc=str(proc), **kw) # the baseline is never limited
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start() # the baseline is never limited
self.assertEqual(sorted(counters._known), [100, 200, 300, 400])
counters.scan_budget = 2.5 # one process per refresh with this timer
_write_fd(proc, 500, 5, _fdinfo(500, self.PDEV, gfx=0, vram=4096))
- seen = set()
- for _ in range(6):
- rows = counters.sample()[0][card.key]
- self.assertTrue({100, 200, 300, 400} <= set(rows)) # no row flickers while the scan is partial
- seen |= set(rows)
- if 500 in rows:
- break
- self.assertIn(500, seen) # a new client is found once the scan comes round to it
+ rows = counters.sample()[0][card.key]
+ self.assertTrue({100, 200, 300, 400} <= set(rows)) # no row flickers while the scan is partial
+ self.assertIn(500, rows) # processes not seen before are scanned first
self.assertIn("scan budget reached", counters.status)
counters.scan_budget = None
counters.sample()
self.assertEqual(counters.status, "")
+ def test_budget_is_checked_inside_one_process(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ ticks = [0.0]
+
+ def timer():
+ ticks[0] += 1.0
+ return ticks[0]
+
+ card, proc, kw = self._setup(Path(tmp), timer)
+ _write_fd(proc, 50, 5, _fdinfo(1, self.PDEV, vram=4096))
+ for fd in range(200): # a process with many fds
+ _write_fd(proc, 100, fd, "pos:\t0\n", target=f"socket:[{fd}]")
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.scan_budget = 3.5 # pid 50 and the first 127 fds of pid 100
+ counters.sample()
+ self.assertIn(50, counters._known) # the first process of a refresh is listed in full
+ self.assertNotIn(100, counters._known) # interrupted partway: deferred, not cached as fd-less
+ self.assertIn("scan budget reached", counters.status)
+
+ def test_one_huge_process_does_not_stall_the_rotation(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ ticks = [0.0]
+
+ def timer():
+ ticks[0] += 1.0
+ return ticks[0]
+
+ card, proc, kw = self._setup(Path(tmp), timer)
+ for fd in range(200):
+ _write_fd(proc, 100, fd, "pos:\t0\n", target=f"socket:[{fd}]")
+ _write_fd(proc, 200, 5, _fdinfo(2, self.PDEV, vram=4096))
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.scan_budget = 1.5
+ for _ in range(3):
+ counters.sample()
+ self.assertEqual(sorted(counters._known), [100, 200])
+
+ def test_partial_scan_charges_no_one(self):
+ # A process the budget skipped may hold the same client, so single ownership isn't proven this refresh.
+ with tempfile.TemporaryDirectory() as tmp:
+ ticks = [0.0]
+
+ def timer():
+ ticks[0] += 1.0
+ return ticks[0]
+
+ card, proc, kw = self._setup(Path(tmp), timer)
+ for pid in (100, 200):
+ _write_fd(proc, pid, 5, _fdinfo(pid, self.PDEV, vram=4096))
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start()
+ counters.scan_budget = 1.5
+ procs = counters.sample()[0][card.key]
+ self.assertTrue(counters.partial)
+ for pid in (100, 200):
+ self.assertEqual(procs[pid].drm_reason, drm.REASON_PARTIAL_SCAN)
+ self.assertIsNone(procs[pid].shown_vram)
+ self.assertNotIn("engine", procs[pid].available)
+ self.assertEqual(procs[pid].resident_vram, 4096 * 1024)
+
+ def test_rotation_resumes_at_a_known_process(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ ticks = [0.0]
+
+ def timer():
+ ticks[0] += 1.0
+ return ticks[0]
+
+ card, proc, kw = self._setup(Path(tmp), timer)
+ for pid in (100, 200, 300):
+ _write_fd(proc, pid, 5, _fdinfo(pid, self.PDEV, vram=4096))
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ counters.start()
+ for pid in (900, 901): # new processes are scanned first and run out the budget
+ _write_fd(proc, pid, 5, _fdinfo(pid, self.PDEV, vram=4096))
+ counters._cursor = 200
+ counters.scan_budget = 1.5
+ counters.sample()
+ self.assertEqual(counters._cursor, 200) # not 901: known processes 200 and 300 aren't skipped
+
+ def test_incomplete_process_has_no_partial_totals(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ card, proc, kw = self._setup(Path(tmp))
+ _write_fd(proc, 100, 5, _fdinfo(1, self.PDEV, gfx=0, vram=4096))
+ _write_fd(proc, 100, 6, _fdinfo(2, self.PDEV, gfx=0, vram=4096))
+ real = open
+
+ def guarded(path, *a, **k):
+ if str(path).endswith(os.path.join("fdinfo", "6")):
+ raise PermissionError(path)
+ return real(path, *a, **k)
+
+ counters = drm.DrmCounters(found=[card], proc=str(proc), **kw)
+ with patch("builtins.open", guarded):
+ counters.start()
+ self.clock.tick()
+ g = counters.sample()[0][card.key][100]
+ self.assertIsNone(g.resident_vram)
+ self.assertNotIn("engine", g.available)
+ self.assertIn("1 processes whose fds or fdinfo", counters.status)
+
+class DrmReportTests(unittest.TestCase):
+ def test_json_and_text_keep_unknown_apart_from_zero(self):
+ from gputaskman.collector import ProcRow, Snapshot
+ from gputaskman.pdh import ProcGpu
+ from gputaskman.procinfo import ProcDetail
+ from gputaskman.report import DRM_NOTES, to_dict, to_text
+
+ private = ProcGpu(dedicated=GiB, shared=0, available={"dedicated", "shared", "engine"},
+ resident_vram=float(GiB), resident_gtt=0.0)
+ compositor = ProcGpu(attributed=False, available={"engine"}, resident_vram=float(2 * GiB),
+ resident_gtt=None, drm_reason=drm.REASON_SHARED_BUFFERS)
+ rows = [ProcRow(100, "llama-server", "LLM", private, ProcDetail(100, "llama-server")),
+ ProcRow(200, "kwin_wayland", "compositor", compositor, ProcDetail(200, "kwin_wayland"))]
+ snap = Snapshot(taken=1, adapter=None, adapters=[], nv=None, driver="", cuda="", vram_total=16 * GiB,
+ vram_used=4 * GiB, shared_used=0, backend="drm", procs=rows, shared_is_spill=True)
+ data = to_dict(snap)
+ self.assertTrue(data["shared_is_spill"])
+ llama, kwin = data["processes"]
+ self.assertEqual((llama["vram_bytes"], llama["shared_bytes"], llama["vram_resident_bytes"]), (GiB, 0, GiB))
+ self.assertIsNone(llama["vram_unattributed_reason"])
+ self.assertEqual((kwin["vram_bytes"], kwin["shared_bytes"]), (None, None))
+ self.assertEqual((kwin["vram_resident_bytes"], kwin["gtt_resident_bytes"]), (2 * GiB, None))
+ self.assertEqual(kwin["vram_unattributed_reason"], "shared_buffers")
+ text = to_text(snap)
+ line = next(x for x in text.splitlines() if x.lstrip().startswith("200"))
+ self.assertEqual(line.split()[3:5], ["n/a", "N/A"]) # VRAM and SPILL unknown, not blank or 0
+ self.assertIn(DRM_NOTES["shared_buffers"], text)
+ self.assertNotIn(DRM_NOTES["legacy"], text)
+
class EngineAndSelectionTests(unittest.TestCase):
def test_engine_names(self):