Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Tighten WSL attribution evidence and first-sample reporting

Scott Hanselmand56e44aca8d1
diff --git a/README.md b/README.md
index 72e9d29..595d958 100644
--- a/README.md
+++ b/README.md
@@ -123,6 +123,10 @@ and per-process SM, encoder, and decoder load from NVML utilization samples. Lin
equivalent of WDDM's shared-memory or commit counters, so Spill and Paged stay blank rather than
showing zero. Only processes you can inspect are listed; other users' processes need `sudo`.
+JSON uses `null` for unavailable counters, not a measured zero: spill/commit/paged values on
+native Linux, and per-process engine load for WSL Linux rows. Native NVML engine readings remain
+available even if NVML hides that process's VRAM. Windows host rows retain their measured counters.
+
**Containers** (Docker, Kubernetes, rented GPU boxes) need one more step: NVML reports *host*
PIDs, which mean nothing in the container's `/proc`, and the container cannot see the host PID of
its own processes. gputaskman pairs NVML's unresolved PIDs with the container's processes that
@@ -172,15 +176,22 @@ gputaskman follows the contexts from sample to sample and labels them with evide
| Marker | Meaning | Established by |
|---|---|---|
-| `*` matched | Windows' exact size; the owner is inferred, and only one explanation fits (or every one that fits gives the same size) | A new Linux GPU process and a new context appearing together; elimination; contexts seen when gputaskman starts while no other Linux process in this distro uses the GPU (remembered for this WSL boot in `~/.cache/gputaskman/`); this monitor's own 0-byte context being newest at startup; candidate contexts that are exactly the same size |
-| `?` guess | Start-order guess for processes that were already running | Newer process ↔ newer context, outside contexts assumed oldest |
+| `*` matched | Windows' exact size; the owner is inferred, and only one explanation fits (or every one that fits gives the same size) | A new Linux GPU process and a new context appearing together; elimination; this monitor's own context being the only zero-byte candidate at startup; candidate contexts that are exactly the same size |
+| `?` guess | Heuristic owner for processes that were already running | Newer process ↔ newer context, outside contexts assumed oldest; cached foreign sizes can rank extra candidates but never establish ownership |
| `n/a` | No context could be assigned | For example, a process that has not created its GPU context yet |
A guess is recomputed every refresh and never counts as evidence for itself. An exited process's
context is held aside until it disappears, so it is not handed to a newcomer, and a context that
-grows by gigabytes between refreshes (weights uploading) keeps its owner. The matched size also
-feeds the LLM weights / KV-cache breakdown. The inspector and `--json` (`vram_inferred_by`) say
-which kind of match each row has.
+grows by gigabytes between refreshes (weights uploading) keeps an established owner. Multiple
+zero-byte contexts at startup do not pin the monitor to an arbitrary one that might later grow.
+Foreign sizes observed while only the monitor is running are cached per WSL boot in
+`~/.cache/gputaskman/`, but remain hints: a process may exit while the monitor is off and a local
+process may reuse its size. Observed foreign-context exits and idle samples clear obsolete hints.
+
+The matched size also feeds VRAM sorting, initial model selection, the bounded model-file scan
+priority, and the LLM weights / KV-cache breakdown. Inferred Linux rows remain below the VM total
+and are never added to it again. The inspector and `--json` (`vram_inferred_by`) say which kind
+of match each row has.
In one live check, a 2 GB and a 5 GB PyTorch allocation were matched to the right PIDs at
2.119 GiB and 4.914 GiB: the allocator's own figure plus about 0.26 GiB of CUDA context.
diff --git a/src/gputaskman/app.py b/src/gputaskman/app.py
index 320c170..f80f533 100644
--- a/src/gputaskman/app.py
+++ b/src/gputaskman/app.py
@@ -20,7 +20,7 @@ from textual.worker import get_current_worker
from .collector import Collector, ProcRow, Snapshot
from .llm import gb
from .nvml import NvDevice, format_metric
-from .report import WSL_EMPTY, inferred_note, vram_label, wsl_note
+from .report import WSL_EMPTY, has_engine_counters, has_wddm_counters, inferred_note, vram_label, wsl_note
KIND_STYLE = {
"LLM": "bold magenta", "decision": "bold orchid", "decision?": "orchid",
@@ -84,7 +84,8 @@ are Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up sh
interop is enabled, the Windows host's counters are read through `powershell.exe`: **WSL VM**
(`vmwp.exe`) is then the exact total of Linux GPU memory, and Windows apps appear as `W:<pid>`
rows. `--no-host` turns this off. Linux rows get their VRAM by matching the VM's GPU contexts
-(newest first) to Linux processes: `*` when only one explanation fits, `?` for a start-order guess.
+(newest first) to Linux processes: `*` when candidates agree on the size, `?` for a heuristic owner.
+Cached foreign sizes only help rank guesses; they never prove ownership across runs.
**Why does nvidia-smi say N/A?** GeForce cards on Windows run in *WDDM* mode: Windows, not the
NVIDIA driver, owns video memory, so NVML can't attribute it to processes. The Windows counters can.
@@ -175,7 +176,7 @@ class GpuTaskManager(App):
Binding("p", "pause", "Pause"),
Binding("question_mark", "help", "Explain"),
]
- SORTS = [("VRAM", lambda r: -r.gpu.shown_vram), ("GPU%", lambda r: -r.gpu.gpu_pct),
+ SORTS = [("VRAM", lambda r: -(r.gpu.shown_vram or 0)), ("GPU%", lambda r: -r.gpu.gpu_pct),
("spill", lambda r: -r.gpu.shared), ("paged", lambda r: -r.gpu.paged_out),
("name", lambda r: r.name.lower())]
CHART_COLORS = {
@@ -279,7 +280,7 @@ class GpuTaskManager(App):
history.append(None)
self.hist_vram.append(100 * snap.vram_used / snap.vram_total if snap.vram_total else None)
if not self._auto_selected:
- llm = next((r for r in sorted(snap.procs, key=lambda r: -r.gpu.shown_vram) if r.llm), None)
+ llm = next((r for r in sorted(snap.procs, key=self.SORTS[0][1]) if r.llm), None)
if llm:
self.selected_key = llm.key
self._auto_selected = True
@@ -428,23 +429,28 @@ class GpuTaskManager(App):
keep = self.selected_key
t.clear()
- def pct(v: float) -> Text:
+ def pct(v: float | None) -> Text:
+ if v is None:
+ return Text("n/a", justify="right", style="grey62")
return Text(f"{v:.0f}" if v >= 0.5 else "", justify="right", style="bold" if v >= 20 else "")
for r in rows:
e = r.gpu.engines
- spill = Text(gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",
+ wddm = has_wddm_counters(s, r)
+ engines = has_engine_counters(s, r)
+ spill = Text(gb(r.gpu.shared) if wddm and r.gpu.shared >= 1 << 20 else "", justify="right",
style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")
- paged = Text(gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")
+ paged = Text(gb(r.gpu.paged_out) if wddm and r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")
t.add_row(
Text(r.pid_label, justify="right", style="grey62" if r.host else ""),
Text(r.name),
Text(r.kind, style=KIND_STYLE.get(r.kind, "")),
Text(" ".join(r.detail.runtimes[:3]), style="grey70"),
Text(vram_label(r.gpu), justify="right", style="bold" if r.gpu.attributed else "grey62"), spill, paged,
- pct(r.gpu.gpu_pct) if r.gpu.attributed else Text("n/a", justify="right", style="grey62"),
- pct(e.get("3d", 0) + e.get("compute", 0)), pct(e.get("copy", 0)),
- pct(e.get("decode", 0) + e.get("encode", 0)),
+ pct(r.gpu.gpu_pct if engines else None),
+ pct(e.get("3d", 0) + e.get("compute", 0) if engines else None),
+ pct(e.get("copy", 0) if engines else None),
+ pct(e.get("decode", 0) + e.get("encode", 0) if engines else None),
Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),
key=r.key,
)
@@ -468,12 +474,12 @@ class GpuTaskManager(App):
panel.border_title = f"{r.name} ({where}{r.pid}) - {r.kind}"
out: list = []
e = r.gpu.engines
- wddm = s.backend == "wddm" or r.origin == "windows" # spill and commit counters only exist in WDDM
+ wddm = has_wddm_counters(s, r)
engines = " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle"
hdr = Text.assemble(("VRAM ", "bold"), (vram_label(r.gpu), "bold"), " spill(RAM) ",
(gb(r.gpu.shared) if wddm else "n/a", "bold red" if r.gpu.shared > 256 << 20 else ""),
" paged(est) ", gb(r.gpu.paged_out) if wddm else "n/a",
- " engines ", engines if r.gpu.attributed else "n/a")
+ " engines ", engines if has_engine_counters(s, r) else "n/a")
if r.detail.runtimes:
hdr.append(" runtime ", style="grey62")
hdr.append(" ".join(r.detail.runtimes), style="cyan")
@@ -485,7 +491,7 @@ class GpuTaskManager(App):
out.append(Text(inferred_note(r.gpu), style="grey62"))
elif s.wsl_split:
out.append(Text(f"No per-process VRAM: {s.wsl_split}.", style="grey62"))
- if r.gpu.paged_out >= 64 << 20:
+ if wddm and r.gpu.paged_out >= 64 << 20:
out.append(Text(f"{gb(r.gpu.paged_out)} of {gb(r.gpu.committed)} committed is not accounted for by local/non-local "
"usage. Estimated non-residency, not proof of eviction or disk paging.",
style="yellow"))
@@ -501,7 +507,7 @@ class GpuTaskManager(App):
v = r.llm
if v:
width = max(min(self.size.width - 8, 100), 20)
- total = max(r.gpu.shown_vram, sum(x.bytes for x in v.segments), 1)
+ total = max(r.gpu.shown_vram or 0, sum(x.bytes for x in v.segments), 1)
out.append(stacked_bar([(x.bytes, SEG_STYLE[x.kind]) for x in v.segments], total, width))
legend = Text()
for x in v.segments:
diff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.py
index 1d414dd..331e935 100644
--- a/src/gputaskman/collector.py
+++ b/src/gputaskman/collector.py
@@ -171,7 +171,6 @@ class Collector:
self._host_luid: int | None = None # host adapter for this GPU, matched once and kept
self.tracker = ContextTracker(self_pid=os.getpid()) if self.host is not None else None
self._own_start = time.time() # our NVML context is created now, not when the process started
- self._inferred: dict[int, float] = {} # WSL: last inferred VRAM per Linux PID
self._host_restarted = time.time()
def _parent_ports(self, pid: int, depth: int = 2) -> list[int]:
@@ -207,32 +206,35 @@ class Collector:
snap.host_status = self.host_status
live = [(pid, g) for pid, g in procs.items() if _is_live(g)]
- # WSL Linux rows are ranked by last sample's inferred size, so the deep-scan budget reaches them too
- live.sort(key=lambda x: -(x[1].dedicated or self._inferred.get(x[0], 0.0)))
- deep_pids = {pid for pid, _ in live[: self.DEEP_TOP_N]} | {pid for pid, g in live if g.gpu_pct >= 2}
- deadline = None if self.scan_budget is None else time.perf_counter() + self.scan_budget
- ollama_runners = []
- for pid, g in live: # largest VRAM first, so the budget goes to what matters
+ live.sort(key=lambda x: -(x[1].shown_vram or 0))
+ for pid, g in live:
d = self.inspector.get(pid)
- # WSL: Linux rows carry no counter of their own; the last inferred size triggers rescans instead
- mem = g.dedicated + g.shared or self._inferred.get(pid, 0.0)
- if (pid in deep_pids and (deadline is None or time.perf_counter() < deadline)
- and self.inspector.needs_scan(d, mem)):
- self.inspector.scan(d, mem)
row = ProcRow(pid=pid, name=d.name, kind=classify(d, g.engines), gpu=g, detail=d)
snap.procs.append(row)
- # Old Ollama: "ollama.exe runner ..."; newer Ollama: its own lib\ollama\llama-server.exe
- exe = d.exe.lower()
- if row.kind in AI_KINDS and ("\\ollama\\" in exe or "/ollama/" in exe or
- (d.name.lower().startswith("ollama") and "runner" in d.cmdline)):
- ollama_runners.append(row)
for pid, g in getattr(self.counters, "outside", {}).get(luid, {}).items():
# A container sees the host's PIDs from NVML; unmatched ones are other containers or the host.
snap.procs.append(ProcRow(pid, "(outside this container)", "external", g,
ProcDetail(pid, "(outside this container)"), host=True, origin="outside"))
- if self.host is not None: # before LLM views, so an inferred VRAM total feeds the breakdown
+ if self.host is not None: # current inferred sizes drive both scan priority and the LLM breakdown
self._add_host_rows(snap)
+ local_rows = [r for r in snap.procs if not r.host]
+ deep_pids = {r.pid for r in local_rows[: self.DEEP_TOP_N]} | {
+ r.pid for r in local_rows if r.gpu.gpu_pct >= 2}
+ deadline = None if self.scan_budget is None else time.perf_counter() + self.scan_budget
+ ollama_runners = []
+ for row in local_rows: # largest VRAM first, so the budget goes to what matters
+ d, g = row.detail, row.gpu
+ mem = (g.shown_vram or 0) + g.shared
+ if (row.pid in deep_pids and (deadline is None or time.perf_counter() < deadline)
+ and self.inspector.needs_scan(d, mem)):
+ self.inspector.scan(d, mem)
+ row.kind = classify(d, g.engines)
+ # Old Ollama: "ollama.exe runner ..."; newer Ollama: its own lib\ollama\llama-server.exe
+ exe = d.exe.lower()
+ if row.kind in AI_KINDS and ("\\ollama\\" in exe or "/ollama/" in exe or
+ (d.name.lower().startswith("ollama") and "runner" in d.cmdline)):
+ ollama_runners.append(row)
for row in snap.procs:
if row.host:
continue
@@ -243,10 +245,9 @@ class Collector:
own = self.prober.get(row.detail.ports)
host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end process
log = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else None
- inferred = row.gpu.inferred
- row.llm = build_view(row.detail, row.gpu.dedicated if inferred is None else inferred, row.gpu.shared,
+ row.llm = build_view(row.detail, row.gpu.shown_vram or 0, row.gpu.shared,
own or host, log, host=host if own else None,
- attributed=row.gpu.attributed or inferred is not None)
+ attributed=row.gpu.shown_vram is not None)
self.inspector.prune(set(procs))
return snap
@@ -295,6 +296,7 @@ class Collector:
at = next((i + 1 for i, r in enumerate(host_rows) if r.kind == "wsl-vm"), 0)
if len(vm_workers) == 1 and self.tracker is not None:
self._attribute(snap, luid, hs.contexts.get(luid, {}).get(vm_workers[0], []))
+ snap.procs.sort(key=lambda r: -(r.gpu.shown_vram or 0))
snap.procs = host_rows[:at] + snap.procs + host_rows[at:]
def _attribute(self, snap: Snapshot, luid: int, contexts: list[float]) -> None:
@@ -303,7 +305,6 @@ class Collector:
holders = [(r.pid, r.detail.create_time) for r in linux] + [(os.getpid(), self._own_start)]
guesses, reason = self.tracker.update(luid, contexts, holders)
snap.wsl_split = reason
- self._inferred = {pid: g.bytes for pid, g in guesses.items()}
for r in linux:
g = guesses.get(r.pid)
if g is not None:
diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.py
index 1f183e2..4116ec6 100644
--- a/src/gputaskman/pdh.py
+++ b/src/gputaskman/pdh.py
@@ -72,18 +72,18 @@ class ProcGpu:
shared: float = 0.0 # bytes resident in shared (system RAM) GPU memory -- "spill" ("Non Local Usage")
committed: float = 0.0 # all GPU memory this process allocated, resident or not ("Total Committed")
engines: dict[str, float] = field(default_factory=lambda: defaultdict(float)) # engtype -> util %
- # False when the OS knows this process uses the GPU but cannot say how much (WSL: the Windows host
- # charges every Linux process's VRAM to the VM). Numbers are then unknown, not zero.
+ # False when per-process VRAM is unavailable. Native NVML may still report engines independently;
+ # WSL charges every Linux process's VRAM and engine activity to the Windows VM worker.
attributed: bool = True
# WSL: VRAM matched from the VM's GPU contexts (see wslhost.ContextTracker). An exact counter value whose
# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.
inferred: float | None = None
- inferred_by: str = "" # "consistent" (only one explanation fits) | "order" (start-order guess)
+ inferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)
@property
- def shown_vram(self) -> float:
- """The VRAM figure a row displays: measured, else inferred (WSL), else 0 for sorting."""
- return self.dedicated if self.attributed else (self.inferred or 0.0)
+ def shown_vram(self) -> float | None:
+ """Measured or inferred bytes for display and priority, not for summing adapter totals."""
+ return self.dedicated if self.attributed else self.inferred
@property
def paged_out(self) -> float:
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index 4c54f4d..a9895f6 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -6,9 +6,10 @@ import dataclasses
import json
import os
-from .collector import Snapshot
+from .collector import ProcRow, Snapshot
from .llm import gb
from .nvml import format_metric
+from .pdh import ProcGpu
WSL_HOST_NOTE = ("WSL + Windows host counters: 'WSL VM' is the exact total of Linux GPU memory (all distros). "
@@ -29,20 +30,27 @@ WSL_NOTE = ("WSL: VRAM and engine load are device-wide. The Windows host charges
"VRAM to the VM, so rows below are processes using the GPU, not their share of it.")
-def vram_label(g) -> str:
+def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:
+ return s.backend == "wddm" or row.origin == "windows"
+
+
+def has_engine_counters(s: Snapshot, row: ProcRow) -> bool:
+ return s.backend != "wsl" or row.origin == "windows"
+
+
+def vram_label(g: ProcGpu) -> str:
"""Measured size; '*' / '?' when the size is exact but which local process owns it is inferred."""
mark = {"order": "?", "consistent": "*"}.get(g.inferred_by, "")
- if g.attributed:
- return gb(g.dedicated) + mark
- return "n/a" if g.inferred is None else gb(g.inferred) + mark
+ return "n/a" if g.shown_vram is None else gb(g.shown_vram) + mark
INFERRED_NOTE = ("* matched: the size is exact (Windows' counter for a WSL VM GPU context, or NVML for a host PID "
"seen from a container); which local process owns it is inferred, and only one explanation fits "
"(or every one that fits gives the same size).")
ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer processes are paired with newer "
- "GPU contexts / PIDs and outside ones are assumed oldest. Becomes * with evidence, such as a process "
- "starting while gputaskman runs.")
+ "GPU contexts / PIDs and outside ones are assumed oldest. Cached foreign-size hints only rank extra "
+ "candidates; they never prove ownership. Becomes * with evidence, such as a process starting while "
+ "gputaskman runs.")
def inferred_note(g) -> str:
@@ -53,20 +61,18 @@ def to_dict(s: Snapshot) -> dict:
nv = dataclasses.asdict(s.nv) if s.nv else None
procs = []
for p in s.procs:
- # Unknown is null, not 0: WSL Linux rows have no per-process counters, and spill/commit only exist
- # in WDDM (native Windows, or Windows processes seen from WSL).
- known = p.gpu.attributed
- wddm = known and (s.backend == "wddm" or p.origin == "windows")
+ wddm = has_wddm_counters(s, p)
+ engines = has_engine_counters(s, p)
item = {
"pid": p.pid, "host": p.origin if p.host else None, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,
"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,
"vram_inferred_by": p.gpu.inferred_by or None,
- "vram_bytes": int(p.gpu.dedicated) if known else None,
+ "vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed else None,
"shared_bytes": int(p.gpu.shared) if wddm else None,
"committed_bytes": int(p.gpu.committed) if wddm else None,
"paged_out_bytes": int(p.gpu.paged_out) if wddm else None,
- "gpu_pct": round(p.gpu.gpu_pct, 1) if known else None,
- "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if known else None,
+ "gpu_pct": round(p.gpu.gpu_pct, 1) if engines else None,
+ "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if engines else None,
"ports": p.detail.ports, "inside": p.inside,
"model_files": [dataclasses.asdict(m) for m in p.detail.models],
"installed_models": [dataclasses.asdict(m) for m in p.detail.candidates],
@@ -123,10 +129,11 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
w = max([7] + [len(p.pid_label) for p in shown]) # host PIDs like W:120880 are wider
out.append(f"{'PID':>{w}} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")
for p in shown:
- spill = gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""
- paged = gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""
+ wddm = has_wddm_counters(s, p)
+ spill = gb(p.gpu.shared) if wddm and p.gpu.shared >= 1 << 20 else ""
+ paged = gb(p.gpu.paged_out) if wddm and p.gpu.paged_out >= 1 << 20 else ""
pid = p.pid_label
- pct = f"{p.gpu.gpu_pct:.0f}" if p.gpu.attributed else "n/a"
+ pct = f"{p.gpu.gpu_pct:.0f}" if has_engine_counters(s, p) else "n/a"
out.append(f"{pid:>{w}} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {vram_label(p.gpu):>9} {spill:>8} "
f"{paged:>8} {pct:>5} {p.inside}")
if any(p.gpu.inferred_by == "consistent" for p in shown):
diff --git a/src/gputaskman/wslhost.py b/src/gputaskman/wslhost.py
index 854d660..0dd0718 100644
--- a/src/gputaskman/wslhost.py
+++ b/src/gputaskman/wslhost.py
@@ -306,7 +306,8 @@ class ContextTracker:
Contexts are aligned sample to sample (survivors keep their order). A match becomes *sure* only from
evidence: a new Linux GPU process and a new context appearing together, elimination (as many unsure
contexts as unsure processes, and just one process), contexts seen at startup while no other Linux process
- holds the GPU (foreign; cached per WSL boot), or this monitor's own 0-byte context being the newest at startup.
+ holds the GPU, or this monitor's own context being the only 0-byte candidate at startup.
+ Cached foreign sizes help rank guesses but never establish ownership across runs.
Everything else is an order guess (newest process <-> newest context, foreign assumed oldest),
recomputed each sample and never used as evidence.
"""
@@ -314,7 +315,6 @@ class ContextTracker:
PENDING = 5.0 # seconds a new process may take to create its context
GONE = 5.0 # seconds an exited process's context may linger in the (older) host sample
TOL_MIN = 64 * MiB
- SELF_MAX = 1 * MiB # an NVML-only context (this monitor) holds 0 bytes
TOL_REL = 0.10
def __init__(self, path: str | None = None, boot_id: str | None = None, self_pid: object = None) -> None:
@@ -400,6 +400,7 @@ class ContextTracker:
self._reset()
start = dict(holders)
first = self.entries is None
+ had_foreign = any(e.label == FOREIGN and e.sure for e in self.entries or [])
cur = [_Entry(v) for v in contexts]
fresh: list[_Entry] = [] # contexts that appeared since the previous sample
if not first:
@@ -446,20 +447,13 @@ class ContextTracker:
if e.label is None:
e.label, e.sure = FOREIGN, True
elif first:
- # Our NVML-only context is 0 bytes and newest at startup, unless another process started after us:
- # its CUDA context could then be the newest one (and small while it initializes).
- later = any(t >= start[self.self_pid] for p, t in start.items() if p != self.self_pid) \
- if self.self_pid in start else True
- if not later and cur and cur[0].value <= self.SELF_MAX:
- cur[0].label, cur[0].sure = self.self_pid, True
- if set(start) == {self.self_pid} and cur and cur[0].label == self.self_pid:
- for e in cur[1:]: # nothing else in this distro holds the GPU: the rest is foreign
- e.label, e.sure = FOREIGN, True
- for v in self.cached_foreign:
- free = [e for e in reversed(cur) if e.label is None and abs(e.value - v) <= self._tol(v)]
- if free:
- best = min(free, key=lambda e: abs(e.value - v))
- best.label, best.sure = FOREIGN, True
+ zeros = [e for e in cur if e.value == 0]
+ if self.self_pid in start and len(zeros) == 1:
+ zeros[0].label, zeros[0].sure = self.self_pid, True
+ if set(start) == {self.self_pid} and zeros:
+ for e in cur:
+ if e.value > 0: # NVML alone holds no VRAM; other zero-byte owners remain ambiguous
+ e.label, e.sure = FOREIGN, True
elif fresh:
pending = sorted((p for p in start if p not in labeled() and p not in self.initial
and now - self.seen[p] <= self.PENDING),
@@ -473,6 +467,10 @@ class ContextTracker:
unsure = sorted((p for p in start if p not in labeled()), key=lambda p: -age(p))
free = [e for e in cur if e.label is None]
+ if unsure and len(free) > len(unsure):
+ # Cached sizes may outlive their owners: rank guesses, never exclude candidates or pin labels.
+ free.sort(key=lambda e: e.value > 0 and any(
+ v > 0 and abs(e.value - v) <= self._tol(v) for v in self.cached_foreign))
reason = ""
if unsure:
if len(free) < len(unsure):
@@ -492,12 +490,11 @@ class ContextTracker:
e.label, e.same = pid, same
self.entries = cur
- # Foreign labels only come from evidence, so the cache follows them, including down to none: a cached
- # context that exited must not later claim a local context of the same size.
- foreign = [e.value for e in cur if e.label == FOREIGN and e.sure]
- if sorted(foreign) != sorted(self.cached_foreign):
- self.cached_foreign = foreign
- self._save()
+ foreign = [e.value for e in cur if e.label == FOREIGN and e.sure and e.value > 0]
+ if not start or set(start) == {self.self_pid} or foreign or had_foreign:
+ if sorted(foreign) != sorted(self.cached_foreign):
+ self.cached_foreign = foreign
+ self._save()
out = {e.label: ContextGuess(e.value, "consistent" if e.sure or e.same else "order")
for e in cur if e.label in start}
return out, reason
diff --git a/tests/test_core.py b/tests/test_core.py
index 3333bf1..ed140cd 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -16,7 +16,7 @@ from rich.console import Console
from rich.table import Table
from textual.widgets import Footer, Select, Static
-from gputaskman.app import GpuTaskManager, spark
+from gputaskman.app import GpuTaskManager, spark, stacked_bar
from gputaskman.cli import main
from gputaskman.collector import ProcRow, Snapshot, select_adapter, select_nv_device
from gputaskman.dxgi import Adapter
@@ -127,6 +127,16 @@ class DeviceSelectionTests(unittest.TestCase):
class MemoryTests(unittest.TestCase):
+ def test_best_vram_keeps_measured_and_inferred_zero_distinct_from_unknown(self):
+ gpu = ProcGpu(dedicated=0, inferred=8 * GiB)
+ self.assertEqual(gpu.shown_vram, 0)
+ gpu.attributed = False
+ self.assertEqual(gpu.shown_vram, 8 * GiB)
+ gpu.inferred = 0
+ self.assertEqual(gpu.shown_vram, 0)
+ gpu.inferred = None
+ self.assertIsNone(gpu.shown_vram)
+
def test_local_counters_are_preferred(self):
self.assertTrue(GpuCounters._COUNTERS["dedicated"][0].endswith(r"\Local Usage"))
self.assertTrue(GpuCounters._COUNTERS["shared"][0].endswith(r"\Non Local Usage"))
@@ -672,6 +682,48 @@ class UiTests(unittest.IsolatedAsyncioTestCase):
Console(file=output, width=widget.content_size.width, color_system=None).print(widget.content)
return output.getvalue()
+ async def test_wsl_sort_selection_and_bar_use_inferred_vram_without_double_counting(self):
+ s = snapshot()
+ s.backend = "wsl"
+ small = ProcRow(10, "small", "LLM", ProcGpu(attributed=False, inferred=2 * GiB, inferred_by="order"),
+ ProcDetail(10, "small"), LlmView())
+ large = ProcRow(20, "large", "LLM", ProcGpu(attributed=False, inferred=8 * GiB, inferred_by="order"),
+ ProcDetail(20, "large"), LlmView())
+ vm = ProcRow(30, "WSL VM", "wsl-vm", ProcGpu(dedicated=10 * GiB), ProcDetail(30, "WSL VM"),
+ host=True, origin="windows")
+ host = ProcRow(40, "browser.exe", "browser", ProcGpu(dedicated=3 * GiB), ProcDetail(40, "browser.exe"),
+ host=True, origin="windows")
+ s.procs = [small, large, host, vm]
+ with patch.object(GpuTaskManager, "poll"), patch("gputaskman.app.stacked_bar", wraps=stacked_bar) as bar:
+ app = GpuTaskManager()
+ async with app.run_test(size=(180, 42)) as pilot:
+ app.apply(s)
+ await pilot.pause()
+ self.assertEqual(app.selected_key, large.key)
+ self.assertEqual([r.key for r in app.visible_rows(s)], [vm.key, large.key, small.key, host.key])
+ self.assertEqual(bar.call_args.args[1], 8 * GiB)
+ table = app.query_one("#procs")
+ self.assertEqual(table.get_cell(large.key, "VRAM").plain, "8.00 GB?")
+ for column in ("GPU%", "3D", "Copy", "Video"):
+ self.assertEqual(table.get_cell(large.key, column).plain, "n/a")
+ self.assertEqual(s.attributed, 13 * GiB)
+
+ async def test_native_hidden_vram_keeps_engine_reading_in_ui(self):
+ s = snapshot()
+ s.backend = "nvml"
+ s.procs = [ProcRow(10, "python3", "AI", ProcGpu(attributed=False, engines={"compute": 65}),
+ ProcDetail(10, "python3"))]
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ app.selected_key = "10"
+ async with app.run_test(size=(180, 42)) as pilot:
+ app.apply(s)
+ await pilot.pause()
+ self.assertEqual(app.query_one("#procs").get_cell("10", "GPU%").plain, "65")
+ hdr = app.query_one("#inspector", Static).content.renderables[0].plain
+ self.assertIn("compute 65%", hdr)
+ self.assertIn("spill(RAM) n/a", hdr)
+
async def test_eighth_chart_records_sm_clock_with_scale_and_missing_readings(self):
with patch.object(GpuTaskManager, "poll"):
app = GpuTaskManager()
diff --git a/tests/test_linux.py b/tests/test_linux.py
index 0e69949..2196ff5 100644
--- a/tests/test_linux.py
+++ b/tests/test_linux.py
@@ -7,7 +7,7 @@ from unittest.mock import patch
from gputaskman import linux
from gputaskman.collector import ProcRow, Snapshot
from gputaskman.gguf import GgufInfo
-from gputaskman.llm import build_view
+from gputaskman.llm import LlmView, build_view
from gputaskman.pdh import ProcGpu
from gputaskman.procinfo import ProcDetail, _lib_name, _match_runtimes, classify
from gputaskman.report import to_dict, to_text
@@ -275,7 +275,8 @@ class UnattributedReportTests(unittest.TestCase):
def test_reports_say_unknown_instead_of_zero(self):
data = to_dict(self.snapshot())
self.assertEqual(data["backend"], "wsl")
- self.assertIsNone(data["processes"][0]["vram_bytes"])
+ for key in ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):
+ self.assertIsNone(data["processes"][0][key], key)
self.assertFalse(data["processes"][0]["vram_attributed"])
text = to_text(self.snapshot())
row = next(line for line in text.splitlines() if "4242" in line)
@@ -306,6 +307,44 @@ class UnattributedReportTests(unittest.TestCase):
key = GpuTaskManager.SORTS[0][1]
self.assertEqual([r.pid for r in sorted(rows, key=key)], [2, 1, 3])
+ def test_native_hidden_vram_does_not_hide_measured_engine_load(self):
+ snap = self.snapshot()
+ snap.backend = "nvml"
+ snap.procs[0].gpu.engines["compute"] = 47
+ row = to_dict(snap)["processes"][0]
+ self.assertIsNone(row["vram_bytes"])
+ self.assertEqual(row["gpu_pct"], 47)
+ self.assertEqual(row["engines"], {"compute": 47})
+ for key in ("shared_bytes", "committed_bytes", "paged_out_bytes"):
+ self.assertIsNone(row[key], key)
+ text = next(line for line in to_text(snap).splitlines() if "4242" in line)
+ self.assertIn("47", text)
+
+ def test_measured_zero_is_not_changed_to_unknown(self):
+ snap = self.snapshot()
+ snap.backend = "nvml"
+ snap.procs[0].gpu.attributed = True
+ row = to_dict(snap)["processes"][0]
+ self.assertEqual(row["vram_bytes"], 0)
+ self.assertEqual(row["gpu_pct"], 0)
+ self.assertEqual(row["engines"], {})
+ self.assertIsNone(row["shared_bytes"])
+ snap.backend = "wsl"
+ snap.procs[0].host, snap.procs[0].origin = True, "windows"
+ row = to_dict(snap)["processes"][0]
+ for key in ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct"):
+ self.assertEqual(row[key], 0, key)
+
+ def test_model_detail_header_uses_inferred_or_unknown_vram(self):
+ for inferred, label in ((None, "n/a"), (0, "0 MB*"), (3 * GiB, "3.00 GB*")):
+ with self.subTest(inferred=inferred):
+ snap = self.snapshot()
+ p = snap.procs[0]
+ p.llm = LlmView()
+ p.gpu.inferred, p.gpu.inferred_by = inferred, "consistent"
+ self.assertIn(f"== python3 (4242) VRAM {label}", to_text(snap))
+ self.assertIsNone(to_dict(snap)["processes"][0]["vram_bytes"])
+
def test_empty_wsl_view_explains_itself(self):
snap = self.snapshot()
snap.procs = []
diff --git a/tests/test_wslhost.py b/tests/test_wslhost.py
index 13cf4f1..38bfd1a 100644
--- a/tests/test_wslhost.py
+++ b/tests/test_wslhost.py
@@ -6,13 +6,14 @@ import os
import tempfile
import time
import unittest
-from unittest.mock import mock_open, patch
+from unittest.mock import Mock, mock_open, patch
from gputaskman import collector as collector_mod
from gputaskman import wslhost
from gputaskman.collector import Collector, ProcRow, Snapshot
+from gputaskman.llm import LlmView
from gputaskman.pdh import AdapterGpu, ProcGpu
-from gputaskman.procinfo import ProcDetail
+from gputaskman.procinfo import MappedModel, ProcDetail
from gputaskman.report import to_dict, to_text
from gputaskman.wslhost import ContextTracker, HostSample, HostStream, match_adapter, parse_line
@@ -171,8 +172,8 @@ class TrackerTests(unittest.TestCase):
def test_a_workload_started_after_us_is_not_taken_for_our_context(self):
t = self.make()
h = [("S", 900), ("W", 950)] # W created its CUDA context between our NVML init and the first sample
- self.check(self.step(t, [20 / 1024, 0.0], h), {"W": (0.02, "?")})
- self.check(self.step(t, [5.0, 0.0], h), {"W": (5.0, "?")})
+ self.check(self.step(t, [20 / 1024, 0.0], h), {"W": (0.02, "*")})
+ self.check(self.step(t, [5.0, 0.0], h), {"W": (5.0, "*")})
def test_cache_forgets_a_foreign_context_that_exited(self):
t = self.make()
@@ -199,10 +200,59 @@ class TrackerTests(unittest.TestCase):
self.check(self.step(t, [0.0, 0.15], h), {"A": (0.15, "?")}) # A's context isn't there yet
self.check(self.step(t, [6.0, 0.0, 0.15], h), {"A": (6.0, "*")})
- def test_startup_with_only_this_monitor_teaches_foreign(self):
+ def test_cached_foreign_sizes_only_rank_startup_guesses(self):
self.step(self.make(), [0.0, 0.15], [("S", 900)]) # only gputaskman holds the GPU: 0.15 is foreign
t = self.make() # a later start in the same boot, with a job already running
- self.check(self.step(t, [0.0, 2.0, 0.15], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})
+ self.check(self.step(t, [0.0, 0.15, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})
+ self.check(self.step(t, [0.0, 0.15, 2.5], [("S", 900), ("X", 5)]), {"X": (2.5, "?")})
+
+ def test_foreign_cache_clears_immediately_when_last_foreign_context_exits(self):
+ t = self.make()
+ self.step(t, [0.0, 2.0], [("S", 900)])
+ with open(self.path, encoding="utf-8") as f:
+ self.assertEqual(json.load(f)["foreign"], [2 * GiB])
+ self.step(t, [0.0], [("S", 900)])
+ self.assertEqual(t.cached_foreign, [])
+ with open(self.path, encoding="utf-8") as f:
+ self.assertEqual(json.load(f)["foreign"], [])
+ self.check(self.step(self.make(), [0.0, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})
+
+ def test_foreign_cache_clears_while_local_workload_remains(self):
+ t = self.make()
+ self.step(t, [0.0, 0.4], [("S", 900)])
+ h = [("S", 900), ("X", 950)]
+ self.step(t, [2.0, 0.0, 0.4], h)
+ self.check(self.step(t, [2.0, 0.0], h), {"X": (2.0, "*")})
+ with open(self.path, encoding="utf-8") as f:
+ self.assertEqual(json.load(f)["foreign"], [])
+
+ def test_stale_cache_never_takes_a_local_context(self):
+ self.step(self.make(), [0.0, 2.0], [("S", 900)])
+ # The foreign process exits while the monitor is off; a local process reuses that size.
+ self.check(self.step(self.make(), [0.0, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})
+
+ def test_first_sample_finds_monitor_zero_instead_of_newest_small_context(self):
+ t = self.make()
+ h = [("S", 900), ("X", 950)]
+ guesses, _ = t.update(RTX, [20 * MiB, 0], h, now=self.clock)
+ self.assertEqual(guesses["S"].bytes, 0)
+ self.assertEqual(guesses["X"].bytes, 20 * MiB)
+ self.check(self.step(t, [5.0, 0.0], h), {"X": (5.0, "*")})
+
+ def test_equal_zero_contexts_never_pin_monitor_ownership(self):
+ t = self.make()
+ h = [("S", 900), ("X", 950)]
+ self.check(self.step(t, [0.0, 0.0], h), {"X": (0.0, "*")})
+ self.assertFalse(any(e.sure for e in t.entries))
+ self.check(self.step(t, [5.0, 0.0], h), {"X": (5.0, "?")})
+ self.assertFalse(any(e.label == "S" and e.value for e in t.entries))
+
+ def test_idle_zero_ambiguity_does_not_pollute_foreign_cache(self):
+ t = self.make()
+ self.step(t, [0.0, 0.15, 0.0], [("S", 900)])
+ self.assertEqual(t.cached_foreign, [0.15 * GiB])
+ self.assertFalse(any(e.label == "S" and e.sure for e in t.entries))
+ self.check(self.step(t, [2.0, 0.0, 0.15, 0.0], [("S", 900), ("X", 950)]), {"X": (2.0, "*")})
def test_a_reused_pid_is_a_new_process(self):
t = self.make()
@@ -221,12 +271,12 @@ class TrackerTests(unittest.TestCase):
for _ in range(5):
self.check(self.step(t, [0.5, 0.0, 2.12, 3.05, 0.0], h), {"D": (0.5, "*"), "C": (2.12, "?"), "B": (3.05, "?")})
- def test_no_holders_teaches_foreign_and_elimination_confirms(self):
+ def test_no_holders_teaches_foreign_without_confirming_future_owners(self):
self.make().update(RTX, at([0.4]), [], now=self.clock) # nothing in this distro: 0.4 is foreign
t = self.make() # a fresh start in the same boot, with a job already running
- self.check(self.step(t, [0.0, 2.0, 0.4], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})
+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})
t = self.make(boot="boot-2") # a new boot knows nothing
- self.check(self.step(t, [0.0, 2.0, 0.4], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})
+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (0.4, "?")})
def test_context_that_lags_its_process_is_still_pinned(self):
t = self.make()
@@ -278,8 +328,9 @@ class TrackerTests(unittest.TestCase):
def test_different_adapter_resets_history(self):
self.make().update(RTX, at([0.4]), [], now=self.clock)
t = self.make()
- self.check(self.step(t, [0.0, 2.0, 0.4], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})
- guesses, _ = t.update(0x1234, at([0.0, 2.0, 0.4]), [("S", 900), ("X", 5)], now=self.clock + 5)
+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})
+ guesses, _ = t.update(0x1234, at([0.0, 0.4, 2.0]), [("S", 900), ("X", 5)], now=self.clock + 5)
+ self.assertEqual(guesses["X"].bytes, 0.4 * GiB)
self.assertEqual(guesses["X"].confidence, "order")
@@ -307,7 +358,6 @@ class HostRowTests(unittest.TestCase):
c._host_restarted = time.time()
c.tracker = ContextTracker(path=os.path.join(self.tmp.name, "ctx.json"), boot_id="boot", self_pid=os.getpid())
c._own_start = time.time()
- c._inferred = {}
return c
def setUp(self):
@@ -349,11 +399,57 @@ class HostRowTests(unittest.TestCase):
row = next(r for r in snap.procs if not r.host)
self.assertEqual((row.gpu.inferred, row.gpu.inferred_by), (3.0 * GiB, "consistent"))
self.assertFalse(row.gpu.attributed) # engines are still unknown
+ row.llm = LlmView()
text = to_text(snap)
self.assertIn("3.00 GB*", text)
+ self.assertIn("== python3 (2744) VRAM 3.00 GB*", text)
self.assertIn("* matched", text)
data = to_dict(snap)["processes"][1]
self.assertEqual((data["vram_inferred_bytes"], data["vram_inferred_by"]), (3.0 * GiB, "consistent"))
+ self.assertEqual(snap.attributed, 3 * GiB) # the Linux child is not added to the VM total again
+
+ def test_current_inferred_vram_sets_first_sample_scan_priority(self):
+ big_pid = Collector.DEEP_TOP_N + 1
+ total = 8 * GiB + Collector.DEEP_TOP_N * MiB
+ busy = line([["local usage", f"pid_24820_{LUID}", v]
+ for v in [0, 8 * GiB] + [MiB] * Collector.DEEP_TOP_N]
+ + [["dedicated usage", LUID, total]], {"24820": "vmwp"})
+ c = self.collector(FakeStream(parse_line(busy, now=time.time())))
+ c.adapter, c.adapters, c.gpu_index, c.scan_budget = None, [], 0, None
+ c.host_status = ""
+ c.nvml = Mock(ok=False, driver="", cuda_version="")
+ c.counters = Mock(outside={})
+ c.counters.sample.return_value = (
+ {RTX: {pid: ProcGpu(attributed=False) for pid in range(1, big_pid + 1)}},
+ {RTX: AdapterGpu(dedicated=total)})
+ details = {pid: ProcDetail(pid, "python3", create_time=pid, parents=[])
+ for pid in range(1, big_pid + 1)}
+ c.inspector = Mock()
+ c.inspector.get.side_effect = details.__getitem__
+ c.inspector.needs_scan.return_value = True
+
+ def scan(d, mem):
+ if d.pid == big_pid:
+ d.models = [MappedModel("large.gguf", 8 * GiB, 0, True, kind="LLM")]
+
+ c.inspector.scan.side_effect = scan
+ c.prober = Mock()
+ c.prober.get.return_value = None
+ with patch.object(collector_mod, "BACKEND", "wsl"), \
+ patch.object(collector_mod, "build_view", return_value=LlmView()) as view:
+ snap = c.sample()
+ self.assertEqual(c.inspector.scan.call_count, Collector.DEEP_TOP_N)
+ self.assertEqual(c.inspector.scan.call_args_list[0].args, (details[big_pid], 8 * GiB))
+ self.assertNotIn(Collector.DEEP_TOP_N, [call.args[0].pid for call in c.inspector.scan.call_args_list])
+ self.assertEqual([r.pid for r in snap.procs[:2]], [24820, big_pid])
+ self.assertEqual(snap.procs[1].kind, "LLM") # classification follows the scan
+ self.assertEqual(view.call_args.args[1], 8 * GiB)
+ self.assertTrue(view.call_args.kwargs["attributed"])
+ self.assertEqual(snap.attributed, total)
+ c.scan_budget = 0.0
+ c.inspector.scan.reset_mock()
+ c.sample()
+ c.inspector.scan.assert_not_called()
def test_order_guess_is_marked(self):
# an extra (foreign) context and no history: placement is a guess