A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
Tighten WSL attribution evidence and first-sample reporting
Scott Hanselmand56e44aca8d1
diff --git a/README.md b/README.mdindex 72e9d29..595d958 100644--- a/README.md+++ b/README.md@@ -123,6 +123,10 @@ and per-process SM, encoder, and decoder load from NVML utilization samples. Linequivalent of WDDM's shared-memory or commit counters, so Spill and Paged stay blank rather thanshowing zero. Only processes you can inspect are listed; other users' processes need `sudo`.+JSON uses `null` for unavailable counters, not a measured zero: spill/commit/paged values on+native Linux, and per-process engine load for WSL Linux rows. Native NVML engine readings remain+available even if NVML hides that process's VRAM. Windows host rows retain their measured counters.+**Containers** (Docker, Kubernetes, rented GPU boxes) need one more step: NVML reports *host*PIDs, which mean nothing in the container's `/proc`, and the container cannot see the host PID ofits own processes. gputaskman pairs NVML's unresolved PIDs with the container's processes that@@ -172,15 +176,22 @@ gputaskman follows the contexts from sample to sample and labels them with evide| Marker | Meaning | Established by ||---|---|---|-| `*` matched | Windows' exact size; the owner is inferred, and only one explanation fits (or every one that fits gives the same size) | A new Linux GPU process and a new context appearing together; elimination; contexts seen when gputaskman starts while no other Linux process in this distro uses the GPU (remembered for this WSL boot in `~/.cache/gputaskman/`); this monitor's own 0-byte context being newest at startup; candidate contexts that are exactly the same size |-| `?` guess | Start-order guess for processes that were already running | Newer process ↔ newer context, outside contexts assumed oldest |+| `*` matched | Windows' exact size; the owner is inferred, and only one explanation fits (or every one that fits gives the same size) | A new Linux GPU process and a new context appearing together; elimination; this monitor's own context being the only zero-byte candidate at startup; candidate contexts that are exactly the same size |+| `?` guess | Heuristic owner for processes that were already running | Newer process ↔ newer context, outside contexts assumed oldest; cached foreign sizes can rank extra candidates but never establish ownership || `n/a` | No context could be assigned | For example, a process that has not created its GPU context yet |A guess is recomputed every refresh and never counts as evidence for itself. An exited process'scontext is held aside until it disappears, so it is not handed to a newcomer, and a context that-grows by gigabytes between refreshes (weights uploading) keeps its owner. The matched size also-feeds the LLM weights / KV-cache breakdown. The inspector and `--json` (`vram_inferred_by`) say-which kind of match each row has.+grows by gigabytes between refreshes (weights uploading) keeps an established owner. Multiple+zero-byte contexts at startup do not pin the monitor to an arbitrary one that might later grow.+Foreign sizes observed while only the monitor is running are cached per WSL boot in+`~/.cache/gputaskman/`, but remain hints: a process may exit while the monitor is off and a local+process may reuse its size. Observed foreign-context exits and idle samples clear obsolete hints.++The matched size also feeds VRAM sorting, initial model selection, the bounded model-file scan+priority, and the LLM weights / KV-cache breakdown. Inferred Linux rows remain below the VM total+and are never added to it again. The inspector and `--json` (`vram_inferred_by`) say which kind+of match each row has.In one live check, a 2 GB and a 5 GB PyTorch allocation were matched to the right PIDs at2.119 GiB and 4.914 GiB: the allocator's own figure plus about 0.26 GiB of CUDA context.diff --git a/src/gputaskman/app.py b/src/gputaskman/app.pyindex 320c170..f80f533 100644--- a/src/gputaskman/app.py+++ b/src/gputaskman/app.py@@ -20,7 +20,7 @@ from textual.worker import get_current_workerfrom .collector import Collector, ProcRow, Snapshotfrom .llm import gbfrom .nvml import NvDevice, format_metric-from .report import WSL_EMPTY, inferred_note, vram_label, wsl_note+from .report import WSL_EMPTY, has_engine_counters, has_wddm_counters, inferred_note, vram_label, wsl_noteKIND_STYLE = {"LLM": "bold magenta", "decision": "bold orchid", "decision?": "orchid",@@ -84,7 +84,8 @@ are Linux processes with `/dev/dxg` open and show `n/a` rather than a made-up shinterop is enabled, the Windows host's counters are read through `powershell.exe`: **WSL VM**(`vmwp.exe`) is then the exact total of Linux GPU memory, and Windows apps appear as `W:<pid>`rows. `--no-host` turns this off. Linux rows get their VRAM by matching the VM's GPU contexts-(newest first) to Linux processes: `*` when only one explanation fits, `?` for a start-order guess.+(newest first) to Linux processes: `*` when candidates agree on the size, `?` for a heuristic owner.+Cached foreign sizes only help rank guesses; they never prove ownership across runs.**Why does nvidia-smi say N/A?** GeForce cards on Windows run in *WDDM* mode: Windows, not theNVIDIA driver, owns video memory, so NVML can't attribute it to processes. The Windows counters can.@@ -175,7 +176,7 @@ class GpuTaskManager(App):Binding("p", "pause", "Pause"),Binding("question_mark", "help", "Explain"),]- SORTS = [("VRAM", lambda r: -r.gpu.shown_vram), ("GPU%", lambda r: -r.gpu.gpu_pct),+ SORTS = [("VRAM", lambda r: -(r.gpu.shown_vram or 0)), ("GPU%", lambda r: -r.gpu.gpu_pct),("spill", lambda r: -r.gpu.shared), ("paged", lambda r: -r.gpu.paged_out),("name", lambda r: r.name.lower())]CHART_COLORS = {@@ -279,7 +280,7 @@ class GpuTaskManager(App):history.append(None)self.hist_vram.append(100 * snap.vram_used / snap.vram_total if snap.vram_total else None)if not self._auto_selected:- llm = next((r for r in sorted(snap.procs, key=lambda r: -r.gpu.shown_vram) if r.llm), None)+ llm = next((r for r in sorted(snap.procs, key=self.SORTS[0][1]) if r.llm), None)if llm:self.selected_key = llm.keyself._auto_selected = True@@ -428,23 +429,28 @@ class GpuTaskManager(App):keep = self.selected_keyt.clear()- def pct(v: float) -> Text:+ def pct(v: float | None) -> Text:+ if v is None:+ return Text("n/a", justify="right", style="grey62")return Text(f"{v:.0f}" if v >= 0.5 else "", justify="right", style="bold" if v >= 20 else "")for r in rows:e = r.gpu.engines- spill = Text(gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",+ wddm = has_wddm_counters(s, r)+ engines = has_engine_counters(s, r)+ spill = Text(gb(r.gpu.shared) if wddm and r.gpu.shared >= 1 << 20 else "", justify="right",style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")- paged = Text(gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")+ paged = Text(gb(r.gpu.paged_out) if wddm and r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")t.add_row(Text(r.pid_label, justify="right", style="grey62" if r.host else ""),Text(r.name),Text(r.kind, style=KIND_STYLE.get(r.kind, "")),Text(" ".join(r.detail.runtimes[:3]), style="grey70"),Text(vram_label(r.gpu), justify="right", style="bold" if r.gpu.attributed else "grey62"), spill, paged,- pct(r.gpu.gpu_pct) if r.gpu.attributed else Text("n/a", justify="right", style="grey62"),- pct(e.get("3d", 0) + e.get("compute", 0)), pct(e.get("copy", 0)),- pct(e.get("decode", 0) + e.get("encode", 0)),+ pct(r.gpu.gpu_pct if engines else None),+ pct(e.get("3d", 0) + e.get("compute", 0) if engines else None),+ pct(e.get("copy", 0) if engines else None),+ pct(e.get("decode", 0) + e.get("encode", 0) if engines else None),Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),key=r.key,)@@ -468,12 +474,12 @@ class GpuTaskManager(App):panel.border_title = f"{r.name} ({where}{r.pid}) - {r.kind}"out: list = []e = r.gpu.engines- wddm = s.backend == "wddm" or r.origin == "windows" # spill and commit counters only exist in WDDM+ wddm = has_wddm_counters(s, r)engines = " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle"hdr = Text.assemble(("VRAM ", "bold"), (vram_label(r.gpu), "bold"), " spill(RAM) ",(gb(r.gpu.shared) if wddm else "n/a", "bold red" if r.gpu.shared > 256 << 20 else "")," paged(est) ", gb(r.gpu.paged_out) if wddm else "n/a",- " engines ", engines if r.gpu.attributed else "n/a")+ " engines ", engines if has_engine_counters(s, r) else "n/a")if r.detail.runtimes:hdr.append(" runtime ", style="grey62")hdr.append(" ".join(r.detail.runtimes), style="cyan")@@ -485,7 +491,7 @@ class GpuTaskManager(App):out.append(Text(inferred_note(r.gpu), style="grey62"))elif s.wsl_split:out.append(Text(f"No per-process VRAM: {s.wsl_split}.", style="grey62"))- if r.gpu.paged_out >= 64 << 20:+ if wddm and r.gpu.paged_out >= 64 << 20:out.append(Text(f"{gb(r.gpu.paged_out)} of {gb(r.gpu.committed)} committed is not accounted for by local/non-local ""usage. Estimated non-residency, not proof of eviction or disk paging.",style="yellow"))@@ -501,7 +507,7 @@ class GpuTaskManager(App):v = r.llmif v:width = max(min(self.size.width - 8, 100), 20)- total = max(r.gpu.shown_vram, sum(x.bytes for x in v.segments), 1)+ total = max(r.gpu.shown_vram or 0, sum(x.bytes for x in v.segments), 1)out.append(stacked_bar([(x.bytes, SEG_STYLE[x.kind]) for x in v.segments], total, width))legend = Text()for x in v.segments:diff --git a/src/gputaskman/collector.py b/src/gputaskman/collector.pyindex 1d414dd..331e935 100644--- a/src/gputaskman/collector.py+++ b/src/gputaskman/collector.py@@ -171,7 +171,6 @@ class Collector:self._host_luid: int | None = None # host adapter for this GPU, matched once and keptself.tracker = ContextTracker(self_pid=os.getpid()) if self.host is not None else Noneself._own_start = time.time() # our NVML context is created now, not when the process started- self._inferred: dict[int, float] = {} # WSL: last inferred VRAM per Linux PIDself._host_restarted = time.time()def _parent_ports(self, pid: int, depth: int = 2) -> list[int]:@@ -207,32 +206,35 @@ class Collector:snap.host_status = self.host_statuslive = [(pid, g) for pid, g in procs.items() if _is_live(g)]- # WSL Linux rows are ranked by last sample's inferred size, so the deep-scan budget reaches them too- live.sort(key=lambda x: -(x[1].dedicated or self._inferred.get(x[0], 0.0)))- deep_pids = {pid for pid, _ in live[: self.DEEP_TOP_N]} | {pid for pid, g in live if g.gpu_pct >= 2}- deadline = None if self.scan_budget is None else time.perf_counter() + self.scan_budget- ollama_runners = []- for pid, g in live: # largest VRAM first, so the budget goes to what matters+ live.sort(key=lambda x: -(x[1].shown_vram or 0))+ for pid, g in live:d = self.inspector.get(pid)- # WSL: Linux rows carry no counter of their own; the last inferred size triggers rescans instead- mem = g.dedicated + g.shared or self._inferred.get(pid, 0.0)- if (pid in deep_pids and (deadline is None or time.perf_counter() < deadline)- and self.inspector.needs_scan(d, mem)):- self.inspector.scan(d, mem)row = ProcRow(pid=pid, name=d.name, kind=classify(d, g.engines), gpu=g, detail=d)snap.procs.append(row)- # Old Ollama: "ollama.exe runner ..."; newer Ollama: its own lib\ollama\llama-server.exe- exe = d.exe.lower()- if row.kind in AI_KINDS and ("\\ollama\\" in exe or "/ollama/" in exe or- (d.name.lower().startswith("ollama") and "runner" in d.cmdline)):- ollama_runners.append(row)for pid, g in getattr(self.counters, "outside", {}).get(luid, {}).items():# A container sees the host's PIDs from NVML; unmatched ones are other containers or the host.snap.procs.append(ProcRow(pid, "(outside this container)", "external", g,ProcDetail(pid, "(outside this container)"), host=True, origin="outside"))- if self.host is not None: # before LLM views, so an inferred VRAM total feeds the breakdown+ if self.host is not None: # current inferred sizes drive both scan priority and the LLM breakdownself._add_host_rows(snap)+ local_rows = [r for r in snap.procs if not r.host]+ deep_pids = {r.pid for r in local_rows[: self.DEEP_TOP_N]} | {+ r.pid for r in local_rows if r.gpu.gpu_pct >= 2}+ deadline = None if self.scan_budget is None else time.perf_counter() + self.scan_budget+ ollama_runners = []+ for row in local_rows: # largest VRAM first, so the budget goes to what matters+ d, g = row.detail, row.gpu+ mem = (g.shown_vram or 0) + g.shared+ if (row.pid in deep_pids and (deadline is None or time.perf_counter() < deadline)+ and self.inspector.needs_scan(d, mem)):+ self.inspector.scan(d, mem)+ row.kind = classify(d, g.engines)+ # Old Ollama: "ollama.exe runner ..."; newer Ollama: its own lib\ollama\llama-server.exe+ exe = d.exe.lower()+ if row.kind in AI_KINDS and ("\\ollama\\" in exe or "/ollama/" in exe or+ (d.name.lower().startswith("ollama") and "runner" in d.cmdline)):+ ollama_runners.append(row)for row in snap.procs:if row.host:continue@@ -243,10 +245,9 @@ class Collector:own = self.prober.get(row.detail.ports)host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end processlog = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else None- inferred = row.gpu.inferred- row.llm = build_view(row.detail, row.gpu.dedicated if inferred is None else inferred, row.gpu.shared,+ row.llm = build_view(row.detail, row.gpu.shown_vram or 0, row.gpu.shared,own or host, log, host=host if own else None,- attributed=row.gpu.attributed or inferred is not None)+ attributed=row.gpu.shown_vram is not None)self.inspector.prune(set(procs))return snap@@ -295,6 +296,7 @@ class Collector:at = next((i + 1 for i, r in enumerate(host_rows) if r.kind == "wsl-vm"), 0)if len(vm_workers) == 1 and self.tracker is not None:self._attribute(snap, luid, hs.contexts.get(luid, {}).get(vm_workers[0], []))+ snap.procs.sort(key=lambda r: -(r.gpu.shown_vram or 0))snap.procs = host_rows[:at] + snap.procs + host_rows[at:]def _attribute(self, snap: Snapshot, luid: int, contexts: list[float]) -> None:@@ -303,7 +305,6 @@ class Collector:holders = [(r.pid, r.detail.create_time) for r in linux] + [(os.getpid(), self._own_start)]guesses, reason = self.tracker.update(luid, contexts, holders)snap.wsl_split = reason- self._inferred = {pid: g.bytes for pid, g in guesses.items()}for r in linux:g = guesses.get(r.pid)if g is not None:diff --git a/src/gputaskman/pdh.py b/src/gputaskman/pdh.pyindex 1f183e2..4116ec6 100644--- a/src/gputaskman/pdh.py+++ b/src/gputaskman/pdh.py@@ -72,18 +72,18 @@ class ProcGpu:shared: float = 0.0 # bytes resident in shared (system RAM) GPU memory -- "spill" ("Non Local Usage")committed: float = 0.0 # all GPU memory this process allocated, resident or not ("Total Committed")engines: dict[str, float] = field(default_factory=lambda: defaultdict(float)) # engtype -> util %- # False when the OS knows this process uses the GPU but cannot say how much (WSL: the Windows host- # charges every Linux process's VRAM to the VM). Numbers are then unknown, not zero.+ # False when per-process VRAM is unavailable. Native NVML may still report engines independently;+ # WSL charges every Linux process's VRAM and engine activity to the Windows VM worker.attributed: bool = True# WSL: VRAM matched from the VM's GPU contexts (see wslhost.ContextTracker). An exact counter value whose# owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.inferred: float | None = None- inferred_by: str = "" # "consistent" (only one explanation fits) | "order" (start-order guess)+ inferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)@property- def shown_vram(self) -> float:- """The VRAM figure a row displays: measured, else inferred (WSL), else 0 for sorting."""- return self.dedicated if self.attributed else (self.inferred or 0.0)+ def shown_vram(self) -> float | None:+ """Measured or inferred bytes for display and priority, not for summing adapter totals."""+ return self.dedicated if self.attributed else self.inferred@propertydef paged_out(self) -> float:diff --git a/src/gputaskman/report.py b/src/gputaskman/report.pyindex 4c54f4d..a9895f6 100644--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@ -6,9 +6,10 @@ import dataclassesimport jsonimport os-from .collector import Snapshot+from .collector import ProcRow, Snapshotfrom .llm import gbfrom .nvml import format_metric+from .pdh import ProcGpuWSL_HOST_NOTE = ("WSL + Windows host counters: 'WSL VM' is the exact total of Linux GPU memory (all distros). "@@ -29,20 +30,27 @@ WSL_NOTE = ("WSL: VRAM and engine load are device-wide. The Windows host charges"VRAM to the VM, so rows below are processes using the GPU, not their share of it.")-def vram_label(g) -> str:+def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:+ return s.backend == "wddm" or row.origin == "windows"+++def has_engine_counters(s: Snapshot, row: ProcRow) -> bool:+ return s.backend != "wsl" or row.origin == "windows"+++def vram_label(g: ProcGpu) -> str:"""Measured size; '*' / '?' when the size is exact but which local process owns it is inferred."""mark = {"order": "?", "consistent": "*"}.get(g.inferred_by, "")- if g.attributed:- return gb(g.dedicated) + mark- return "n/a" if g.inferred is None else gb(g.inferred) + mark+ return "n/a" if g.shown_vram is None else gb(g.shown_vram) + markINFERRED_NOTE = ("* matched: the size is exact (Windows' counter for a WSL VM GPU context, or NVML for a host PID ""seen from a container); which local process owns it is inferred, and only one explanation fits ""(or every one that fits gives the same size).")ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer processes are paired with newer "- "GPU contexts / PIDs and outside ones are assumed oldest. Becomes * with evidence, such as a process "- "starting while gputaskman runs.")+ "GPU contexts / PIDs and outside ones are assumed oldest. Cached foreign-size hints only rank extra "+ "candidates; they never prove ownership. Becomes * with evidence, such as a process starting while "+ "gputaskman runs.")def inferred_note(g) -> str:@@ -53,20 +61,18 @@ def to_dict(s: Snapshot) -> dict:nv = dataclasses.asdict(s.nv) if s.nv else Noneprocs = []for p in s.procs:- # Unknown is null, not 0: WSL Linux rows have no per-process counters, and spill/commit only exist- # in WDDM (native Windows, or Windows processes seen from WSL).- known = p.gpu.attributed- wddm = known and (s.backend == "wddm" or p.origin == "windows")+ wddm = has_wddm_counters(s, p)+ engines = has_engine_counters(s, p)item = {"pid": p.pid, "host": p.origin if p.host else None, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,"vram_inferred_by": p.gpu.inferred_by or None,- "vram_bytes": int(p.gpu.dedicated) if known else None,+ "vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed else None,"shared_bytes": int(p.gpu.shared) if wddm else None,"committed_bytes": int(p.gpu.committed) if wddm else None,"paged_out_bytes": int(p.gpu.paged_out) if wddm else None,- "gpu_pct": round(p.gpu.gpu_pct, 1) if known else None,- "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if known else None,+ "gpu_pct": round(p.gpu.gpu_pct, 1) if engines else None,+ "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if engines else None,"ports": p.detail.ports, "inside": p.inside,"model_files": [dataclasses.asdict(m) for m in p.detail.models],"installed_models": [dataclasses.asdict(m) for m in p.detail.candidates],@@ -123,10 +129,11 @@ def to_text(s: Snapshot, limit: int = 30) -> str:w = max([7] + [len(p.pid_label) for p in shown]) # host PIDs like W:120880 are widerout.append(f"{'PID':>{w}} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")for p in shown:- spill = gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""- paged = gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""+ wddm = has_wddm_counters(s, p)+ spill = gb(p.gpu.shared) if wddm and p.gpu.shared >= 1 << 20 else ""+ paged = gb(p.gpu.paged_out) if wddm and p.gpu.paged_out >= 1 << 20 else ""pid = p.pid_label- pct = f"{p.gpu.gpu_pct:.0f}" if p.gpu.attributed else "n/a"+ pct = f"{p.gpu.gpu_pct:.0f}" if has_engine_counters(s, p) else "n/a"out.append(f"{pid:>{w}} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {vram_label(p.gpu):>9} {spill:>8} "f"{paged:>8} {pct:>5} {p.inside}")if any(p.gpu.inferred_by == "consistent" for p in shown):diff --git a/src/gputaskman/wslhost.py b/src/gputaskman/wslhost.pyindex 854d660..0dd0718 100644--- a/src/gputaskman/wslhost.py+++ b/src/gputaskman/wslhost.py@@ -306,7 +306,8 @@ class ContextTracker:Contexts are aligned sample to sample (survivors keep their order). A match becomes *sure* only fromevidence: a new Linux GPU process and a new context appearing together, elimination (as many unsurecontexts as unsure processes, and just one process), contexts seen at startup while no other Linux process- holds the GPU (foreign; cached per WSL boot), or this monitor's own 0-byte context being the newest at startup.+ holds the GPU, or this monitor's own context being the only 0-byte candidate at startup.+ Cached foreign sizes help rank guesses but never establish ownership across runs.Everything else is an order guess (newest process <-> newest context, foreign assumed oldest),recomputed each sample and never used as evidence."""@@ -314,7 +315,6 @@ class ContextTracker:PENDING = 5.0 # seconds a new process may take to create its contextGONE = 5.0 # seconds an exited process's context may linger in the (older) host sampleTOL_MIN = 64 * MiB- SELF_MAX = 1 * MiB # an NVML-only context (this monitor) holds 0 bytesTOL_REL = 0.10def __init__(self, path: str | None = None, boot_id: str | None = None, self_pid: object = None) -> None:@@ -400,6 +400,7 @@ class ContextTracker:self._reset()start = dict(holders)first = self.entries is None+ had_foreign = any(e.label == FOREIGN and e.sure for e in self.entries or [])cur = [_Entry(v) for v in contexts]fresh: list[_Entry] = [] # contexts that appeared since the previous sampleif not first:@@ -446,20 +447,13 @@ class ContextTracker:if e.label is None:e.label, e.sure = FOREIGN, Trueelif first:- # Our NVML-only context is 0 bytes and newest at startup, unless another process started after us:- # its CUDA context could then be the newest one (and small while it initializes).- later = any(t >= start[self.self_pid] for p, t in start.items() if p != self.self_pid) \- if self.self_pid in start else True- if not later and cur and cur[0].value <= self.SELF_MAX:- cur[0].label, cur[0].sure = self.self_pid, True- if set(start) == {self.self_pid} and cur and cur[0].label == self.self_pid:- for e in cur[1:]: # nothing else in this distro holds the GPU: the rest is foreign- e.label, e.sure = FOREIGN, True- for v in self.cached_foreign:- free = [e for e in reversed(cur) if e.label is None and abs(e.value - v) <= self._tol(v)]- if free:- best = min(free, key=lambda e: abs(e.value - v))- best.label, best.sure = FOREIGN, True+ zeros = [e for e in cur if e.value == 0]+ if self.self_pid in start and len(zeros) == 1:+ zeros[0].label, zeros[0].sure = self.self_pid, True+ if set(start) == {self.self_pid} and zeros:+ for e in cur:+ if e.value > 0: # NVML alone holds no VRAM; other zero-byte owners remain ambiguous+ e.label, e.sure = FOREIGN, Trueelif fresh:pending = sorted((p for p in start if p not in labeled() and p not in self.initialand now - self.seen[p] <= self.PENDING),@@ -473,6 +467,10 @@ class ContextTracker:unsure = sorted((p for p in start if p not in labeled()), key=lambda p: -age(p))free = [e for e in cur if e.label is None]+ if unsure and len(free) > len(unsure):+ # Cached sizes may outlive their owners: rank guesses, never exclude candidates or pin labels.+ free.sort(key=lambda e: e.value > 0 and any(+ v > 0 and abs(e.value - v) <= self._tol(v) for v in self.cached_foreign))reason = ""if unsure:if len(free) < len(unsure):@@ -492,12 +490,11 @@ class ContextTracker:e.label, e.same = pid, sameself.entries = cur- # Foreign labels only come from evidence, so the cache follows them, including down to none: a cached- # context that exited must not later claim a local context of the same size.- foreign = [e.value for e in cur if e.label == FOREIGN and e.sure]- if sorted(foreign) != sorted(self.cached_foreign):- self.cached_foreign = foreign- self._save()+ foreign = [e.value for e in cur if e.label == FOREIGN and e.sure and e.value > 0]+ if not start or set(start) == {self.self_pid} or foreign or had_foreign:+ if sorted(foreign) != sorted(self.cached_foreign):+ self.cached_foreign = foreign+ self._save()out = {e.label: ContextGuess(e.value, "consistent" if e.sure or e.same else "order")for e in cur if e.label in start}return out, reasondiff --git a/tests/test_core.py b/tests/test_core.pyindex 3333bf1..ed140cd 100644--- a/tests/test_core.py+++ b/tests/test_core.py@@ -16,7 +16,7 @@ from rich.console import Consolefrom rich.table import Tablefrom textual.widgets import Footer, Select, Static-from gputaskman.app import GpuTaskManager, spark+from gputaskman.app import GpuTaskManager, spark, stacked_barfrom gputaskman.cli import mainfrom gputaskman.collector import ProcRow, Snapshot, select_adapter, select_nv_devicefrom gputaskman.dxgi import Adapter@@ -127,6 +127,16 @@ class DeviceSelectionTests(unittest.TestCase):class MemoryTests(unittest.TestCase):+ def test_best_vram_keeps_measured_and_inferred_zero_distinct_from_unknown(self):+ gpu = ProcGpu(dedicated=0, inferred=8 * GiB)+ self.assertEqual(gpu.shown_vram, 0)+ gpu.attributed = False+ self.assertEqual(gpu.shown_vram, 8 * GiB)+ gpu.inferred = 0+ self.assertEqual(gpu.shown_vram, 0)+ gpu.inferred = None+ self.assertIsNone(gpu.shown_vram)+def test_local_counters_are_preferred(self):self.assertTrue(GpuCounters._COUNTERS["dedicated"][0].endswith(r"\Local Usage"))self.assertTrue(GpuCounters._COUNTERS["shared"][0].endswith(r"\Non Local Usage"))@@ -672,6 +682,48 @@ class UiTests(unittest.IsolatedAsyncioTestCase):Console(file=output, width=widget.content_size.width, color_system=None).print(widget.content)return output.getvalue()+ async def test_wsl_sort_selection_and_bar_use_inferred_vram_without_double_counting(self):+ s = snapshot()+ s.backend = "wsl"+ small = ProcRow(10, "small", "LLM", ProcGpu(attributed=False, inferred=2 * GiB, inferred_by="order"),+ ProcDetail(10, "small"), LlmView())+ large = ProcRow(20, "large", "LLM", ProcGpu(attributed=False, inferred=8 * GiB, inferred_by="order"),+ ProcDetail(20, "large"), LlmView())+ vm = ProcRow(30, "WSL VM", "wsl-vm", ProcGpu(dedicated=10 * GiB), ProcDetail(30, "WSL VM"),+ host=True, origin="windows")+ host = ProcRow(40, "browser.exe", "browser", ProcGpu(dedicated=3 * GiB), ProcDetail(40, "browser.exe"),+ host=True, origin="windows")+ s.procs = [small, large, host, vm]+ with patch.object(GpuTaskManager, "poll"), patch("gputaskman.app.stacked_bar", wraps=stacked_bar) as bar:+ app = GpuTaskManager()+ async with app.run_test(size=(180, 42)) as pilot:+ app.apply(s)+ await pilot.pause()+ self.assertEqual(app.selected_key, large.key)+ self.assertEqual([r.key for r in app.visible_rows(s)], [vm.key, large.key, small.key, host.key])+ self.assertEqual(bar.call_args.args[1], 8 * GiB)+ table = app.query_one("#procs")+ self.assertEqual(table.get_cell(large.key, "VRAM").plain, "8.00 GB?")+ for column in ("GPU%", "3D", "Copy", "Video"):+ self.assertEqual(table.get_cell(large.key, column).plain, "n/a")+ self.assertEqual(s.attributed, 13 * GiB)++ async def test_native_hidden_vram_keeps_engine_reading_in_ui(self):+ s = snapshot()+ s.backend = "nvml"+ s.procs = [ProcRow(10, "python3", "AI", ProcGpu(attributed=False, engines={"compute": 65}),+ ProcDetail(10, "python3"))]+ with patch.object(GpuTaskManager, "poll"):+ app = GpuTaskManager()+ app.selected_key = "10"+ async with app.run_test(size=(180, 42)) as pilot:+ app.apply(s)+ await pilot.pause()+ self.assertEqual(app.query_one("#procs").get_cell("10", "GPU%").plain, "65")+ hdr = app.query_one("#inspector", Static).content.renderables[0].plain+ self.assertIn("compute 65%", hdr)+ self.assertIn("spill(RAM) n/a", hdr)+async def test_eighth_chart_records_sm_clock_with_scale_and_missing_readings(self):with patch.object(GpuTaskManager, "poll"):app = GpuTaskManager()diff --git a/tests/test_linux.py b/tests/test_linux.pyindex 0e69949..2196ff5 100644--- a/tests/test_linux.py+++ b/tests/test_linux.py@@ -7,7 +7,7 @@ from unittest.mock import patchfrom gputaskman import linuxfrom gputaskman.collector import ProcRow, Snapshotfrom gputaskman.gguf import GgufInfo-from gputaskman.llm import build_view+from gputaskman.llm import LlmView, build_viewfrom gputaskman.pdh import ProcGpufrom gputaskman.procinfo import ProcDetail, _lib_name, _match_runtimes, classifyfrom gputaskman.report import to_dict, to_text@@ -275,7 +275,8 @@ class UnattributedReportTests(unittest.TestCase):def test_reports_say_unknown_instead_of_zero(self):data = to_dict(self.snapshot())self.assertEqual(data["backend"], "wsl")- self.assertIsNone(data["processes"][0]["vram_bytes"])+ for key in ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):+ self.assertIsNone(data["processes"][0][key], key)self.assertFalse(data["processes"][0]["vram_attributed"])text = to_text(self.snapshot())row = next(line for line in text.splitlines() if "4242" in line)@@ -306,6 +307,44 @@ class UnattributedReportTests(unittest.TestCase):key = GpuTaskManager.SORTS[0][1]self.assertEqual([r.pid for r in sorted(rows, key=key)], [2, 1, 3])+ def test_native_hidden_vram_does_not_hide_measured_engine_load(self):+ snap = self.snapshot()+ snap.backend = "nvml"+ snap.procs[0].gpu.engines["compute"] = 47+ row = to_dict(snap)["processes"][0]+ self.assertIsNone(row["vram_bytes"])+ self.assertEqual(row["gpu_pct"], 47)+ self.assertEqual(row["engines"], {"compute": 47})+ for key in ("shared_bytes", "committed_bytes", "paged_out_bytes"):+ self.assertIsNone(row[key], key)+ text = next(line for line in to_text(snap).splitlines() if "4242" in line)+ self.assertIn("47", text)++ def test_measured_zero_is_not_changed_to_unknown(self):+ snap = self.snapshot()+ snap.backend = "nvml"+ snap.procs[0].gpu.attributed = True+ row = to_dict(snap)["processes"][0]+ self.assertEqual(row["vram_bytes"], 0)+ self.assertEqual(row["gpu_pct"], 0)+ self.assertEqual(row["engines"], {})+ self.assertIsNone(row["shared_bytes"])+ snap.backend = "wsl"+ snap.procs[0].host, snap.procs[0].origin = True, "windows"+ row = to_dict(snap)["processes"][0]+ for key in ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct"):+ self.assertEqual(row[key], 0, key)++ def test_model_detail_header_uses_inferred_or_unknown_vram(self):+ for inferred, label in ((None, "n/a"), (0, "0 MB*"), (3 * GiB, "3.00 GB*")):+ with self.subTest(inferred=inferred):+ snap = self.snapshot()+ p = snap.procs[0]+ p.llm = LlmView()+ p.gpu.inferred, p.gpu.inferred_by = inferred, "consistent"+ self.assertIn(f"== python3 (4242) VRAM {label}", to_text(snap))+ self.assertIsNone(to_dict(snap)["processes"][0]["vram_bytes"])+def test_empty_wsl_view_explains_itself(self):snap = self.snapshot()snap.procs = []diff --git a/tests/test_wslhost.py b/tests/test_wslhost.pyindex 13cf4f1..38bfd1a 100644--- a/tests/test_wslhost.py+++ b/tests/test_wslhost.py@@ -6,13 +6,14 @@ import osimport tempfileimport timeimport unittest-from unittest.mock import mock_open, patch+from unittest.mock import Mock, mock_open, patchfrom gputaskman import collector as collector_modfrom gputaskman import wslhostfrom gputaskman.collector import Collector, ProcRow, Snapshot+from gputaskman.llm import LlmViewfrom gputaskman.pdh import AdapterGpu, ProcGpu-from gputaskman.procinfo import ProcDetail+from gputaskman.procinfo import MappedModel, ProcDetailfrom gputaskman.report import to_dict, to_textfrom gputaskman.wslhost import ContextTracker, HostSample, HostStream, match_adapter, parse_line@@ -171,8 +172,8 @@ class TrackerTests(unittest.TestCase):def test_a_workload_started_after_us_is_not_taken_for_our_context(self):t = self.make()h = [("S", 900), ("W", 950)] # W created its CUDA context between our NVML init and the first sample- self.check(self.step(t, [20 / 1024, 0.0], h), {"W": (0.02, "?")})- self.check(self.step(t, [5.0, 0.0], h), {"W": (5.0, "?")})+ self.check(self.step(t, [20 / 1024, 0.0], h), {"W": (0.02, "*")})+ self.check(self.step(t, [5.0, 0.0], h), {"W": (5.0, "*")})def test_cache_forgets_a_foreign_context_that_exited(self):t = self.make()@@ -199,10 +200,59 @@ class TrackerTests(unittest.TestCase):self.check(self.step(t, [0.0, 0.15], h), {"A": (0.15, "?")}) # A's context isn't there yetself.check(self.step(t, [6.0, 0.0, 0.15], h), {"A": (6.0, "*")})- def test_startup_with_only_this_monitor_teaches_foreign(self):+ def test_cached_foreign_sizes_only_rank_startup_guesses(self):self.step(self.make(), [0.0, 0.15], [("S", 900)]) # only gputaskman holds the GPU: 0.15 is foreignt = self.make() # a later start in the same boot, with a job already running- self.check(self.step(t, [0.0, 2.0, 0.15], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})+ self.check(self.step(t, [0.0, 0.15, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})+ self.check(self.step(t, [0.0, 0.15, 2.5], [("S", 900), ("X", 5)]), {"X": (2.5, "?")})++ def test_foreign_cache_clears_immediately_when_last_foreign_context_exits(self):+ t = self.make()+ self.step(t, [0.0, 2.0], [("S", 900)])+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [2 * GiB])+ self.step(t, [0.0], [("S", 900)])+ self.assertEqual(t.cached_foreign, [])+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [])+ self.check(self.step(self.make(), [0.0, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})++ def test_foreign_cache_clears_while_local_workload_remains(self):+ t = self.make()+ self.step(t, [0.0, 0.4], [("S", 900)])+ h = [("S", 900), ("X", 950)]+ self.step(t, [2.0, 0.0, 0.4], h)+ self.check(self.step(t, [2.0, 0.0], h), {"X": (2.0, "*")})+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [])++ def test_stale_cache_never_takes_a_local_context(self):+ self.step(self.make(), [0.0, 2.0], [("S", 900)])+ # The foreign process exits while the monitor is off; a local process reuses that size.+ self.check(self.step(self.make(), [0.0, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})++ def test_first_sample_finds_monitor_zero_instead_of_newest_small_context(self):+ t = self.make()+ h = [("S", 900), ("X", 950)]+ guesses, _ = t.update(RTX, [20 * MiB, 0], h, now=self.clock)+ self.assertEqual(guesses["S"].bytes, 0)+ self.assertEqual(guesses["X"].bytes, 20 * MiB)+ self.check(self.step(t, [5.0, 0.0], h), {"X": (5.0, "*")})++ def test_equal_zero_contexts_never_pin_monitor_ownership(self):+ t = self.make()+ h = [("S", 900), ("X", 950)]+ self.check(self.step(t, [0.0, 0.0], h), {"X": (0.0, "*")})+ self.assertFalse(any(e.sure for e in t.entries))+ self.check(self.step(t, [5.0, 0.0], h), {"X": (5.0, "?")})+ self.assertFalse(any(e.label == "S" and e.value for e in t.entries))++ def test_idle_zero_ambiguity_does_not_pollute_foreign_cache(self):+ t = self.make()+ self.step(t, [0.0, 0.15, 0.0], [("S", 900)])+ self.assertEqual(t.cached_foreign, [0.15 * GiB])+ self.assertFalse(any(e.label == "S" and e.sure for e in t.entries))+ self.check(self.step(t, [2.0, 0.0, 0.15, 0.0], [("S", 900), ("X", 950)]), {"X": (2.0, "*")})def test_a_reused_pid_is_a_new_process(self):t = self.make()@@ -221,12 +271,12 @@ class TrackerTests(unittest.TestCase):for _ in range(5):self.check(self.step(t, [0.5, 0.0, 2.12, 3.05, 0.0], h), {"D": (0.5, "*"), "C": (2.12, "?"), "B": (3.05, "?")})- def test_no_holders_teaches_foreign_and_elimination_confirms(self):+ def test_no_holders_teaches_foreign_without_confirming_future_owners(self):self.make().update(RTX, at([0.4]), [], now=self.clock) # nothing in this distro: 0.4 is foreignt = self.make() # a fresh start in the same boot, with a job already running- self.check(self.step(t, [0.0, 2.0, 0.4], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})t = self.make(boot="boot-2") # a new boot knows nothing- self.check(self.step(t, [0.0, 2.0, 0.4], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (0.4, "?")})def test_context_that_lags_its_process_is_still_pinned(self):t = self.make()@@ -278,8 +328,9 @@ class TrackerTests(unittest.TestCase):def test_different_adapter_resets_history(self):self.make().update(RTX, at([0.4]), [], now=self.clock)t = self.make()- self.check(self.step(t, [0.0, 2.0, 0.4], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})- guesses, _ = t.update(0x1234, at([0.0, 2.0, 0.4]), [("S", 900), ("X", 5)], now=self.clock + 5)+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})+ guesses, _ = t.update(0x1234, at([0.0, 0.4, 2.0]), [("S", 900), ("X", 5)], now=self.clock + 5)+ self.assertEqual(guesses["X"].bytes, 0.4 * GiB)self.assertEqual(guesses["X"].confidence, "order")@@ -307,7 +358,6 @@ class HostRowTests(unittest.TestCase):c._host_restarted = time.time()c.tracker = ContextTracker(path=os.path.join(self.tmp.name, "ctx.json"), boot_id="boot", self_pid=os.getpid())c._own_start = time.time()- c._inferred = {}return cdef setUp(self):@@ -349,11 +399,57 @@ class HostRowTests(unittest.TestCase):row = next(r for r in snap.procs if not r.host)self.assertEqual((row.gpu.inferred, row.gpu.inferred_by), (3.0 * GiB, "consistent"))self.assertFalse(row.gpu.attributed) # engines are still unknown+ row.llm = LlmView()text = to_text(snap)self.assertIn("3.00 GB*", text)+ self.assertIn("== python3 (2744) VRAM 3.00 GB*", text)self.assertIn("* matched", text)data = to_dict(snap)["processes"][1]self.assertEqual((data["vram_inferred_bytes"], data["vram_inferred_by"]), (3.0 * GiB, "consistent"))+ self.assertEqual(snap.attributed, 3 * GiB) # the Linux child is not added to the VM total again++ def test_current_inferred_vram_sets_first_sample_scan_priority(self):+ big_pid = Collector.DEEP_TOP_N + 1+ total = 8 * GiB + Collector.DEEP_TOP_N * MiB+ busy = line([["local usage", f"pid_24820_{LUID}", v]+ for v in [0, 8 * GiB] + [MiB] * Collector.DEEP_TOP_N]+ + [["dedicated usage", LUID, total]], {"24820": "vmwp"})+ c = self.collector(FakeStream(parse_line(busy, now=time.time())))+ c.adapter, c.adapters, c.gpu_index, c.scan_budget = None, [], 0, None+ c.host_status = ""+ c.nvml = Mock(ok=False, driver="", cuda_version="")+ c.counters = Mock(outside={})+ c.counters.sample.return_value = (+ {RTX: {pid: ProcGpu(attributed=False) for pid in range(1, big_pid + 1)}},+ {RTX: AdapterGpu(dedicated=total)})+ details = {pid: ProcDetail(pid, "python3", create_time=pid, parents=[])+ for pid in range(1, big_pid + 1)}+ c.inspector = Mock()+ c.inspector.get.side_effect = details.__getitem__+ c.inspector.needs_scan.return_value = True++ def scan(d, mem):+ if d.pid == big_pid:+ d.models = [MappedModel("large.gguf", 8 * GiB, 0, True, kind="LLM")]++ c.inspector.scan.side_effect = scan+ c.prober = Mock()+ c.prober.get.return_value = None+ with patch.object(collector_mod, "BACKEND", "wsl"), \+ patch.object(collector_mod, "build_view", return_value=LlmView()) as view:+ snap = c.sample()+ self.assertEqual(c.inspector.scan.call_count, Collector.DEEP_TOP_N)+ self.assertEqual(c.inspector.scan.call_args_list[0].args, (details[big_pid], 8 * GiB))+ self.assertNotIn(Collector.DEEP_TOP_N, [call.args[0].pid for call in c.inspector.scan.call_args_list])+ self.assertEqual([r.pid for r in snap.procs[:2]], [24820, big_pid])+ self.assertEqual(snap.procs[1].kind, "LLM") # classification follows the scan+ self.assertEqual(view.call_args.args[1], 8 * GiB)+ self.assertTrue(view.call_args.kwargs["attributed"])+ self.assertEqual(snap.attributed, total)+ c.scan_budget = 0.0+ c.inspector.scan.reset_mock()+ c.sample()+ c.inspector.scan.assert_not_called()def test_order_guess_is_marked(self):# an extra (foreign) context and no history: placement is a guess