A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
merge: upstream main with AMD support
Kcedcd033706b
diff --cc README.mdindex b560062,595d958..173d5f1--- a/README.md+++ b/README.md@@@ -2,8 -2,9 +2,9 @@@**An `htop` for what's inside your GPU, not just how full it is.**- Windows + NVIDIA or AMD. A terminal UI that connects per-process GPU memory and engine activity- with loaded runtimes, model files, and LLM buffer breakdowns.-Windows + NVIDIA, with Linux and WSL support (see [Linux and WSL](#linux-and-wsl)). A terminal UI++Windows + NVIDIA/AMD, with NVIDIA support on Linux and WSL (see [Linux and WSL](#linux-and-wsl)). A terminal UI+ that connects per-process GPU memory and engine activity with loaded runtimes, model files, and+ LLM buffer breakdowns.Task Manager says "15 GB used." This tries to answer: *by what? Model weights? KV cache?Your browser? The desktop compositor? And why is so much GPU memory in system RAM?*@@@ -186,8 -283,9 +300,9 @@@ but a runtime DLL or filename alone doe| Question | Evidence | Confidence / limitation ||---|---|---|-| How full and busy is the card? | WDDM adapter counters and NVML | Measured; driver support varies by metric |+| How full and busy is the card? | WDDM adapter counters and NVML or AMD ADL/SMI | Measured; driver support varies by metric |- | Which process uses GPU memory? | WDDM process counters | Measured accounting, not a resource ownership graph |+ | Which process uses GPU memory? | WDDM process counters; NVML on Linux | Measured accounting, not a resource ownership graph |+ | Which WSL process uses GPU memory? | Windows host counters for `vmwp.exe`, matched to Linux processes | Exact VM total; per-process owner inferred (`*`) or guessed (`?`) || Which engines are active? | WDDM engine counters | CUDA can appear on the **3D** engine || Which runtime is loaded? | Mapped DLL names | Detection, not proof that every library is doing GPU work || Which model file is present? | Mapped files, CLI flags, GGUF magic/header | A file mapping does not prove GPU placement |@@@ -367,8 -466,14 +483,14 @@@ file paths, but model and process name| Symptom | What to check ||---|---|| `uv` or `git` is not recognized | Install the prerequisites, then open a new terminal |- | Windows-only message | Run on native Windows, not Linux, macOS, or WSL |+| Empty or incomplete GPU readings | Check the GPU driver and WDDM 2.0+ support in `dxdiag`; use a normal local desktop session |+ | "supports Windows and Linux/WSL only" | macOS is not supported |+ | WSL: `Windows host counters: off (WSL interop is disabled)` | Enable `[interop]` in `/etc/wsl.conf`, then `wsl --shutdown`; or accept device-wide numbers |+ | WSL: `starting` for a few seconds | The first Windows counter sample takes about 2.5 s |+ | WSL: `unmatched` | No Windows adapter's usage matched the NVML device; multi-GPU WSL is not validated |+ | WSL: Linux rows show `?` | Processes were running before gputaskman started; they become `*` with evidence (see the matching table) |+ | Linux: a GPU process is missing | Processes owned by other users need `sudo`; Docker and other WSL distros are not visible from this one |+ | Linux container: `H:<pid>` rows | GPU processes outside this container's PID namespace (host or other containers); NVML sees them, the container cannot name them |-| Empty or incomplete GPU readings | Check the NVIDIA driver and WDDM 2.0+ support in `dxdiag`; use a normal local desktop session || Missing DLLs or model details | Protected processes may deny inspection; basic counters can still work without elevation || No exact LLM breakdown | Enable compatible buffer logging, or use the GGUF estimate; API/log access is optional || The screen looks cramped | Widen the terminal or use `--once` / `--json` |@@@ -395,15 -501,20 +518,20 @@@ uv run --locked --group release python# On a native ARM64 interpreter, use --arch arm64 instead.```+ The same command on Linux (`tools/build_portable.py`) produces a `.tar.gz`, which keeps the+ executable bit. The builder refuses to package NVIDIA driver libraries (`libnvidia-ml.so`,+ `libcuda.so`) on either platform; they come from the installed driver.+The builder verifies the interpreter and executable architectures, includes Textual resourcesand dependency licenses, and writes the complete portable ZIP and SHA-256 file to `dist`.-It does not bundle Windows/NVIDIA DLLs, inference servers, or models. Keep uv for development;+It does not bundle Windows or GPU driver DLLs, inference servers, or models. Keep uv for development;the executable ZIP is for running the monitor without a Python environment.- The **Portable Windows release** GitHub Actions workflow builds on `windows-2022` (x64) and- `windows-11-arm` (ARM64). A manual run produces downloadable workflow artifacts without publishing- a release. A pushed `vVERSION` tag must match `project.version` in `pyproject.toml`; once both builds- pass, the workflow publishes both ZIPs and checksums together with generated release notes.+ The **Portable release** GitHub Actions workflow builds on `windows-2022` (x64), `windows-11-arm`+ (ARM64), `ubuntu-22.04` (Linux x64), and `ubuntu-22.04-arm` (Linux ARM64). Linux builds use the+ oldest supported Ubuntu because the frozen binary needs the build machine's glibc or newer. A manual run produces downloadable workflow artifacts without publishing+ a release. A pushed `vVERSION` tag must match `project.version` in `pyproject.toml`; once all builds+ pass, the workflow publishes all four archives and checksums together with generated release notes.Build jobs have read-only permissions; only the tag-triggered publish job can write releases.CI checks the extracted executable's `--help` outside the checkout with Python removed from `PATH`;interactive rendering and real GPU snapshots require local hardware testing.@@@ -420,8 -531,9 +548,10 @@@ downloads are bundled in this repositor| Module | Responsibility ||---|---|| `pdh.py` / `dxgi.py` | Windows counters and adapter enumeration |+ | `linux.py` | Linux backend: NVML adapters and per-process usage, container PID matching, WSL detection, `/dev/dxg` holders |+ | `wslhost.py` | WSL: interop check, Windows host counter stream, adapter pairing, context matching || `nvml.py` | NVIDIA device readings |+| `adl.py`, `amdsmi.py` | Optional AMD device readings || `procinfo.py` | Runtime, mapped-file, and process discovery || `stores.py` | Friendly names from Ollama, LM Studio, and Hugging Face model stores || `gguf.py` | Model metadata, kind, and size estimates |diff --cc src/gputaskman/app.pyindex f022019,f80f533..8292a48--- a/src/gputaskman/app.py+++ b/src/gputaskman/app.py@@@ -262,12 -278,11 +287,12 @@@ class GpuTaskManager(App)for history in (self.hist_util, self.hist_clock, self.hist_bw, self.hist_temp,self.hist_power, self.hist_enc, self.hist_dec):history.append(None)- self.hist_vram.append(100 * snap.vram_used / snap.vram_total if snap.vram_total else None)+ self.hist_vram.append(100 * snap.vram_used / snap.vram_total+ if snap.vram_total and snap.vram_used is not None else None)if not self._auto_selected:- llm = next((r for r in sorted(snap.procs, key=lambda r: -r.gpu.dedicated) if r.llm), None)+ llm = next((r for r in sorted(snap.procs, key=self.SORTS[0][1]) if r.llm), None)if llm:- self.selected_pid = llm.pid+ self.selected_key = llm.keyself._auto_selected = Trueself.query_one("#device", Static).update(self.render_device(snap))self.render_charts()@@@ -286,12 -300,13 +311,13 @@@# Fit category proportions to the adapter total; row samples need not sum exactly.by_kind: dict[str, float] = {}for r in s.procs:- by_kind[r.kind] = by_kind.get(r.kind, 0) + r.gpu.dedicated+ if r.gpu.attributed: # unknown shares stay in "unattributed" instead of showing as 0+ by_kind[r.kind] = by_kind.get(r.kind, 0) + r.gpu.dedicatedattributed = sum(by_kind.values())- scale = min(1.0, s.vram_used / attributed) if attributed else 1.0+ scale = min(1.0, s.vram_used / attributed) if attributed and s.vram_used is not None else 1.0order = sorted(by_kind.items(), key=lambda kv: -kv[1])- parts = [(v * scale, KIND_STYLE.get(k, "white")) for k, v in order]- other = max(s.vram_used - attributed * scale, 0)+ parts = [(v * scale, KIND_STYLE.get(k, "white")) for k, v in order] if s.vram_used is not None else []+ other = max(s.vram_used - attributed * scale, 0) if s.vram_used is not None and s.attributed is not None else 0parts.append((other, "grey35"))width = max(min(self.size.width - 40, 90), 20)vram = Text("VRAM ")@@@ -418,14 -436,13 +456,17 @@@for r in rows:e = r.gpu.engines- spill = Text("N/A" if "shared" not in r.gpu.available else gb(r.gpu.shared) if r.gpu.shared >= 1 << 20 else "", justify="right",+ wddm = has_wddm_counters(s, r)+ engines = has_engine_counters(s, r)+ spill = Text(gb(r.gpu.shared) if wddm and r.gpu.shared >= 1 << 20 else "", justify="right",style="bold red" if r.gpu.shared >= 256 << 20 and r.kind in ("LLM", "AI") else "yellow")- paged = Text("N/A" if not r.gpu.paged_available else gb(r.gpu.paged_out) if r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")- vram = gb(r.gpu.dedicated) if "dedicated" in r.gpu.available else "N/A"- activity = pct(r.gpu.gpu_pct) if "engine" in r.gpu.available else Text("N/A", justify="right")- engine = lambda value: pct(value) if "engine" in r.gpu.available else Text("N/A", justify="right")+ paged = Text(gb(r.gpu.paged_out) if wddm and r.gpu.paged_out >= 1 << 20 else "", justify="right", style="grey62")++ if wddm and "shared" not in r.gpu.available:++ spill = Text("n/a", justify="right", style="grey62")++ if wddm and not r.gpu.paged_available:++ paged = Text("n/a", justify="right", style="grey62")t.add_row(- Text(str(r.pid), justify="right"),+ Text(r.pid_label, justify="right", style="grey62" if r.host else ""),Text(r.name),Text(r.kind, style=KIND_STYLE.get(r.kind, "")),Text(" ".join(r.detail.runtimes[:3]), style="grey70"),@@@ -449,13 -470,16 +494,16 @@@if r is None:panel.update(Text("Select a process (↑/↓) to see what's inside its GPU memory. Press ? to learn what's knowable.", style="grey62"))return- panel.border_title = f"{r.name} ({r.pid}) - {r.kind}"+ where = {"windows": "Windows PID ", "outside": "host PID "}.get(r.origin, "") if r.host else ""+ panel.border_title = f"{r.name} ({where}{r.pid}) - {r.kind}"out: list = []e = r.gpu.engines- hdr = Text.assemble(("VRAM ", "bold"), (gb(r.gpu.dedicated) if "dedicated" in r.gpu.available else "N/A", "bold"), " spill(RAM) ",- (gb(r.gpu.shared) if "shared" in r.gpu.available else "N/A", "bold red" if r.gpu.shared > 256 << 20 else ""),- " paged(est) ", gb(r.gpu.paged_out) if r.gpu.paged_available else "N/A",- " engines ", (" ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle") if "engine" in r.gpu.available else "N/A")+ wddm = has_wddm_counters(s, r)+ engines = " ".join(f"{k} {v:.0f}%" for k, v in e.items() if v >= 0.5) or "idle"+ hdr = Text.assemble(("VRAM ", "bold"), (vram_label(r.gpu), "bold"), " spill(RAM) ",- (gb(r.gpu.shared) if wddm else "n/a", "bold red" if r.gpu.shared > 256 << 20 else ""),- " paged(est) ", gb(r.gpu.paged_out) if wddm else "n/a",++ (gb(r.gpu.shared) if wddm and "shared" in r.gpu.available else "n/a", "bold red" if r.gpu.shared > 256 << 20 else ""),++ " paged(est) ", gb(r.gpu.paged_out) if wddm and r.gpu.paged_available else "n/a",+ " engines ", engines if has_engine_counters(s, r) else "n/a")if r.detail.runtimes:hdr.append(" runtime ", style="grey62")hdr.append(" ".join(r.detail.runtimes), style="cyan")diff --cc src/gputaskman/cli.pyindex bb2fa81,737de75..182691d--- a/src/gputaskman/cli.py+++ b/src/gputaskman/cli.py@@@ -13,7 -13,8 +13,9 @@@ def main() -> Noneap.add_argument("--once", action="store_true", help="print one snapshot and exit (no TUI)")ap.add_argument("--json", action="store_true", help="print one JSON snapshot and exit (implies --once)")ap.add_argument("--gpu", type=int, default=0, help="GPU index (default 0)")+ ap.add_argument("--list-gpus", action="store_true", help="list selectable GPU adapters and exit")+ ap.add_argument("--no-host", action="store_true",+ help="WSL: don't read Windows host GPU counters through interop (WSL VM total, Windows apps)")ap.add_argument("-i", "--interval", type=float, default=1.0, help="refresh seconds (default 1)")args = ap.parse_args()if args.gpu < 0:@@@ -21,16 -22,9 +23,16 @@@if not math.isfinite(args.interval) or args.interval <= 0:ap.error("--interval must be a finite number greater than zero")- if sys.platform != "win32":- sys.exit("gputaskman currently supports Windows (WDDM) only.")+ if sys.platform != "win32" and not sys.platform.startswith("linux"):+ sys.exit("gputaskman supports Windows (WDDM) and Linux/WSL (NVML) only.")+ if args.list_gpus:+ from .collector import enumerate_adapters, ordered_adapters++ for i, adapter in enumerate(ordered_adapters(enumerate_adapters())):+ print(f"{i}: {adapter.name} ({adapter.vendor}, LUID 0x{adapter.luid & 0xffffffffffffffff:016x})")+ return+if args.gpu:from .collector import enumerate_adapters, select_adapterdiff --cc src/gputaskman/collector.pyindex 1e3829b,331e935..fef6b9f--- a/src/gputaskman/collector.py+++ b/src/gputaskman/collector.py@@@ -8,16 -9,26 +9,29 @@@ from dataclasses import dataclass, fielimport psutil+from .amdsmi import AmdSmi, _bdf+from .adl import Adl- from .dxgi import VENDOR_NVIDIA, Adapter, enumerate_adapters+ from .dxgi import VENDOR_NVIDIA, Adapterfrom .handy import HandyEvidence, inspect_handyfrom .llm import LlmView, ServerProber, build_view, gb, ollama_log_pathfrom .nvml import Nvml, NvDevice- from .pdh import GpuCounters, ProcGpu+ from .pdh import ProcGpufrom .procinfo import AI_KINDS, ProcDetail, ProcInspector, classify+from .telemetry import DeviceTelemetry+ if sys.platform == "win32":+ from .dxgi import enumerate_adapters+ from .pdh import GpuCounters++ BACKEND = "wddm"+ else:+ from .linux import LinuxCounters as GpuCounters+ from .linux import enumerate_adapters, is_wsl++ BACKEND = "wsl" if is_wsl() else "nvml"++ from .wslhost import ContextTracker, HostStream, interop_enabled, match_adapter+MiB = 1 << 20@@@ -76,19 -100,31 +103,31 @@@ class Snapshotdriver: strcuda: strvram_total: int- vram_used: int # adapter-wide dedicated usage (WDDM counters, or NVML on Linux): the ground truth- shared_used: int+ vram_used: int | None # WDDM adapter-wide dedicated usage+ shared_used: int | Noneprocs: list[ProcRow] = field(default_factory=list)+ backend: str = "wddm" # wddm (Windows) | nvml (native Linux) | wsl (device-wide only)+ host_status: str = "" # WSL only: Windows host counters via interop -- ok | starting | off (...) | ...+ wsl_split: str = "" # WSL only: why Linux rows have no (inferred) VRAM, when they don't++ @property+ def host_data(self) -> bool:+ return self.host_status.startswith("ok")@property- def attributed(self) -> int:+ def attributed(self) -> int | None:+ if any("dedicated" not in p.gpu.available for p in self.procs):+ return Nonereturn int(sum(p.gpu.dedicated for p in self.procs))+ def _is_live(g: ProcGpu) -> bool:+ return (not g.attributed or g.dedicated >= MiB or g.gpu_pct >= 0.5 or g.shared >= 16 * MiB+ or g.paged_out >= 16 * MiB)++def select_adapter(adapters: list[Adapter], gpu_index: int) -> Adapter | None:- hardware = [a for a in adapters if not a.software]- choices = [a for a in hardware if a.vendor_id == VENDOR_NVIDIA] or hardware- choices.sort(key=lambda a: -a.dedicated_vram)+ choices = ordered_adapters(adapters)if gpu_index < 0:raise ValueError("--gpu must be a non-negative index")if gpu_index >= len(choices):@@@ -109,15 -138,11 +148,19 @@@ def select_nv_device(adapter: Adapterif adapter:if adapter.vendor_id != VENDOR_NVIDIA:return None- matches = [d for d in devices if d.name.casefold() == adapter.name.casefold()]++ if sys.platform != "win32":++ from .linux import pci_key++ matches = [d for d in devices if d.pci_bus and pci_key(d.pci_bus) == adapter.luid]++ return matches[0] if len(matches) == 1 else None+ if adapter.pci_address is None:+ return None+ if adapters is not None and sum(a.pci_address == adapter.pci_address for a in adapters) != 1:+ return None+ matches = [d for d in devices if _bdf(d.pci_bus) == adapter.pci_address and d.driver_model != "MCDM"]if len(matches) == 1:return matches[0]- fallback = devices[gpu_index] if gpu_index < len(devices) else None- return None if adapter and fallback and fallback.driver_model == "MCDM" else fallback+ return None+ return devices[gpu_index] if gpu_index < len(devices) else Noneclass Collector:@@@ -164,48 -199,119 +221,123 @@@snap = Snapshot(taken=time.time(), adapter=self.adapter, adapters=self.adapters, nv=nv,- driver=self.nvml.driver, cuda=self.nvml.cuda_version, vram_total=total,- vram_used=int(a.dedicated) if a else (nv.mem_used if nv else 0),- shared_used=int(a.shared) if a else 0, backend=BACKEND,+ driver=(self.amdsmi.driver if nv and nv.source == "AMD SMI" else self.nvml.driver if nv and nv.source == "NVML" else ""),+ cuda=self.nvml.cuda_version if nv and nv.source == "NVML" else "", vram_total=total,+ vram_used=int(a.dedicated) if a and a.dedicated is not None else None,- shared_used=int(a.shared) if a and a.shared is not None else None,++ shared_used=int(a.shared) if a and a.shared is not None else None, backend=BACKEND,)- live = [(pid, g) for pid, g in procs.items()- if g.dedicated >= MiB or g.gpu_pct >= 0.5 or g.shared >= 16 * MiB or g.paged_out >= 16 * MiB]- live.sort(key=lambda x: -x[1].dedicated)- deep_pids = {pid for pid, _ in live[: self.DEEP_TOP_N]} | {pid for pid, g in live if g.gpu_pct >= 2}+ snap.host_status = self.host_status+ live = [(pid, g) for pid, g in procs.items() if _is_live(g)]+ live.sort(key=lambda x: -(x[1].shown_vram or 0))+ for pid, g in live:+ d = self.inspector.get(pid)+ row = ProcRow(pid=pid, name=d.name, kind=classify(d, g.engines), gpu=g, detail=d)+ snap.procs.append(row)++ for pid, g in getattr(self.counters, "outside", {}).get(luid, {}).items():+ # A container sees the host's PIDs from NVML; unmatched ones are other containers or the host.+ snap.procs.append(ProcRow(pid, "(outside this container)", "external", g,+ ProcDetail(pid, "(outside this container)"), host=True, origin="outside"))+ if self.host is not None: # current inferred sizes drive both scan priority and the LLM breakdown+ self._add_host_rows(snap)+ local_rows = [r for r in snap.procs if not r.host]+ deep_pids = {r.pid for r in local_rows[: self.DEEP_TOP_N]} | {+ r.pid for r in local_rows if r.gpu.gpu_pct >= 2}deadline = None if self.scan_budget is None else time.perf_counter() + self.scan_budgetollama_runners = []- for pid, g in live: # largest VRAM first, so the budget goes to what matters- d = self.inspector.get(pid)- mem = g.dedicated + g.shared- if (pid in deep_pids and (deadline is None or time.perf_counter() < deadline)+ for row in local_rows: # largest VRAM first, so the budget goes to what matters+ d, g = row.detail, row.gpu+ mem = (g.shown_vram or 0) + g.shared+ if (row.pid in deep_pids and (deadline is None or time.perf_counter() < deadline)and self.inspector.needs_scan(d, mem)):self.inspector.scan(d, mem)- row = ProcRow(pid=pid, name=d.name, kind=classify(d, g.engines), gpu=g, detail=d)- snap.procs.append(row)+ row.kind = classify(d, g.engines)# Old Ollama: "ollama.exe runner ..."; newer Ollama: its own lib\ollama\llama-server.exe- if row.kind in AI_KINDS and ("\\ollama\\" in d.exe.lower() or+ exe = d.exe.lower()+ if row.kind in AI_KINDS and ("\\ollama\\" in exe or "/ollama/" in exe or(d.name.lower().startswith("ollama") and "runner" in d.cmdline)):ollama_runners.append(row)-for row in snap.procs:- if os.path.basename(row.detail.exe).lower() == "handy.exe":+ if row.host:+ continue+ if os.path.basename(row.detail.exe).lower() in ("handy.exe", "handy"):row.handy = inspect_handy(row.detail.create_time)- if row.detail.is_diffusion or row.kind not in AI_KINDS or not (row.detail.models or row.kind == "LLM"):+ if (row.detail.is_diffusion or row.kind not in AI_KINDS or not (row.detail.models or row.kind == "LLM")+ or not {"dedicated", "shared"} <= row.gpu.available):continue- own = self.prober.get(row.detail.ports)+ own = self.prober.get(self.inspector.exclusive_ports(row.pid))host = self.prober.get(self._parent_ports(row.pid)) # Ollama / LM Studio front-end processlog = ollama_log_path() if row in ollama_runners and len(ollama_runners) == 1 else None- row.llm = build_view(row.detail, row.gpu.dedicated, row.gpu.shared, own or host, log,- host=host if own else None)+ row.llm = build_view(row.detail, row.gpu.shown_vram or 0, row.gpu.shared,+ own or host, log, host=host if own else None,+ attributed=row.gpu.shown_vram is not None)self.inspector.prune(set(procs))return snap+ def _add_host_rows(self, snap: Snapshot) -> None:+ """Windows processes on this GPU, plus the WSL VM itself (vmwp.exe: the exact total of Linux VRAM)."""+ if self.host.ended and time.time() - self._host_restarted > 30:+ self.host.close() # PowerShell exited (e.g. killed, or the host was busy): try again+ self.host = HostStream()+ self._host_restarted = time.time()+ hs = self.host.latest()+ if hs is None:+ snap.host_status = "starting" if self.host.status in ("starting", "ok") else self.host.status+ return+ # Match the host adapter once: NVML is read now while the host sample can be a second old, so+ # re-matching every refresh fails whenever memory moves fast (a model loading) and the rows flicker.+ luid = self._host_luid if self._host_luid in hs.adapters else match_adapter(hs, snap.vram_used)+ if luid is None:+ # Memory can swing by GBs between the two readings while a workload loads: retry for a while.+ early = time.time() - self._host_restarted < 20+ snap.host_status = "starting" if early else "unmatched (no Windows adapter matches this GPU's usage)"+ return+ self._host_luid = luid+ age = time.time() - hs.taken+ snap.host_status = "ok" if age < 3 else f"ok (last Windows sample {age:.0f}s old)"+ a = hs.adapters.get(luid)+ if a is not None:- snap.shared_used = int(a.shared)++ snap.shared_used = int(a.shared) if a.shared is not None else None+ procs = hs.procs.get(luid, {})+ vm_workers = [pid for pid in procs if hs.names.get(pid, "").lower() == "vmwp"]+ host_rows: list[ProcRow] = []+ for pid, g in procs.items():+ if not _is_live(g):+ continue+ raw = hs.names.get(pid, "?")+ if raw.lower() == "vmwp":+ # One VM worker is WSL's; several could include Hyper-V VMs or Windows Sandbox.+ name = "WSL VM (vmwp.exe)" if len(vm_workers) == 1 else "VM worker (vmwp.exe)"+ kind = "wsl-vm"+ else:+ name = raw if raw in ("System", "Idle", "?") or raw.lower().endswith(".exe") else f"{raw}.exe"+ kind = "kernel" if pid == 4 else classify(ProcDetail(pid, name), g.engines)+ host_rows.append(ProcRow(pid, name, kind, g, ProcDetail(pid, name), host=True, origin="windows"))+ # Largest first; Linux rows sit right under the WSL VM row, whose total they share.+ host_rows.sort(key=lambda r: -r.gpu.dedicated)+ at = next((i + 1 for i, r in enumerate(host_rows) if r.kind == "wsl-vm"), 0)+ if len(vm_workers) == 1 and self.tracker is not None:+ self._attribute(snap, luid, hs.contexts.get(luid, {}).get(vm_workers[0], []))+ snap.procs.sort(key=lambda r: -(r.gpu.shown_vram or 0))+ snap.procs = host_rows[:at] + snap.procs + host_rows[at:]++ def _attribute(self, snap: Snapshot, luid: int, contexts: list[float]) -> None:+ linux = [r for r in snap.procs if not r.host and not r.gpu.attributed]+ # This monitor holds a (0-byte) context too, through NVML.+ holders = [(r.pid, r.detail.create_time) for r in linux] + [(os.getpid(), self._own_start)]+ guesses, reason = self.tracker.update(luid, contexts, holders)+ snap.wsl_split = reason+ for r in linux:+ g = guesses.get(r.pid)+ if g is not None:+ r.gpu.inferred, r.gpu.inferred_by = g.bytes, g.confidence+def close(self) -> None:self.counters.close()self.nvml.close()+ self.amdsmi.close()+ self.adl.close()+ if self.host is not None:+ self.host.close()diff --cc src/gputaskman/pdh.pyindex fafbc43,4116ec6..aa99ea6--- a/src/gputaskman/pdh.py+++ b/src/gputaskman/pdh.py@@@ -63,11 -72,18 +72,23 @@@ class ProcGpushared: float = 0.0 # bytes resident in shared (system RAM) GPU memory -- "spill" ("Non Local Usage")committed: float = 0.0 # all GPU memory this process allocated, resident or not ("Total Committed")engines: dict[str, float] = field(default_factory=lambda: defaultdict(float)) # engtype -> util %+ available: set[str] = field(default_factory=lambda: {"dedicated", "shared", "committed", "engine"})++ @property+ def paged_available(self) -> bool:+ return {"dedicated", "shared", "committed"} <= self.available+ # False when per-process VRAM is unavailable. Native NVML may still report engines independently;+ # WSL charges every Linux process's VRAM and engine activity to the Windows VM worker.+ attributed: bool = True+ # WSL: VRAM matched from the VM's GPU contexts (see wslhost.ContextTracker). An exact counter value whose+ # owner is inferred; kept apart from `dedicated` so it isn't counted twice next to the WSL VM row.+ inferred: float | None = None+ inferred_by: str = "" # "consistent" (candidates agree on bytes) | "order" (heuristic owner)++ @property+ def shown_vram(self) -> float | None:+ """Measured or inferred bytes for display and priority, not for summing adapter totals."""- return self.dedicated if self.attributed else self.inferred++ return (self.dedicated if "dedicated" in self.available else None) if self.attributed else self.inferred@propertydef paged_out(self) -> float:diff --cc src/gputaskman/procinfo.pyindex bb308b5,7d589bf..b64922b--- a/src/gputaskman/procinfo.py+++ b/src/gputaskman/procinfo.py@@@ -13,25 -14,27 +14,28 @@@ import psutifrom .gguf import is_gguf, read_gguffrom .stores import store_name- # (tag, dll patterns). Order = display priority (most specific first).+ # (tag, library patterns: Windows DLLs and Linux shared objects with the version suffix removed, see+ # _lib_name). Order = display priority (most specific first).RUNTIMES: list[tuple[str, tuple[str, ...]]] = [- ("llama.cpp", ("llama.dll", "libllama*.dll", "llama-*.dll", "llm_engine*")),- ("whisper.cpp", ("whisper*.dll",)),- ("ggml", ("ggml*.dll",)),- ("PyTorch", ("torch_cuda*.dll", "torch_cpu.dll")),- ("TensorRT", ("nvinfer*.dll",)),- ("ONNX RT", ("onnxruntime*.dll",)),+ ("llama.cpp", ("llama.dll", "libllama*.dll", "llama-*.dll", "llm_engine*", "libllama*.so")),+ ("whisper.cpp", ("whisper*.dll", "libwhisper*.so")),+ ("ggml", ("ggml*.dll", "libggml*.so")),+ ("PyTorch", ("torch_cuda*.dll", "torch_cpu.dll", "libtorch_cuda*.so", "libtorch_cpu.so")),+ ("TensorRT", ("nvinfer*.dll", "libnvinfer*.so")),+ ("ONNX RT", ("onnxruntime*.dll", "libonnxruntime*.so")),("DirectML", ("directml.dll",)),- ("cuDNN", ("cudnn*.dll",)),- ("CUDA", ("nvcuda.dll", "nvcuda64.dll", "cudart64_*.dll", "cublas64_*.dll")),- ("HIP", ("amdhip64.dll", "amdhip64_*.dll")),- ("Vulkan", ("vulkan-1.dll",)),- ("D3D12", ("d3d12core.dll",)),- ("OpenGL", ("opengl32.dll",)),++ ("HIP", ("amdhip64.dll", "amdhip64_*.dll", "libamdhip64.so")),+ ("cuDNN", ("cudnn*.dll", "libcudnn*.so")),+ ("CUDA", ("nvcuda.dll", "nvcuda64.dll", "cudart64_*.dll", "cublas64_*.dll",+ "libcuda.so", "libcudart.so", "libcublas.so")),+ ("Vulkan", ("vulkan-1.dll", "libvulkan.so")),+ ("D3D12", ("d3d12core.dll", "libd3d12core.so")),+ ("OpenGL", ("opengl32.dll", "libgl.so", "libglx*.so", "libegl*.so")),("D3D11", ("d3d11.dll",)),- ("NVENC", ("nvencodeapi64.dll",)),- ("NVDEC", ("nvcuvid.dll", "nvcuvid64.dll")),+ ("NVENC", ("nvencodeapi64.dll", "libnvidia-encode.so")),+ ("NVDEC", ("nvcuvid.dll", "nvcuvid64.dll", "libnvcuvid.so")),]+ _SO = re.compile(r"^(.+\.so)(?:\.[0-9][0-9a-z.]*)?$")AI_RUNTIMES = {"llama.cpp", "whisper.cpp", "ggml", "PyTorch", "TensorRT", "ONNX RT", "DirectML", "cuDNN"}_RUNTIME_RX = [(tag, re.compile("|".join(fnmatch.translate(p) for p in pats))) for tag, pats in RUNTIMES]MODEL_EXTS = {".gguf", ".ggml", ".safetensors", ".onnx", ".pt", ".pth", ".bin", ".engine", ".plan", ".tflite", ".ckpt"}diff --cc src/gputaskman/report.pyindex 7106f84,a9895f6..14eec75--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@@ -6,23 -6,73 +6,73 @@@ import dataclasseimport jsonimport os- from .collector import Snapshot+ from .collector import ProcRow, Snapshotfrom .llm import gbfrom .nvml import format_metric+ from .pdh import ProcGpu+++ WSL_HOST_NOTE = ("WSL + Windows host counters: 'WSL VM' is the exact total of Linux GPU memory (all distros). "+ "Linux rows get their size by matching the VM's GPU contexts (* matched, ? guess). "+ "W: rows are Windows processes.")+++ def wsl_note(s: Snapshot) -> str:+ if s.host_data:+ return WSL_HOST_NOTE+ status = f" Windows host counters: {s.host_status}." if s.host_status else ""+ return WSL_NOTE + status+++ WSL_EMPTY = ("No Linux process in this distro has the GPU open (other WSL distros and Docker Desktop "+ "are not visible from here).")+ WSL_NOTE = ("WSL: VRAM and engine load are device-wide. The Windows host charges every Linux process's "+ "VRAM to the VM, so rows below are processes using the GPU, not their share of it.")+++ def has_wddm_counters(s: Snapshot, row: ProcRow) -> bool:+ return s.backend == "wddm" or row.origin == "windows"+++ def has_engine_counters(s: Snapshot, row: ProcRow) -> bool:- return s.backend != "wsl" or row.origin == "windows"++ return (s.backend != "wsl" or row.origin == "windows") and "engine" in row.gpu.available+++ def vram_label(g: ProcGpu) -> str:+ """Measured size; '*' / '?' when the size is exact but which local process owns it is inferred."""+ mark = {"order": "?", "consistent": "*"}.get(g.inferred_by, "")+ return "n/a" if g.shown_vram is None else gb(g.shown_vram) + mark+++ INFERRED_NOTE = ("* matched: the size is exact (Windows' counter for a WSL VM GPU context, or NVML for a host PID "+ "seen from a container); which local process owns it is inferred, and only one explanation fits "+ "(or every one that fits gives the same size).")+ ORDER_NOTE = ("? order guess: the size is exact but several owners fit, so newer processes are paired with newer "+ "GPU contexts / PIDs and outside ones are assumed oldest. Cached foreign-size hints only rank extra "+ "candidates; they never prove ownership. Becomes * with evidence, such as a process starting while "+ "gputaskman runs.")+++ def inferred_note(g) -> str:+ return ORDER_NOTE if g.inferred_by == "order" else INFERRED_NOTEdef to_dict(s: Snapshot) -> dict:nv = dataclasses.asdict(s.nv) if s.nv else Noneprocs = []for p in s.procs:+ wddm = has_wddm_counters(s, p)+ engines = has_engine_counters(s, p)item = {- "pid": p.pid, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,- "vram_bytes": int(p.gpu.dedicated) if "dedicated" in p.gpu.available else None,- "shared_bytes": int(p.gpu.shared) if "shared" in p.gpu.available else None,- "committed_bytes": int(p.gpu.committed) if "committed" in p.gpu.available else None,- "paged_out_bytes": int(p.gpu.paged_out) if p.gpu.paged_available else None,- "gpu_pct": round(p.gpu.gpu_pct, 1) if "engine" in p.gpu.available else None,- "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if "engine" in p.gpu.available else None,+ "pid": p.pid, "host": p.origin if p.host else None, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,+ "vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,+ "vram_inferred_by": p.gpu.inferred_by or None,- "vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed else None,- "shared_bytes": int(p.gpu.shared) if wddm else None,- "committed_bytes": int(p.gpu.committed) if wddm else None,- "paged_out_bytes": int(p.gpu.paged_out) if wddm else None,++ "vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed and "dedicated" in p.gpu.available else None,++ "shared_bytes": int(p.gpu.shared) if wddm and "shared" in p.gpu.available else None,++ "committed_bytes": int(p.gpu.committed) if wddm and "committed" in p.gpu.available else None,++ "paged_out_bytes": int(p.gpu.paged_out) if wddm and p.gpu.paged_available else None,+ "gpu_pct": round(p.gpu.gpu_pct, 1) if engines else None,+ "engines": {k: round(v, 1) for k, v in p.gpu.engines.items() if v} if engines else None,"ports": p.detail.ports, "inside": p.inside,"model_files": [dataclasses.asdict(m) for m in p.detail.models],"installed_models": [dataclasses.asdict(m) for m in p.detail.candidates],@@@ -48,9 -98,9 +98,10 @@@procs.append(item)return {"taken": s.taken, "gpu": s.adapter.name if s.adapter else None, "driver": s.driver, "cuda": s.cuda,+ "adapter": dataclasses.asdict(s.adapter) if s.adapter else None,"vram_total": s.vram_total, "vram_used": s.vram_used, "shared_used": s.shared_used,- "attributed": s.attributed, "device": nv, "processes": procs,+ "attributed": s.attributed, "backend": s.backend, "host_status": s.host_status or None,+ "wsl_split": s.wsl_split or None, "device": nv, "processes": procs,}@@@ -58,13 -108,11 +109,18 @@@ def to_text(s: Snapshot, limit: int = 3out = []nv = s.nvname = s.adapter.name if s.adapter else "GPU"- out.append(f"{name} driver {s.driver} CUDA {s.cuda} {nv.driver_model if nv else ''}")- shared = f" shared (in system RAM) {gb(s.shared_used)}" if s.backend == "wddm" or s.host_data else ""- out.append(f"VRAM {gb(s.vram_used)} / {gb(s.vram_total)}{shared}")+ info = f" driver {s.driver}" if s.driver else ""+ if s.cuda:+ info += f" CUDA {s.cuda}"+ out.append(f"{name}{info} {nv.driver_model if nv else ''}")+ used = gb(s.vram_used) if s.vram_used is not None else "N/A"+ shared = gb(s.shared_used) if s.shared_used is not None else "N/A"- out.append(f"VRAM {used} / {gb(s.vram_total)} shared (in system RAM) {shared}")++ memory = f"VRAM {used} / {gb(s.vram_total)}"++ if s.backend == "wddm" or s.host_data:++ memory += f" shared (in system RAM) {shared}"++ out.append(memory)+ if s.backend == "wsl":+ out.append(wsl_note(s))if nv:rx = nv.pcie_rx_kbs / 1024 if nv.pcie_rx_kbs is not None else Nonetx = nv.pcie_tx_kbs / 1024 if nv.pcie_tx_kbs is not None else None@@@ -81,14 -125,26 +137,30 @@@f"PCIe rx {format_metric(rx, '.0f')} MB/s "f"tx {format_metric(tx, '.0f')} MB/s")out.append("")- out.append(f"{'PID':>7} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")- for p in s.procs[:limit]:- spill = "N/A" if "shared" not in p.gpu.available else gb(p.gpu.shared) if p.gpu.shared >= 1 << 20 else ""- paged = "N/A" if not p.gpu.paged_available else gb(p.gpu.paged_out) if p.gpu.paged_out >= 1 << 20 else ""- vram = gb(p.gpu.dedicated) if "dedicated" in p.gpu.available else "N/A"- gpu_pct = f"{p.gpu.gpu_pct:.0f}" if "engine" in p.gpu.available else "N/A"- out.append(f"{p.pid:>7} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {vram:>9} {spill:>8} "- f"{paged:>8} {gpu_pct:>5} {p.inside}")+ shown = s.procs[:limit]+ w = max([7] + [len(p.pid_label) for p in shown]) # host PIDs like W:120880 are wider+ out.append(f"{'PID':>{w}} {'PROCESS':<24} {'KIND':<10} {'RUNTIME':<24} {'VRAM':>9} {'SPILL':>8} {'PAGED':>8} {'GPU%':>5} INSIDE")+ for p in shown:+ wddm = has_wddm_counters(s, p)+ spill = gb(p.gpu.shared) if wddm and p.gpu.shared >= 1 << 20 else ""+ paged = gb(p.gpu.paged_out) if wddm and p.gpu.paged_out >= 1 << 20 else ""++ if wddm and "shared" not in p.gpu.available:++ spill = "N/A"++ if wddm and not p.gpu.paged_available:++ paged = "N/A"+ pid = p.pid_label+ pct = f"{p.gpu.gpu_pct:.0f}" if has_engine_counters(s, p) else "n/a"+ out.append(f"{pid:>{w}} {p.name[:24]:<24} {p.kind:<10} {p.runtime[:24]:<24} {vram_label(p.gpu):>9} {spill:>8} "+ f"{paged:>8} {pct:>5} {p.inside}")+ if any(p.gpu.inferred_by == "consistent" for p in shown):+ out.append(INFERRED_NOTE)+ if any(p.gpu.inferred_by == "order" for p in shown):+ out.append(ORDER_NOTE)+ if s.backend == "wsl" and s.wsl_split and any(not p.host and not p.gpu.attributed and p.gpu.inferred is None+ for p in s.procs):+ out.append(f"Linux VRAM split: {s.wsl_split}")+ if s.backend == "wsl" and not any(not p.host for p in s.procs):+ out.append(f"{'':>{w}} {WSL_EMPTY}")for p in s.procs:if p.handy:out.append("")diff --cc src/gputaskman/wslhost.pyindex 0000000,0dd0718..44434demode 000000,100644..100644--- a/src/gputaskman/wslhost.py+++ b/src/gputaskman/wslhost.py@@@ -1,0 -1,500 +1,503 @@@+ """Windows host GPU counters, read from inside WSL through interop.++ Linux cannot see how much VRAM WSL uses: the Windows host charges all of it to the VM worker+ (vmwp.exe). When WSL interop is enabled we can still ask Windows. One long-lived powershell.exe+ streams the same WDDM counters the Windows backend reads (one JSON line per second), so the+ ~1.5 s PowerShell startup is paid once, not per refresh.+ """++ from __future__ import annotations++ import base64+ import glob+ import itertools+ import json+ import math+ import os+ import shutil+ import subprocess+ import threading+ import time+ from collections import defaultdict+ from dataclasses import dataclass, field++ from .pdh import _ADAPTER_RE, _ENGINE_RE, _PROC_RE, AdapterGpu, ProcGpu, _norm_engine, luid_key++ _POWERSHELL_FALLBACK = "/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe"++ # PowerShell's Get-Counter resolves counters by instance *name*. vmwp.exe has one instance per guest GPU+ # context, all with the same name, so Get-Counter reads the wrong one (or one twice) whenever contexts come+ # and go. The PDH array API returns every instance as it exists at each collection, which is what the+ # Windows backend (pdh.py) uses; this compiles the same calls once and streams one JSON line per second.+ # Counter paths are English (PdhAddEnglishCounter), so this works on localized Windows too.+ _SCRIPT = r"""+ $ErrorActionPreference = 'Stop'+ [Console]::OutputEncoding = New-Object System.Text.UTF8Encoding $false # not the OEM code page+ Add-Type -TypeDefinition @'+ using System;+ using System.Runtime.InteropServices;+ using System.Text;+ public static class GtmPdh {+ [DllImport("pdh.dll", CharSet = CharSet.Unicode)]+ static extern uint PdhOpenQueryW(string source, IntPtr user, out IntPtr query);+ [DllImport("pdh.dll", CharSet = CharSet.Unicode)]+ static extern uint PdhAddEnglishCounterW(IntPtr query, string path, IntPtr user, out IntPtr counter);+ [DllImport("pdh.dll")]+ static extern uint PdhCollectQueryData(IntPtr query);+ [DllImport("pdh.dll", CharSet = CharSet.Unicode)]+ static extern uint PdhGetFormattedCounterArrayW(IntPtr counter, uint format, ref uint size, out uint count, IntPtr items);+ static IntPtr query;+ static string[] keys;+ static IntPtr[] handles;+ public static void Open(string[] paths, string[] names) {+ PdhOpenQueryW(null, IntPtr.Zero, out query);+ keys = names;+ handles = new IntPtr[paths.Length];+ for (int i = 0; i < paths.Length; i++) PdhAddEnglishCounterW(query, paths[i], IntPtr.Zero, out handles[i]);+ PdhCollectQueryData(query);+ }+ public static string Sample() {+ PdhCollectQueryData(query);+ var sb = new StringBuilder("[");+ bool first = true;+ int stride = IntPtr.Size + 16; // PDH_FMT_COUNTERVALUE_ITEM_W: name pointer, CStatus (+pad), double+ for (int i = 0; i < handles.Length; i++) {+ if (handles[i] == IntPtr.Zero) continue;+ uint size = 0, count;+ if (PdhGetFormattedCounterArrayW(handles[i], 0x8200, ref size, out count, IntPtr.Zero) != 0x800007D2) continue;+ IntPtr buf = Marshal.AllocHGlobal((int)size);+ try {+ if (PdhGetFormattedCounterArrayW(handles[i], 0x8200, ref size, out count, buf) != 0) continue;+ for (int k = 0; k < count; k++) {+ IntPtr item = buf + k * stride;+ uint status = (uint)Marshal.ReadInt32(item, IntPtr.Size);+ double value = BitConverter.Int64BitsToDouble(Marshal.ReadInt64(item, IntPtr.Size + 8));+ // Zero-size GPU contexts still matter: vmwp.exe's instances are matched to Linux+ // processes by position, so "local usage" keeps them; everything else drops zeros.+ if ((status != 0 && status != 1) || (value == 0 && i != 0)) continue;+ string name = Marshal.PtrToStringUni(Marshal.ReadIntPtr(item)).Replace("\\", "\\\\").Replace("\"", "\\\"");+ if (!first) sb.Append(',');+ first = false;+ sb.Append("[\"").Append(keys[i]).Append("\",\"").Append(name).Append("\",").Append(value.ToString("R", System.Globalization.CultureInfo.InvariantCulture)).Append(']');+ }+ } finally { Marshal.FreeHGlobal(buf); }+ }+ return sb.Append(']').ToString();+ }+ }+ '@+ [GtmPdh]::Open(+ @('\GPU Process Memory(*)\Local Usage', '\GPU Process Memory(*)\Non Local Usage',+ '\GPU Process Memory(*)\Total Committed', '\GPU Engine(*)\Utilization Percentage',+ '\GPU Adapter Memory(*)\Dedicated Usage', '\GPU Adapter Memory(*)\Shared Usage'),+ @('local usage', 'non local usage', 'total committed', 'utilization percentage', 'dedicated usage', 'shared usage'))+ while ($true) {+ Start-Sleep -Milliseconds 1000+ $s = [GtmPdh]::Sample()+ $names = @{}+ foreach ($p in [System.Diagnostics.Process]::GetProcesses()) { $names[[string]$p.Id] = $p.ProcessName }+ [Console]::Out.WriteLine('{"s":' + $s + ',"n":' + ($names | ConvertTo-Json -Compress) + '}')+ [Console]::Out.Flush()+ }+ """++ _PROC_FIELDS = {"local usage": "dedicated", "non local usage": "shared", "total committed": "committed"}+ _ADAPTER_FIELDS = {"dedicated usage": "dedicated", "shared usage": "shared"}+++ def interop_enabled() -> bool:+ """True when this WSL distro can run Windows programs (interop not disabled in /etc/wsl.conf)."""+ for path in glob.glob("/proc/sys/fs/binfmt_misc/WSLInterop*"):+ try:+ with open(path, encoding="ascii") as f:+ if f.readline().strip() == "enabled":+ return True+ except OSError:+ continue+ return False+++ def powershell_path() -> str | None:+ found = shutil.which("powershell.exe")+ if found:+ return found+ return _POWERSHELL_FALLBACK if os.path.exists(_POWERSHELL_FALLBACK) else None+++ @dataclass+ class HostSample:+ taken: float+ procs: dict[int, dict[int, ProcGpu]] = field(default_factory=dict) # {host luid: {host pid: ProcGpu}}+ adapters: dict[int, AdapterGpu] = field(default_factory=dict) # {host luid: AdapterGpu}+ names: dict[int, str] = field(default_factory=dict) # {host pid: process name without .exe}+ # {host luid: {host pid: [Local Usage per instance]}}. vmwp.exe has one instance per guest GPU context+ # (roughly one per Linux process using the GPU), all with the same name: sizes without owners.+ contexts: dict[int, dict[int, list[float]]] = field(default_factory=dict)+++ def parse_line(line: str, now: float | None = None) -> HostSample | None:+ try:+ data = json.loads(line)+ except ValueError:+ return None+ if not isinstance(data, dict):+ return None+ sample = HostSample(taken=now if now is not None else time.time())+ procs: dict[int, dict[int, ProcGpu]] = defaultdict(dict)+ adapters: dict[int, AdapterGpu] = defaultdict(AdapterGpu)+ contexts: dict[int, dict[int, list[float]]] = defaultdict(lambda: defaultdict(list))+ for row in data.get("s") or []:+ try:+ counter, instance, value = str(row[0]).lower(), str(row[1]).lower(), float(row[2])+ except (TypeError, ValueError, IndexError):+ continue+ if counter == "utilization percentage":+ m = _ENGINE_RE.match(instance)+ if m:+ g = procs[luid_key(m[2], m[3])].setdefault(int(m[1]), ProcGpu())+ g.engines[_norm_engine(m[6])] += value+ elif counter in _PROC_FIELDS:+ m = _PROC_RE.match(instance)+ if m:+ g = procs[luid_key(m[2], m[3])].setdefault(int(m[1]), ProcGpu())+ attr = _PROC_FIELDS[counter]+ setattr(g, attr, getattr(g, attr) + value) # vmwp.exe has one instance per guest context+ if attr == "dedicated":+ contexts[luid_key(m[2], m[3])][int(m[1])].append(value)+ elif counter in _ADAPTER_FIELDS:+ m = _ADAPTER_RE.match(instance)+ if m:+ a = adapters[luid_key(m[1], m[2])]+ attr = _ADAPTER_FIELDS[counter]- setattr(a, attr, getattr(a, attr) + value)++ setattr(a, attr, (getattr(a, attr) or 0.0) + value)+ for pid, name in (data.get("n") or {}).items():+ try:+ sample.names[int(pid)] = str(name)+ except ValueError:+ continue+ sample.procs, sample.adapters = dict(procs), dict(adapters)+ sample.contexts = {luid: dict(per) for luid, per in contexts.items()}+ return sample++-def match_adapter(sample: HostSample, device_used: float) -> int | None:++def match_adapter(sample: HostSample, device_used: float | None) -> int | None:+ """WSL's adapter LUIDs differ from the host's, so pair the host adapter whose dedicated usage matches+ NVML's device-wide usage. Ambiguous or distant matches return None: no host data beats wrong data."""+ if not device_used or not sample.adapters:+ return None- ranked = sorted(sample.adapters.items(), key=lambda kv: abs(kv[1].dedicated - device_used))++ ranked = sorted(((luid, a) for luid, a in sample.adapters.items() if a.dedicated is not None),++ key=lambda kv: abs(kv[1].dedicated - device_used))++ if not ranked:++ return None+ luid, best = ranked[0]+ tolerance = max(512 << 20, 0.15 * device_used)+ if abs(best.dedicated - device_used) > tolerance:+ return None+ if len(ranked) > 1 and abs(ranked[1][1].dedicated - device_used) <= tolerance:+ return None+ return luid+++ class HostStream:+ """Background reader of the PowerShell counter stream. status: starting | ok | off | unavailable (...)."""++ STALE_AFTER = 30.0 # keep showing the last sample through PowerShell stalls; drop it only after this++ def __init__(self, exe: str | None = None, popen=subprocess.Popen) -> None:+ self.status = "starting"+ self._latest: HostSample | None = None+ self._ready = threading.Event()+ self._proc = None+ exe = exe or powershell_path()+ if exe is None:+ self.status = "unavailable (powershell.exe not found)"+ self._ready.set()+ return+ try:+ encoded = base64.b64encode(_SCRIPT.encode("utf-16-le")).decode("ascii")+ self._proc = popen([exe, "-NoProfile", "-NonInteractive", "-EncodedCommand", encoded],+ stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,+ text=True, encoding="utf-8", errors="replace", bufsize=1)+ except (OSError, ValueError) as error:+ self.status = f"unavailable ({error})"+ self._ready.set()+ return+ threading.Thread(target=self._read, name="wsl-host-counters", daemon=True).start()++ def _read(self) -> None:+ try:+ for line in self._proc.stdout:+ try:+ sample = parse_line(line)+ except Exception: # one malformed line must not end the stream+ continue+ if sample is not None and (sample.procs or sample.adapters):+ self._latest = sample+ self.status = "ok"+ self._ready.set()+ except Exception:+ pass+ finally:+ # always report the end, so the collector restarts the stream instead of waiting forever+ if self.status != "off":+ self.status = "unavailable (counter stream ended)"+ self._ready.set()++ def wait(self, timeout: float) -> bool:+ return self._ready.wait(timeout)++ def latest(self) -> HostSample | None:+ s = self._latest+ return s if s is not None and time.time() - s.taken <= self.STALE_AFTER else None++ @property+ def ended(self) -> bool:+ return self.status.startswith("unavailable")++ def close(self) -> None:+ self.status = "off"+ if self._proc is not None and self._proc.poll() is None:+ self._proc.terminate()+ try:+ self._proc.wait(timeout=2)+ except subprocess.TimeoutExpired:+ self._proc.kill()+++ # ---------------------------------------------------------------- per-process VRAM by elimination++ MiB = 1 << 20+++ def _boot_id() -> str:+ try:+ with open("/proc/sys/kernel/random/boot_id", encoding="ascii") as f:+ return f.read().strip()+ except OSError:+ return ""+++ def _default_cache() -> str:+ base = os.environ.get("XDG_CACHE_HOME") or os.path.join(os.path.expanduser("~"), ".cache")+ return os.path.join(base, "gputaskman", "wsl-idle-contexts.json")+++ @dataclass+ class ContextGuess:+ bytes: float+ confidence: str # "consistent": every equally good explanation agrees | "order": start-order guess+++ FOREIGN = "foreign" # a context from outside this distro (WSLg, other distros, Docker, hidden services)+++ @dataclass+ class _Entry:+ value: float+ label: object = None # holder pid | FOREIGN | ("gone", pid): its holder exited, waiting to vanish+ sure: bool = False # established by an event or elimination, never by a guess+ same: bool = False # this sample only: all candidates are the same size, so the bytes are right either way+++ class ContextTracker:+ """Matches vmwp.exe's GPU contexts to Linux processes by following them over time.++ Measured on WSL2 (driver 610.88): every Linux process that opens /dev/dxg gets one vmwp.exe context+ (an NVML-only process too, at 0 bytes), and the PDH instance array lists contexts newest first,+ keeping that order as others come and go. Contexts from outside this distro are interleaved by age.++ Contexts are aligned sample to sample (survivors keep their order). A match becomes *sure* only from+ evidence: a new Linux GPU process and a new context appearing together, elimination (as many unsure+ contexts as unsure processes, and just one process), contexts seen at startup while no other Linux process+ holds the GPU, or this monitor's own context being the only 0-byte candidate at startup.+ Cached foreign sizes help rank guesses but never establish ownership across runs.+ Everything else is an order guess (newest process <-> newest context, foreign assumed oldest),+ recomputed each sample and never used as evidence.+ """++ PENDING = 5.0 # seconds a new process may take to create its context+ GONE = 5.0 # seconds an exited process's context may linger in the (older) host sample+ TOL_MIN = 64 * MiB+ TOL_REL = 0.10++ def __init__(self, path: str | None = None, boot_id: str | None = None, self_pid: object = None) -> None:+ self.path = path or _default_cache()+ self.boot_id = _boot_id() if boot_id is None else boot_id+ self.self_pid = self_pid+ self.luid: int | None = None+ self.cached_foreign: list[float] = []+ self._reset()+ try:+ with open(self.path, encoding="utf-8") as f:+ data = json.load(f)+ if data.get("boot_id") == self.boot_id and self.boot_id:+ self.cached_foreign = [float(v) for v in data["foreign"]]+ self.luid = int(data["luid"])+ except (OSError, ValueError, KeyError, TypeError):+ pass++ def _reset(self) -> None:+ self.entries: list[_Entry] | None = None+ self.seen: dict[object, float] = {} # holder pid -> first seen+ self.initial: set = set() # holders already running at the first sample+ self.gone: dict[object, float] = {}++ def _tol(self, v: float) -> float:+ return max(self.TOL_MIN, self.TOL_REL * v)++ @staticmethod+ def _soft(x: float, y: float) -> float:+ return min(abs(x - y) / max(x, y, 256 * MiB), 1.0)++ EVENT_COST = 1.0 # per context appearing or vanishing beyond what process starts/exits explain++ def _align(self, prev: list[_Entry], cur: list[float], max_new: int, max_gone: int) -> list[_Entry | None]:+ """Previous context for each current one (None = new). New contexts are always a prefix (the list+ is newest first); vanished ones can be anywhere. Sizes decide among the structures that process+ starts/exits allow; other structures cost EVENT_COST per extra context, so a context that grows by+ GBs between samples (a model loading) stays the same context."""+ n, best, best_cost = len(prev), None, float("inf")+ for k in range(len(cur) + 1):+ surv = cur[k:]+ m = len(surv)+ if m > n:+ continue+ # f[i][j]: cost of matching all of surv[j:] in order to a subsequence of prev[i:]+ inf = float("inf")+ f = [[inf] * (m + 1) for _ in range(n + 1)]+ for i in range(n + 1):+ f[i][m] = 0.0+ for i in range(n - 1, -1, -1):+ for j in range(m - 1, -1, -1):+ f[i][j] = min(f[i + 1][j], self._soft(prev[i].value, surv[j]) + f[i + 1][j + 1])+ gone = n - m+ cost = f[0][0] + self.EVENT_COST * (max(0, k - max_new) + max(0, gone - max_gone))+ if cost < best_cost - 1e-9:+ out: list[_Entry | None] = [None] * len(cur)+ i = j = 0+ while j < m:+ if self._soft(prev[i].value, surv[j]) + f[i + 1][j + 1] <= f[i][j] + 1e-9:+ out[k + j] = prev[i]+ j += 1+ i += 1+ best, best_cost = out, cost+ return best++ def _save(self) -> None:+ try:+ os.makedirs(os.path.dirname(self.path), exist_ok=True)+ tmp = f"{self.path}.{os.getpid()}.tmp"+ with open(tmp, "w", encoding="utf-8") as f:+ json.dump({"boot_id": self.boot_id, "luid": self.luid, "foreign": self.cached_foreign}, f)+ os.replace(tmp, self.path)+ except OSError:+ pass++ def update(self, luid: int, contexts: list[float], holders: list[tuple[object, float]],+ now: float | None = None) -> tuple[dict[object, ContextGuess], str]:+ """contexts: vmwp.exe sizes, newest first. holders: (pid, start time) of every Linux process in this+ distro with /dev/dxg open, this monitor included. Returns ({pid: guess}, reason)."""+ now = time.time() if now is None else now+ if luid != self.luid:+ self.luid, self.cached_foreign = luid, []+ self._reset()+ start = dict(holders)+ first = self.entries is None+ had_foreign = any(e.label == FOREIGN and e.sure for e in self.entries or [])+ cur = [_Entry(v) for v in contexts]+ fresh: list[_Entry] = [] # contexts that appeared since the previous sample+ if not first:+ sure = {e.label for e in self.entries if e.sure}+ waiting = [p for p in start if p not in self.initial and p not in sure] # new or lagging processes+ exited = [p for p in self.seen if p not in start]+ lingering = sum(1 for e in self.entries if isinstance(e.label, tuple))+ for e, old in zip(cur, self._align(self.entries, contexts, len(waiting), len(exited) + lingering)):+ if old is None:+ fresh.append(e)+ else:+ e.label, e.sure = old.label, old.sure++ for pid in start:+ if pid not in self.seen:+ self.seen[pid] = now+ self.gone.pop(pid, None) # reused PID: not the process that exited+ if first:+ self.initial.add(pid)+ for pid in [p for p in self.seen if p not in start]:+ del self.seen[pid]+ self.initial.discard(pid) # a reused PID is a new process, not one that was running at startup+ self.gone[pid] = now+ for e in cur:+ if e.label in self.gone and not isinstance(e.label, tuple):+ e.label, e.sure = ("gone", e.label), False+ if isinstance(e.label, tuple) and now - self.gone.get(e.label[1], 0) > self.GONE:+ e.label = None+ if e.label is not None and not isinstance(e.label, tuple) and e.label != FOREIGN and e.label not in start:+ e.label, e.sure = None, False+ self.gone = {p: t for p, t in self.gone.items() if now - t <= self.GONE}+ for e in cur: # guesses are recomputed below, never carried as evidence+ if not e.sure and not isinstance(e.label, tuple):+ e.label = None++ def labeled() -> set:+ return {e.label for e in cur if e.sure}++ def age(pid) -> float: # larger = newer GPU context+ return start.get(pid, 0.0) if pid in self.initial else self.seen[pid]++ if not start:+ for e in cur:+ if e.label is None:+ e.label, e.sure = FOREIGN, True+ elif first:+ zeros = [e for e in cur if e.value == 0]+ if self.self_pid in start and len(zeros) == 1:+ zeros[0].label, zeros[0].sure = self.self_pid, True+ if set(start) == {self.self_pid} and zeros:+ for e in cur:+ if e.value > 0: # NVML alone holds no VRAM; other zero-byte owners remain ambiguous+ e.label, e.sure = FOREIGN, True+ elif fresh:+ pending = sorted((p for p in start if p not in labeled() and p not in self.initial+ and now - self.seen[p] <= self.PENDING),+ key=lambda p: -age(p))+ fresh = [e for e in fresh if e.label is None]+ if not pending:+ for e in fresh:+ e.label, e.sure = FOREIGN, True+ elif len(fresh) == len(pending) == 1:+ fresh[0].label, fresh[0].sure = pending[0], True++ unsure = sorted((p for p in start if p not in labeled()), key=lambda p: -age(p))+ free = [e for e in cur if e.label is None]+ if unsure and len(free) > len(unsure):+ # Cached sizes may outlive their owners: rank guesses, never exclude candidates or pin labels.+ free.sort(key=lambda e: e.value > 0 and any(+ v > 0 and abs(e.value - v) <= self._tol(v) for v in self.cached_foreign))+ reason = ""+ if unsure:+ if len(free) < len(unsure):+ reason = (f"{len(unsure)} Linux GPU process(es) without a VM context to match "+ "(one may not have created its GPU context yet)")+ # A process seen moments ago may not have created its context yet; the one free context could+ # then be a foreign one (WSLg, another distro), so wait for its own context instead of eliminating.+ recent = any(p not in self.initial and now - self.seen[p] <= self.PENDING for p in unsure)+ if len(free) == len(unsure) == 1 and not recent:+ free[0].label, free[0].sure = unsure[0], True+ else:+ # Exactly equal only: within tolerance, a swapped owner would show the wrong bytes.+ same = (len(free) == len(unsure) > 1 and not recent+ and all(e.value == free[0].value for e in free))+ for pid, e in zip(unsure, free): # newest process <-> newest context; leftovers are foreign+ # Equal sizes make the shown bytes right, not the owner: never carried as evidence.+ e.label, e.same = pid, same++ self.entries = cur+ foreign = [e.value for e in cur if e.label == FOREIGN and e.sure and e.value > 0]+ if not start or set(start) == {self.self_pid} or foreign or had_foreign:+ if sorted(foreign) != sorted(self.cached_foreign):+ self.cached_foreign = foreign+ self._save()+ out = {e.label: ContextGuess(e.value, "consistent" if e.sure or e.same else "order")+ for e in cur if e.label in start}+ return out, reasondiff --cc tests/test_core.pyindex c66a628,ed140cd..9fc755e--- a/tests/test_core.py+++ b/tests/test_core.py@@@ -17,9 -16,7 +17,9 @@@ from rich.console import Consolfrom rich.table import Tablefrom textual.widgets import Footer, Select, Static- from gputaskman.app import GpuTaskManager, spark+from gputaskman.amdsmi import AmdSmi+from gputaskman.adl import Adl, _Metrics+ from gputaskman.app import GpuTaskManager, spark, stacked_barfrom gputaskman.cli import mainfrom gputaskman.collector import ProcRow, Snapshot, select_adapter, select_nv_devicefrom gputaskman.dxgi import Adapter@@@ -201,10 -115,10 +206,11 @@@ class DeviceSelectionTests(unittest.Teswith self.assertRaisesRegex(ValueError, "out of range"):select_adapter([], 1)- def test_nvml_matches_unique_adapter_name_before_index(self):- gpu = Adapter(1, "NVIDIA RTX Spark", 0x10DE, 64 * GiB, 0, False)++ @patch("gputaskman.collector.sys.platform", "win32")+ def test_nvml_matches_pci_address_not_name_or_index(self):+ gpu = Adapter(1, "NVIDIA RTX Spark", 0x10DE, 64 * GiB, 0, False, (10, 0, 0))npu = NvDevice(index=0, name="NVIDIA NPU", driver_model="MCDM")- nv_gpu = NvDevice(index=1, name="NVIDIA RTX SPARK", driver_model="WDDM")+ nv_gpu = NvDevice(index=1, name="NVIDIA RTX SPARK", pci_bus="00000000:0A:00.0", driver_model="WDDM")self.assertIs(select_nv_device(gpu, [npu, nv_gpu], 0), nv_gpu)self.assertIs(select_nv_device(None, [npu, nv_gpu], 0), npu)self.assertIsNone(select_nv_device(gpu, [npu], 0))@@@ -214,50 -127,15 +220,59 @@@class MemoryTests(unittest.TestCase):+ def test_failed_collection_does_not_reuse_previous_counters(self):+ counters = GpuCounters.__new__(GpuCounters)+ counters._query = None+ with patch("gputaskman.pdh._pdh", SimpleNamespace(PdhCollectQueryData=lambda query: 1)), \+ patch.object(counters, "_read") as read:+ self.assertEqual(counters.sample(), ({}, {}))+ read.assert_not_called()++ def test_counter_samples_track_each_process_field(self):+ counters = GpuCounters.__new__(GpuCounters)+ counters._query = None+ data = {"dedicated": [("pid_42_luid_0x0_0x1_phys_0", 0.0)],+ "engine": [("pid_42_luid_0x0_0x1_phys_0_eng_0_engtype_Compute", 10.0)]}+ with patch.object(counters, "_read", side_effect=lambda key: data.get(key, [])), \+ patch("gputaskman.pdh._pdh", SimpleNamespace(PdhCollectQueryData=lambda query: 0)):+ processes, _ = counters.sample()+ gpu = processes[1][42]+ self.assertEqual(gpu.available, {"dedicated", "engine"})+ self.assertEqual(gpu.dedicated, 0)+ self.assertEqual(gpu.gpu_pct, 10)+ self.assertFalse(gpu.paged_available)++ def test_missing_process_counters_are_not_reported_as_zero(self):+ s = snapshot()+ s.procs[0].gpu.available = {"dedicated"}+ row = to_dict(s)["processes"][0]+ self.assertEqual(row["vram_bytes"], 8 * GiB)+ for name in ("shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):+ self.assertIsNone(row[name])+ self.assertIn("N/A", to_text(s))+ s.procs[0].gpu.available.clear()+ self.assertIsNone(to_dict(s)["attributed"])++ def test_missing_adapter_memory_is_not_reported_as_zero(self):+ s = snapshot()+ s.vram_used = None+ s.shared_used = None+ self.assertIsNone(to_dict(s)["vram_used"])+ self.assertIsNone(to_dict(s)["shared_used"])+ self.assertIn("VRAM N/A", to_text(s))+ self.assertIn("shared (in system RAM) N/A", to_text(s))+ output = io.StringIO()+ Console(file=output, width=120).print(GpuTaskManager().render_device(s))+ self.assertIn("N/A", output.getvalue())+ def test_best_vram_keeps_measured_and_inferred_zero_distinct_from_unknown(self):+ gpu = ProcGpu(dedicated=0, inferred=8 * GiB)+ self.assertEqual(gpu.shown_vram, 0)+ gpu.attributed = False+ self.assertEqual(gpu.shown_vram, 8 * GiB)+ gpu.inferred = 0+ self.assertEqual(gpu.shown_vram, 0)+ gpu.inferred = None+ self.assertIsNone(gpu.shown_vram)def test_local_counters_are_preferred(self):self.assertTrue(GpuCounters._COUNTERS["dedicated"][0].endswith(r"\Local Usage"))diff --cc tests/test_linux.pyindex 0000000,2196ff5..158e3a1mode 000000,100644..100644--- a/tests/test_linux.py+++ b/tests/test_linux.py@@@ -1,0 -1,367 +1,378 @@@+ from __future__ import annotations++ import unittest+ from types import SimpleNamespace+ from unittest.mock import patch++ from gputaskman import linux-from gputaskman.collector import ProcRow, Snapshot++from gputaskman.collector import ProcRow, Snapshot, select_nv_device++from gputaskman.dxgi import Adapter++from gputaskman.nvml import NvDevice+ from gputaskman.gguf import GgufInfo+ from gputaskman.llm import LlmView, build_view+ from gputaskman.pdh import ProcGpu+ from gputaskman.procinfo import ProcDetail, _lib_name, _match_runtimes, classify+ from gputaskman.report import to_dict, to_text++ GiB = 1 << 30+ NOT_AVAILABLE = (1 << 64) - 1 # NVML_VALUE_NOT_AVAILABLE as an unsigned long long+++ class FakeNvml:+ """Just enough of pynvml for LinuxCounters: one GPU, scripted process lists and utilization samples."""++ def __init__(self, compute=(), graphics=(), samples=(), used=4 * GiB):+ self.compute, self.graphics, self.samples, self.used = list(compute), list(graphics), list(samples), used+ self.util_calls = []++ def nvmlInit(self):+ pass++ def nvmlShutdown(self):+ pass++ def nvmlDeviceGetCount(self):+ return 1++ def nvmlDeviceGetHandleByIndex(self, i):+ return "h0"++ def nvmlDeviceGetPciInfo(self, h):+ return SimpleNamespace(busId=b"00000000:01:00.0")++ def nvmlDeviceGetName(self, h):+ return "NVIDIA GeForce RTX 3090 Ti"++ def nvmlDeviceGetMemoryInfo(self, h):+ return SimpleNamespace(total=24 * GiB, used=self.used)++ def nvmlDeviceGetComputeRunningProcesses(self, h):+ return self.compute++ def nvmlDeviceGetGraphicsRunningProcesses(self, h):+ return self.graphics++ def nvmlDeviceGetProcessUtilization(self, h, since):+ self.util_calls.append(since)+ return self.samples+++ def proc(pid, used):+ return SimpleNamespace(pid=pid, usedGpuMemory=used)+++ def sample(pid, sm, enc=0, dec=0, ts=1):+ return SimpleNamespace(pid=pid, smUtil=sm, encUtil=enc, decUtil=dec, timeStamp=ts)+++ KEY = linux.pci_key("00000000:01:00.0")+++ class LinuxLibraryTests(unittest.TestCase):++ @patch("gputaskman.collector.sys.platform", "linux")++ def test_nvml_device_matches_linux_pci_key(self):++ gpu = Adapter(KEY, "NVIDIA", 0x10DE, 24 * GiB, 0, False)++ device = NvDevice(index=1, pci_bus="00000000:01:00.0")++ other = NvDevice(index=0, pci_bus="00000000:02:00.0")++ self.assertIs(select_nv_device(gpu, [other, device], 0), device)++ self.assertIsNone(select_nv_device(gpu, [other], 0))++ self.assertIsNone(select_nv_device(gpu, [device, device], 0))+++ def test_versioned_shared_objects_are_libraries(self):+ self.assertEqual(_lib_name("libcudart.so.12"), "libcudart.so")+ self.assertEqual(_lib_name("libcuda.so.1.1"), "libcuda.so")+ self.assertEqual(_lib_name("libtorch_cuda.so"), "libtorch_cuda.so")+ self.assertEqual(_lib_name("nvcuda.dll"), "nvcuda.dll")+ self.assertIsNone(_lib_name("model.gguf"))+ self.assertIsNone(_lib_name("weights.safetensors"))+ self.assertIsNone(_lib_name("model.so-weights.bin"))++ def test_linux_runtimes_are_detected(self):+ libs = {_lib_name(n) for n in ("libggml-cuda.so", "libllama.so", "libcudart.so.12", "libcuda.so.1",+ "libvulkan.so.1", "libtorch_cuda.so", "libnvidia-encode.so.1")}+ self.assertEqual(_match_runtimes(libs), ["llama.cpp", "ggml", "PyTorch", "CUDA", "Vulkan", "NVENC"])++ def test_linux_compositors_and_browsers(self):+ self.assertEqual(classify(ProcDetail(900, "Xorg"), {}), "compositor")+ self.assertEqual(classify(ProcDetail(901, "gnome-shell"), {}), "compositor")+ self.assertEqual(classify(ProcDetail(902, "firefox"), {}), "browser")++ def test_pid_4_is_only_special_on_windows(self):+ with patch("gputaskman.procinfo.sys.platform", "linux"):+ self.assertEqual(classify(ProcDetail(4, "kworker/0:0"), {}), "app")+ with patch("gputaskman.procinfo.sys.platform", "win32"):+ self.assertEqual(classify(ProcDetail(4, "System"), {}), "kernel")+++ class PdhImportTests(unittest.TestCase):+ def test_pdh_module_is_complete_without_loading_the_dll(self):+ from gputaskman import pdh+ for name in ("PDH_FMT_DOUBLE", "PDH_FMT_NOCAP100", "PDH_MORE_DATA", "_Item", "_load_pdh"):+ self.assertTrue(hasattr(pdh, name), name)+++ class LinuxCountersTests(unittest.TestCase):+ def setUp(self):+ # Same PID namespace (bare metal): NVML's PIDs are local processes holding an NVIDIA device.+ for p in (patch.object(linux, "nvidia_holders", return_value={100, 101, 200, 300, 555}),+ patch.object(linux, "in_child_pid_namespace", return_value=False)):+ p.start()+ self.addCleanup(p.stop)++ def counters(self, fake, wsl):+ with patch.object(linux, "N", fake), patch.object(linux, "is_wsl", return_value=wsl):+ return linux.LinuxCounters()++ def test_native_linux_attributes_vram_and_engines_per_pid(self):+ fake = FakeNvml(compute=[proc(100, 3 * GiB), proc(200, 1 * GiB)],+ graphics=[proc(200, 1 * GiB), proc(300, 256 << 20)],+ samples=[sample(100, 80, ts=5), sample(100, 90, ts=9), sample(300, 30, dec=20, ts=7)])+ c = self.counters(fake, wsl=False)+ with patch.object(linux, "N", fake):+ procs, adapters = c.sample()+ c.sample()+ per = procs[KEY]+ self.assertEqual(adapters[KEY].dedicated, 4 * GiB)+ self.assertEqual(per[100].dedicated, 3 * GiB)+ self.assertEqual(per[200].dedicated, 1 * GiB) # in both lists: counted once+ self.assertEqual(per[100].engines["compute"], 90) # latest sample wins+ self.assertEqual(per[300].engines["3d"], 30) # graphics-only process: SM load is 3D work+ self.assertEqual(per[300].engines["decode"], 20)+ self.assertTrue(all(g.attributed for g in per.values()))+ self.assertEqual(fake.util_calls, [0, 9]) # next poll asks only for newer samples++ def test_hidden_per_process_memory_is_unknown_not_zero(self):+ fake = FakeNvml(compute=[proc(100, NOT_AVAILABLE), proc(101, None)])+ c = self.counters(fake, wsl=False)+ with patch.object(linux, "N", fake):+ procs, _ = c.sample()+ self.assertFalse(procs[KEY][100].attributed)+ self.assertFalse(procs[KEY][101].attributed)++ def test_utilization_for_unlisted_pids_is_ignored(self):+ fake = FakeNvml(compute=[proc(100, GiB)], samples=[sample(555, 50)])+ c = self.counters(fake, wsl=False)+ with patch.object(linux, "N", fake):+ procs, _ = c.sample()+ self.assertNotIn(555, procs[KEY])++ def test_wsl_never_uses_windows_host_pids(self):+ # Under WSL NVML reports Windows PIDs (dwm 2744, System 4); they must not become Linux rows.+ fake = FakeNvml(samples=[sample(2744, 12), sample(4, 88)], used=7 * GiB)+ c = self.counters(fake, wsl=True)+ with patch.object(linux, "N", fake), patch.object(linux, "dxg_pids", return_value={4242}):+ procs, adapters = c.sample()+ self.assertEqual(adapters[KEY].dedicated, 7 * GiB)+ self.assertEqual(set(procs[KEY]), {4242})+ self.assertFalse(procs[KEY][4242].attributed)+ self.assertEqual(fake.util_calls, [])++ def test_adapters_come_from_nvml(self):+ with patch.object(linux, "N", FakeNvml()):+ adapters = linux.enumerate_adapters()+ self.assertEqual([(a.luid, a.name, a.dedicated_vram) for a in adapters],+ [(KEY, "NVIDIA GeForce RTX 3090 Ti", 24 * GiB)])++ def test_a_failing_gpu_is_skipped_not_fatal(self):+ fake = FakeNvml()+ fake.nvmlDeviceGetCount = lambda: 2+ real = fake.nvmlDeviceGetHandleByIndex+ def handle(i):+ if i == 0:+ raise RuntimeError("NVML_ERROR_GPU_IS_LOST")+ return real(0)+ fake.nvmlDeviceGetHandleByIndex = handle+ with patch.object(linux, "N", fake):+ self.assertEqual(len(linux.enumerate_adapters()), 1)+ fake.nvmlDeviceGetCount = lambda: (_ for _ in ()).throw(RuntimeError("NVML_ERROR_UNKNOWN"))+ with patch.object(linux, "N", fake):+ self.assertEqual(linux.enumerate_adapters(), [])++ def test_other_users_processes_stay_local_on_bare_metal(self):+ # Without root, another user's /proc/<pid>/fd is unreadable, so it isn't an NVIDIA holder.+ fake = FakeNvml(compute=[proc(6000, 20 * GiB)])+ c = self.counters(fake, wsl=False)+ with patch.object(linux, "N", fake), patch.object(linux, "nvidia_holders", return_value={100}), \+ patch.object(linux, "has_cuda", return_value=True):+ procs, _ = c.sample()+ self.assertEqual((procs[KEY][6000].dedicated, procs[KEY][6000].inferred_by), (20 * GiB, ""))+ self.assertNotIn(100, procs[KEY])+ self.assertEqual(c.outside, {})+++ class ContainerTests(unittest.TestCase):+ """Measured on a Vast.ai Docker container: NVML lists host PIDs (2029701...) while the processes are+ 547 and 563 inside; the host's display server shows up only in the graphics list."""++ def sample(self, c, fake, holders, cuda):+ with patch.object(linux, "N", fake), patch.object(linux, "nvidia_holders", return_value=set(holders)), \+ patch.object(linux, "has_cuda", side_effect=lambda pid: pid in cuda), \+ patch.object(linux, "in_child_pid_namespace", return_value=self.child):+ return c.sample()[0][KEY]++ child = True++ def test_host_pids_are_matched_to_container_pids_by_creation_order(self):+ fake = FakeNvml(compute=[proc(2029701, GiB), proc(2029747, 2 * GiB)],+ graphics=[proc(1145, 58 << 20), proc(1360, 21 << 20)],+ samples=[sample(2029747, 95, ts=3)])+ with patch.object(linux, "N", fake), patch.object(linux, "is_wsl", return_value=False):+ c = linux.LinuxCounters()+ # 564 is an NVML-only tool (nvidia-smi, this monitor): it holds the devices but has no CUDA context+ per = self.sample(c, fake, holders={547, 563, 564}, cuda={547, 563})+ self.assertEqual({pid: (g.dedicated, g.inferred_by) for pid, g in per.items()},+ {547: (GiB, "consistent"), 563: (2 * GiB, "consistent")})+ self.assertEqual(per[563].engines["compute"], 95) # utilization follows the match+ self.assertEqual(set(c.outside[KEY]), {1145, 1360}) # never looked up in the container's /proc++ def test_extra_outside_compute_is_a_guess_until_a_start_event(self):+ fake = FakeNvml(compute=[proc(9000, GiB), proc(2029701, 3 * GiB)]) # 9000: another container+ with patch.object(linux, "N", fake), patch.object(linux, "is_wsl", return_value=False):+ c = linux.LinuxCounters()+ per = self.sample(c, fake, holders={547}, cuda={547})+ self.assertEqual((per[547].dedicated, per[547].inferred_by), (3 * GiB, "order"))+ self.assertEqual(set(c.outside[KEY]), {9000})+ fake.compute.append(proc(2029800, 2 * GiB)) # a new local job starts: one new PID on each side+ per = self.sample(c, fake, holders={547, 600}, cuda={547, 600})+ self.assertEqual((per[600].dedicated, per[600].inferred_by), (2 * GiB, "consistent"))++ def test_bare_metal_is_unchanged(self):+ fake = FakeNvml(compute=[proc(547, GiB)])+ with patch.object(linux, "N", fake), patch.object(linux, "is_wsl", return_value=False):+ c = linux.LinuxCounters()+ self.child = False+ per = self.sample(c, fake, holders={547}, cuda={547})+ self.assertEqual((per[547].dedicated, per[547].inferred_by), (GiB, ""))+ self.assertEqual(c.outside, {})++ def test_a_host_pid_equal_to_a_local_pid_is_still_a_host_pid(self):+ # Host PIDs 100 and 200 are container PIDs 10 and 100: local PID 100 is not host PID 100.+ fake = FakeNvml(compute=[proc(100, GiB), proc(200, 4 * GiB)])+ with patch.object(linux, "N", fake), patch.object(linux, "is_wsl", return_value=False):+ c = linux.LinuxCounters()+ per = self.sample(c, fake, holders={10, 100}, cuda={10, 100})+ self.assertEqual({pid: (g.dedicated, g.inferred_by) for pid, g in per.items()},+ {10: (GiB, "consistent"), 100: (4 * GiB, "consistent")})++ def test_namespace_detection(self):+ linux.in_child_pid_namespace.cache_clear()+ self.addCleanup(linux.in_child_pid_namespace.cache_clear)+ with patch.object(linux.os, "readlink", return_value="pid:[4026531836]"):+ self.assertFalse(linux.in_child_pid_namespace())+ linux.in_child_pid_namespace.cache_clear()+ with patch.object(linux.os, "readlink", return_value="pid:[4026532481]"):+ self.assertTrue(linux.in_child_pid_namespace())+ linux.in_child_pid_namespace.cache_clear()+ with patch.object(linux.os, "readlink", side_effect=OSError):+ self.assertIsNone(linux.in_child_pid_namespace())++ def test_outside_rows_never_borrow_a_local_identity(self):+ from gputaskman.collector import ProcRow as Row+ row = Row(1145, "(outside this container)", "external", ProcGpu(dedicated=58 << 20),+ ProcDetail(1145, "(outside this container)"), host=True, origin="outside")+ self.assertEqual((row.key, row.pid_label), ("o1145", "H:1145"))+ snap = Snapshot(taken=1, adapter=None, adapters=[], nv=None, driver="", cuda="", vram_total=8 * GiB,+ vram_used=GiB, shared_used=0, backend="nvml", procs=[row])+ self.assertIn("H:1145", to_text(snap))+ self.assertEqual(to_dict(snap)["processes"][0]["host"], "outside")+++ class UnattributedReportTests(unittest.TestCase):+ def snapshot(self):+ return Snapshot(taken=1, adapter=None, adapters=[], nv=None, driver="", cuda="", vram_total=24 * GiB,+ vram_used=7 * GiB, shared_used=0, backend="wsl",+ procs=[ProcRow(4242, "python3", "AI", ProcGpu(attributed=False), ProcDetail(4242, "python3"))])++ def test_reports_say_unknown_instead_of_zero(self):+ data = to_dict(self.snapshot())+ self.assertEqual(data["backend"], "wsl")+ for key in ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):+ self.assertIsNone(data["processes"][0][key], key)+ self.assertFalse(data["processes"][0]["vram_attributed"])+ text = to_text(self.snapshot())+ row = next(line for line in text.splitlines() if "4242" in line)+ self.assertIn("n/a", row)+ self.assertNotIn("0 B", row)+ self.assertIn("WSL:", text)+ self.assertNotIn("shared (in system RAM)", text)++ def test_json_unknown_counters_are_null_not_zero(self):+ snap = self.snapshot()+ win = ProcRow(2744, "dwm.exe", "compositor", ProcGpu(dedicated=GiB, shared=6 << 20, committed=2 * GiB),+ ProcDetail(2744, "dwm.exe"), host=True, origin="windows")+ snap.procs.append(win)+ linux_row, win_row = to_dict(snap)["processes"]+ for k in ("shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct", "engines"):+ self.assertIsNone(linux_row[k], k)+ self.assertEqual((win_row["shared_bytes"], win_row["committed_bytes"]), (6 << 20, 2 * GiB))+ self.assertEqual(win_row["gpu_pct"], 0.0) # measured idle+ snap.backend = "nvml" # native Linux: NVML measures VRAM and load, but there is no spill/commit+ snap.procs = [ProcRow(547, "python3", "AI", ProcGpu(dedicated=GiB), ProcDetail(547, "python3"))]+ row = to_dict(snap)["processes"][0]+ self.assertEqual((row["vram_bytes"], row["gpu_pct"], row["shared_bytes"]), (GiB, 0.0, None))++ def test_wsl_rows_sort_by_the_size_they_show(self):+ from gputaskman.app import GpuTaskManager+ rows = [ProcRow(p, "python3", "AI", ProcGpu(attributed=False, inferred=v, inferred_by="consistent"),+ ProcDetail(p, "python3")) for p, v in ((1, GiB), (2, 5 * GiB), (3, None))]+ key = GpuTaskManager.SORTS[0][1]+ self.assertEqual([r.pid for r in sorted(rows, key=key)], [2, 1, 3])++ def test_native_hidden_vram_does_not_hide_measured_engine_load(self):+ snap = self.snapshot()+ snap.backend = "nvml"+ snap.procs[0].gpu.engines["compute"] = 47+ row = to_dict(snap)["processes"][0]+ self.assertIsNone(row["vram_bytes"])+ self.assertEqual(row["gpu_pct"], 47)+ self.assertEqual(row["engines"], {"compute": 47})+ for key in ("shared_bytes", "committed_bytes", "paged_out_bytes"):+ self.assertIsNone(row[key], key)+ text = next(line for line in to_text(snap).splitlines() if "4242" in line)+ self.assertIn("47", text)++ def test_measured_zero_is_not_changed_to_unknown(self):+ snap = self.snapshot()+ snap.backend = "nvml"+ snap.procs[0].gpu.attributed = True+ row = to_dict(snap)["processes"][0]+ self.assertEqual(row["vram_bytes"], 0)+ self.assertEqual(row["gpu_pct"], 0)+ self.assertEqual(row["engines"], {})+ self.assertIsNone(row["shared_bytes"])+ snap.backend = "wsl"+ snap.procs[0].host, snap.procs[0].origin = True, "windows"+ row = to_dict(snap)["processes"][0]+ for key in ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes", "gpu_pct"):+ self.assertEqual(row[key], 0, key)++ def test_model_detail_header_uses_inferred_or_unknown_vram(self):+ for inferred, label in ((None, "n/a"), (0, "0 MB*"), (3 * GiB, "3.00 GB*")):+ with self.subTest(inferred=inferred):+ snap = self.snapshot()+ p = snap.procs[0]+ p.llm = LlmView()+ p.gpu.inferred, p.gpu.inferred_by = inferred, "consistent"+ self.assertIn(f"== python3 (4242) VRAM {label}", to_text(snap))+ self.assertIsNone(to_dict(snap)["processes"][0]["vram_bytes"])++ def test_empty_wsl_view_explains_itself(self):+ snap = self.snapshot()+ snap.procs = []+ self.assertIn("No Linux process", to_text(snap))++ def test_llm_view_does_not_claim_cpu_inference_when_vram_is_unknown(self):+ d = ProcDetail(4242, "llama-server", cmdline=["llama-server", "-c", "4096"])+ d.models = [SimpleNamespace(path="/m/x.gguf", gguf=True, sources=["mapped"], store_name=None,+ kind="", kind_hint="", size=GiB)]+ with patch("gputaskman.llm.dataclasses.replace", side_effect=lambda m, **kw: m), \+ patch("gputaskman.llm.read_gguf", return_value=GgufInfo(path="/m/x.gguf", file_size=GiB, version=3)):+ known = build_view(d, 0, 0, None)+ unknown = build_view(d, 0, 0, None, attributed=False)+ self.assertTrue(any("CPU inference" in n for n in known.notes))+ self.assertFalse(any("CPU inference" in n for n in unknown.notes))+ self.assertTrue(any("unknown" in n for n in unknown.notes))+++ if __name__ == "__main__":+ unittest.main()diff --cc tests/test_wslhost.pyindex 0000000,38bfd1a..f352f9dmode 000000,100644..100644--- a/tests/test_wslhost.py+++ b/tests/test_wslhost.py@@@ -1,0 -1,530 +1,534 @@@+ from __future__ import annotations++ import io+ import json+ import os+ import tempfile+ import time+ import unittest+ from unittest.mock import Mock, mock_open, patch++ from gputaskman import collector as collector_mod+ from gputaskman import wslhost+ from gputaskman.collector import Collector, ProcRow, Snapshot+ from gputaskman.llm import LlmView+ from gputaskman.pdh import AdapterGpu, ProcGpu+ from gputaskman.procinfo import MappedModel, ProcDetail+ from gputaskman.report import to_dict, to_text+ from gputaskman.wslhost import ContextTracker, HostSample, HostStream, match_adapter, parse_line++ GiB = 1 << 30+ MiB = 1 << 20+ LUID = "luid_0x00000000_0x0000ff9b_phys_0"+ RTX = 0xFF9B+++ def line(rows, names):+ return json.dumps({"s": rows, "n": names})+++ SAMPLE = line(+ [+ ["local usage", f"pid_24820_{LUID}", 2.0 * GiB], # vmwp: one instance per guest context, summed+ ["local usage", f"pid_24820_{LUID}", 0.4 * GiB],+ ["local usage", f"pid_2744_{LUID}", 300 * MiB],+ ["non local usage", f"pid_2744_{LUID}", 20 * MiB],+ ["utilization percentage", f"pid_24820_{LUID}_eng_0_engtype_3D", 40.0],+ ["utilization percentage", f"pid_2744_{LUID}_eng_0_engtype_3D", 3.0],+ ["dedicated usage", LUID, 3.0 * GiB],+ ["dedicated usage", "luid_0x00000000_0x00011431_phys_0", 2 * MiB],+ ["local usage", "garbage", 1.0],+ ],+ {"24820": "vmwp", "2744": "dwm", "4": "System"},+ )+++ class ParseTests(unittest.TestCase):+ def test_parses_counters_like_the_windows_backend(self):+ s = parse_line(SAMPLE, now=1.0)+ self.assertEqual(s.procs[RTX][24820].dedicated, 2.4 * GiB)+ self.assertEqual(s.procs[RTX][24820].engines["3d"], 40.0)+ self.assertEqual(s.procs[RTX][2744].shared, 20 * MiB)+ self.assertEqual(s.adapters[RTX].dedicated, 3.0 * GiB)+ self.assertEqual(s.names[24820], "vmwp")++ def test_bad_lines_are_ignored(self):+ self.assertIsNone(parse_line("WARNING: something"))+ self.assertIsNone(parse_line("[1, 2]"))+ self.assertEqual(parse_line(line([["local usage"]], {})).procs, {})+++ class MatchTests(unittest.TestCase):+ def sample(self, *usages):+ return HostSample(taken=0, adapters={i + 1: AdapterGpu(dedicated=u) for i, u in enumerate(usages)})++ def test_matches_adapter_by_device_usage(self):+ self.assertEqual(match_adapter(self.sample(5.1 * GiB, 0), 5.3 * GiB), 1)+++ def test_ignores_adapters_without_dedicated_counters(self):++ self.assertEqual(match_adapter(self.sample(None, 5 * GiB), 5 * GiB), 2)++ self.assertIsNone(match_adapter(self.sample(None), 5 * GiB))+++ def test_refuses_distant_or_ambiguous_matches(self):+ self.assertIsNone(match_adapter(self.sample(1 * GiB), 5 * GiB))+ self.assertIsNone(match_adapter(self.sample(5.0 * GiB, 5.1 * GiB), 5.05 * GiB))+ self.assertIsNone(match_adapter(self.sample(5 * GiB), 0))+++ class InteropTests(unittest.TestCase):+ def test_interop_flag_reads_binfmt_entry(self):+ with patch("gputaskman.wslhost.glob.glob", return_value=["/proc/sys/fs/binfmt_misc/WSLInterop"]):+ with patch("builtins.open", mock_open(read_data="enabled\ninterpreter /init\n")):+ self.assertTrue(wslhost.interop_enabled())+ with patch("builtins.open", mock_open(read_data="disabled\n")):+ self.assertFalse(wslhost.interop_enabled())+ with patch("gputaskman.wslhost.glob.glob", return_value=[]):+ self.assertFalse(wslhost.interop_enabled())++ def test_stream_reads_lines_from_powershell(self):+ calls = []++ class FakeProc:+ def __init__(self, args, **kw):+ calls.append(args)+ self.stdout = io.StringIO(SAMPLE + "\n")++ def poll(self):+ return 0++ h = HostStream(exe="powershell.exe", popen=FakeProc)+ self.assertTrue(h.wait(2))+ deadline = time.time() + 2+ while h.status == "starting" and time.time() < deadline:+ time.sleep(0.01)+ self.assertIsNotNone(h.latest())+ self.assertEqual(calls[0][:4], ["powershell.exe", "-NoProfile", "-NonInteractive", "-EncodedCommand"])++ def test_a_bad_byte_ends_the_stream_visibly_so_it_restarts(self):+ class BadStdout:+ def __iter__(self):+ yield SAMPLE + "\n"+ raise UnicodeDecodeError("utf-8", b"\x82", 0, 1, "invalid start byte")++ class FakeProc:+ def __init__(self, args, **kw):+ self.encoding = kw.get("encoding")+ self.stdout = BadStdout()++ def poll(self):+ return 0++ h = HostStream(exe="powershell.exe", popen=FakeProc)+ deadline = time.time() + 2+ while not h.ended and time.time() < deadline:+ time.sleep(0.01)+ self.assertTrue(h.ended)+ self.assertEqual(h._proc.encoding, "utf-8")++ def test_missing_powershell_is_reported(self):+ with patch("gputaskman.wslhost.powershell_path", return_value=None):+ h = HostStream()+ self.assertTrue(h.status.startswith("unavailable"))+ self.assertIsNone(h.latest())+++ def at(values_gib):+ return [v * GiB for v in values_gib]+++ class TrackerTests(unittest.TestCase):+ """Replays context sequences measured on WSL2: contexts newest first, one per /dev/dxg holder."""++ def setUp(self):+ self.tmp = tempfile.TemporaryDirectory()+ self.addCleanup(self.tmp.cleanup)+ self.path = os.path.join(self.tmp.name, "ctx.json")+ self.clock = 1000.0++ def make(self, boot="boot-1"):+ return ContextTracker(path=self.path, boot_id=boot, self_pid="S")++ def step(self, t, contexts, holders):+ self.clock += 1.1+ return t.update(RTX, at(contexts), holders, now=self.clock)++ def check(self, result, expected, reason=""):+ guesses, why = result+ self.assertIn(reason, why)+ got = {pid: (round(g.bytes / GiB, 2), "*" if g.confidence == "consistent" else "?")+ for pid, g in guesses.items() if pid != "S"}+ self.assertEqual(got, expected)++ def test_processes_started_while_watching_are_pinned(self):+ t = self.make()+ h = [("S", 900)]+ self.check(self.step(t, [0.0, 0.0], h), {}) # our context (newest) and one foreign+ h.append(("A", 950))+ self.check(self.step(t, [1.19, 0.0, 0.0], h), {"A": (1.19, "*")})+ h.append(("F1", 960))+ self.check(self.step(t, [0.4, 1.19, 0.0, 0.0], h), {"A": (1.19, "*"), "F1": (0.4, "*")})+ h.append(("B", 970))+ self.check(self.step(t, [3.05, 0.4, 1.19, 0.0, 0.0], h),+ {"A": (1.19, "*"), "F1": (0.4, "*"), "B": (3.05, "*")})+ h.remove(("A", 950)) # the oldest job exits; the others keep their contexts+ self.check(self.step(t, [3.05, 0.4, 0.0, 0.0], h), {"F1": (0.4, "*"), "B": (3.05, "*")})++ def test_a_workload_started_after_us_is_not_taken_for_our_context(self):+ t = self.make()+ h = [("S", 900), ("W", 950)] # W created its CUDA context between our NVML init and the first sample+ self.check(self.step(t, [20 / 1024, 0.0], h), {"W": (0.02, "*")})+ self.check(self.step(t, [5.0, 0.0], h), {"W": (5.0, "*")})++ def test_cache_forgets_a_foreign_context_that_exited(self):+ t = self.make()+ self.check(self.step(t, [0.0, 2.0], [("S", 900)]), {}) # only us: 2.0 is foreign+ self.check(self.step(t, [0.0], [("S", 900)]), {}) # it exits+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [])+ t = self.make() # next run, same boot: a local 2 GiB job must not be taken for the old foreign one+ self.check(self.step(t, [0.0, 2.0], [("S", 900), ("A", 800)]), {"A": (2.0, "*")})++ def test_equal_sizes_are_right_now_but_never_evidence_later(self):+ t = self.make()+ h = [("S", 900), ("A", 13), ("B", 14)]+ self.check(self.step(t, [0.0, 0.2, 0.2], h), {"B": (0.2, "*"), "A": (0.2, "*")})+ # one of them loads a model: which one is unknown, so neither keeps the "*"+ self.check(self.step(t, [0.0, 4.0, 0.2], h), {"B": (4.0, "?"), "A": (0.2, "?")})++ def test_new_process_waits_for_its_own_context_instead_of_eliminating(self):+ t = self.make()+ self.check(self.step(t, [0.0, 2.0, 0.15], [("S", 900), ("B", 10)]), {"B": (2.0, "?")})+ h = [("S", 900)] # B exits; 0.15 (WSLg) was never seen without B, so it isn't known as foreign+ self.check(self.step(t, [0.0, 0.15], h), {})+ h.append(("A", 1001))+ self.check(self.step(t, [0.0, 0.15], h), {"A": (0.15, "?")}) # A's context isn't there yet+ self.check(self.step(t, [6.0, 0.0, 0.15], h), {"A": (6.0, "*")})++ def test_cached_foreign_sizes_only_rank_startup_guesses(self):+ self.step(self.make(), [0.0, 0.15], [("S", 900)]) # only gputaskman holds the GPU: 0.15 is foreign+ t = self.make() # a later start in the same boot, with a job already running+ self.check(self.step(t, [0.0, 0.15, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})+ self.check(self.step(t, [0.0, 0.15, 2.5], [("S", 900), ("X", 5)]), {"X": (2.5, "?")})++ def test_foreign_cache_clears_immediately_when_last_foreign_context_exits(self):+ t = self.make()+ self.step(t, [0.0, 2.0], [("S", 900)])+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [2 * GiB])+ self.step(t, [0.0], [("S", 900)])+ self.assertEqual(t.cached_foreign, [])+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [])+ self.check(self.step(self.make(), [0.0, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})++ def test_foreign_cache_clears_while_local_workload_remains(self):+ t = self.make()+ self.step(t, [0.0, 0.4], [("S", 900)])+ h = [("S", 900), ("X", 950)]+ self.step(t, [2.0, 0.0, 0.4], h)+ self.check(self.step(t, [2.0, 0.0], h), {"X": (2.0, "*")})+ with open(self.path, encoding="utf-8") as f:+ self.assertEqual(json.load(f)["foreign"], [])++ def test_stale_cache_never_takes_a_local_context(self):+ self.step(self.make(), [0.0, 2.0], [("S", 900)])+ # The foreign process exits while the monitor is off; a local process reuses that size.+ self.check(self.step(self.make(), [0.0, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "*")})++ def test_first_sample_finds_monitor_zero_instead_of_newest_small_context(self):+ t = self.make()+ h = [("S", 900), ("X", 950)]+ guesses, _ = t.update(RTX, [20 * MiB, 0], h, now=self.clock)+ self.assertEqual(guesses["S"].bytes, 0)+ self.assertEqual(guesses["X"].bytes, 20 * MiB)+ self.check(self.step(t, [5.0, 0.0], h), {"X": (5.0, "*")})++ def test_equal_zero_contexts_never_pin_monitor_ownership(self):+ t = self.make()+ h = [("S", 900), ("X", 950)]+ self.check(self.step(t, [0.0, 0.0], h), {"X": (0.0, "*")})+ self.assertFalse(any(e.sure for e in t.entries))+ self.check(self.step(t, [5.0, 0.0], h), {"X": (5.0, "?")})+ self.assertFalse(any(e.label == "S" and e.value for e in t.entries))++ def test_idle_zero_ambiguity_does_not_pollute_foreign_cache(self):+ t = self.make()+ self.step(t, [0.0, 0.15, 0.0], [("S", 900)])+ self.assertEqual(t.cached_foreign, [0.15 * GiB])+ self.assertFalse(any(e.label == "S" and e.sure for e in t.entries))+ self.check(self.step(t, [2.0, 0.0, 0.15, 0.0], [("S", 900), ("X", 950)]), {"X": (2.0, "*")})++ def test_a_reused_pid_is_a_new_process(self):+ t = self.make()+ self.check(self.step(t, [0.0, 1.0], [("S", 900), ("P", 10)]), {"P": (1.0, "*")})+ self.check(self.step(t, [0.0], [("S", 900)]), {})+ self.check(self.step(t, [2.0, 0.0], [("S", 900), ("P", 2000)]), {"P": (2.0, "*")})+ self.check(self.step(t, [2.0, 0.0], [("S", 900), ("P", 2000)]), {"P": (2.0, "*")})++ def test_already_running_processes_are_guesses_until_evidence(self):+ t = self.make()+ h = [("S", 900), ("C", 14), ("B", 13)]+ self.check(self.step(t, [0.0, 2.12, 3.05, 0.0], h), {"C": (2.12, "?"), "B": (3.05, "?")})+ h.append(("D", 950))+ self.check(self.step(t, [0.5, 0.0, 2.12, 3.05, 0.0], h), {"D": (0.5, "*"), "C": (2.12, "?"), "B": (3.05, "?")})+ # the guesses never promote themselves, however long they stay unchanged+ for _ in range(5):+ self.check(self.step(t, [0.5, 0.0, 2.12, 3.05, 0.0], h), {"D": (0.5, "*"), "C": (2.12, "?"), "B": (3.05, "?")})++ def test_no_holders_teaches_foreign_without_confirming_future_owners(self):+ self.make().update(RTX, at([0.4]), [], now=self.clock) # nothing in this distro: 0.4 is foreign+ t = self.make() # a fresh start in the same boot, with a job already running+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})+ t = self.make(boot="boot-2") # a new boot knows nothing+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (0.4, "?")})++ def test_context_that_lags_its_process_is_still_pinned(self):+ t = self.make()+ self.step(t, [0.0], [("S", 900)])+ h = [("S", 900), ("P", 950)]+ self.check(self.step(t, [0.0], h), {}, reason="without a VM context")+ self.check(self.step(t, [0.8, 0.0], h), {"P": (0.8, "*")})++ def test_exited_process_context_is_not_handed_to_a_newcomer(self):+ t = self.make()+ self.step(t, [0.0], [("S", 900)])+ self.step(t, [1.19, 0.0], [("S", 900), ("A", 950)])+ # A exits and B starts; the host sample still shows A's context, B's hasn't appeared yet+ h = [("S", 900), ("B", 960)]+ guesses, _ = self.step(t, [1.19, 0.0], h)+ self.assertNotIn("B", guesses)+ self.check(self.step(t, [2.5, 0.0], h), {"B": (2.5, "*")})++ def test_model_loading_keeps_its_context(self):+ t = self.make()+ h = [("S", 900)]+ self.step(t, [0.0], h)+ h.append(("L", 950))+ self.check(self.step(t, [0.3, 0.0], h), {"L": (0.3, "*")})+ for gib in (2.5, 6.0, 11.0): # GBs per refresh while weights upload+ self.check(self.step(t, [gib, 0.0], h), {"L": (gib, "*")})++ def test_context_without_a_new_process_is_foreign(self):+ t = self.make()+ h = [("S", 900)]+ self.step(t, [0.0], h)+ h.append(("A", 950))+ self.step(t, [1.0, 0.0], h)+ # another distro / Docker starts something: a new context, but no new Linux process here+ self.check(self.step(t, [0.7, 1.0, 0.0], h), {"A": (1.0, "*")})+ h.append(("B", 990))+ self.check(self.step(t, [2.0, 0.7, 1.0, 0.0], h), {"A": (1.0, "*"), "B": (2.0, "*")})++ def test_equal_sizes_are_sure_even_when_order_is_not(self):+ t = self.make()+ self.check(self.step(t, [0.0, 0.4, 0.4], [("S", 900), ("F1", 5), ("F2", 6)]),+ {"F1": (0.4, "*"), "F2": (0.4, "*")})++ def test_nearly_equal_sizes_stay_guesses(self):+ t = self.make() # 0.40 vs 0.42 GiB: within tolerance, but swapping owners changes the shown bytes+ self.check(self.step(t, [0.0, 0.4, 0.42], [("S", 900), ("F1", 5), ("F2", 6)]),+ {"F1": (0.42, "?"), "F2": (0.4, "?")})++ def test_different_adapter_resets_history(self):+ self.make().update(RTX, at([0.4]), [], now=self.clock)+ t = self.make()+ self.check(self.step(t, [0.0, 0.4, 2.0], [("S", 900), ("X", 5)]), {"X": (2.0, "?")})+ guesses, _ = t.update(0x1234, at([0.0, 0.4, 2.0]), [("S", 900), ("X", 5)], now=self.clock + 5)+ self.assertEqual(guesses["X"].bytes, 0.4 * GiB)+ self.assertEqual(guesses["X"].confidence, "order")+++ class FakeStream:+ def __init__(self, sample, status="ok"):+ self.sample, self.status = sample, status+ self.closed = False++ @property+ def ended(self):+ return self.status.startswith("unavailable")++ def latest(self):+ return self.sample++ def close(self):+ self.closed = True+++ class HostRowTests(unittest.TestCase):+ def collector(self, stream):+ c = Collector.__new__(Collector) # only _add_host_rows is exercised+ c.host = stream+ c._host_luid = None+ c._host_restarted = time.time()+ c.tracker = ContextTracker(path=os.path.join(self.tmp.name, "ctx.json"), boot_id="boot", self_pid=os.getpid())+ c._own_start = time.time()+ return c++ def setUp(self):+ self.tmp = tempfile.TemporaryDirectory()+ self.addCleanup(self.tmp.cleanup)++ def snap(self, used=3.1 * GiB):+ linux = ProcRow(2744, "python3", "AI", ProcGpu(attributed=False), ProcDetail(2744, "python3"))+ return Snapshot(taken=1, adapter=None, adapters=[], nv=None, driver="", cuda="", vram_total=24 * GiB,+ vram_used=int(used), shared_used=0, backend="wsl", procs=[linux], host_status="")++ def test_wsl_vm_and_windows_rows(self):+ snap = self.snap()+ self.collector(FakeStream(parse_line(SAMPLE, now=time.time())))._add_host_rows(snap)+ self.assertEqual(snap.host_status, "ok")+ rows = {r.key: r for r in snap.procs}+ # Linux PID 2744 and Windows dwm PID 2744 coexist: distinct keys+ self.assertEqual(set(rows), {"2744", "w2744", "w24820"})+ vm = rows["w24820"]+ self.assertEqual((vm.name, vm.kind, vm.gpu.dedicated), ("WSL VM (vmwp.exe)", "wsl-vm", 2.4 * GiB))+ self.assertEqual((rows["w2744"].name, rows["w2744"].kind), ("dwm.exe", "compositor"))- self.assertEqual(snap.shared_used, 0)++ self.assertIsNone(snap.shared_used) # sample has no adapter shared-memory counter++ text = to_text(snap)+ self.assertIn("W:24820", text)+ self.assertIn("WSL VM (vmwp.exe)", text)+ self.assertIn("exact total", text)+ data = to_dict(snap)+ self.assertEqual({p["host"] for p in data["processes"]}, {None, "windows"})+ self.assertEqual(data["host_status"], "ok")++ def test_linux_process_gets_matched_vram(self):+ # contexts newest first: this monitor's own 0-byte NVML context, then the workload (3 GB)+ busy = line([["local usage", f"pid_24820_{LUID}", v * GiB] for v in (0.0, 3.0)]+ + [["dedicated usage", LUID, 3.1 * GiB]], {"24820": "vmwp"})+ c = self.collector(FakeStream(parse_line(busy, now=time.time())))+ snap = self.snap()+ c._add_host_rows(snap)+ row = next(r for r in snap.procs if not r.host)+ self.assertEqual((row.gpu.inferred, row.gpu.inferred_by), (3.0 * GiB, "consistent"))+ self.assertFalse(row.gpu.attributed) # engines are still unknown+ row.llm = LlmView()+ text = to_text(snap)+ self.assertIn("3.00 GB*", text)+ self.assertIn("== python3 (2744) VRAM 3.00 GB*", text)+ self.assertIn("* matched", text)+ data = to_dict(snap)["processes"][1]+ self.assertEqual((data["vram_inferred_bytes"], data["vram_inferred_by"]), (3.0 * GiB, "consistent"))+ self.assertEqual(snap.attributed, 3 * GiB) # the Linux child is not added to the VM total again++ def test_current_inferred_vram_sets_first_sample_scan_priority(self):+ big_pid = Collector.DEEP_TOP_N + 1+ total = 8 * GiB + Collector.DEEP_TOP_N * MiB+ busy = line([["local usage", f"pid_24820_{LUID}", v]+ for v in [0, 8 * GiB] + [MiB] * Collector.DEEP_TOP_N]+ + [["dedicated usage", LUID, total]], {"24820": "vmwp"})+ c = self.collector(FakeStream(parse_line(busy, now=time.time())))+ c.adapter, c.adapters, c.gpu_index, c.scan_budget = None, [], 0, None+ c.host_status = ""+ c.nvml = Mock(ok=False, driver="", cuda_version="")+ c.counters = Mock(outside={})+ c.counters.sample.return_value = (+ {RTX: {pid: ProcGpu(attributed=False) for pid in range(1, big_pid + 1)}},+ {RTX: AdapterGpu(dedicated=total)})+ details = {pid: ProcDetail(pid, "python3", create_time=pid, parents=[])+ for pid in range(1, big_pid + 1)}+ c.inspector = Mock()+ c.inspector.get.side_effect = details.__getitem__+ c.inspector.needs_scan.return_value = True++ def scan(d, mem):+ if d.pid == big_pid:+ d.models = [MappedModel("large.gguf", 8 * GiB, 0, True, kind="LLM")]++ c.inspector.scan.side_effect = scan+ c.prober = Mock()+ c.prober.get.return_value = None+ with patch.object(collector_mod, "BACKEND", "wsl"), \+ patch.object(collector_mod, "build_view", return_value=LlmView()) as view:+ snap = c.sample()+ self.assertEqual(c.inspector.scan.call_count, Collector.DEEP_TOP_N)+ self.assertEqual(c.inspector.scan.call_args_list[0].args, (details[big_pid], 8 * GiB))+ self.assertNotIn(Collector.DEEP_TOP_N, [call.args[0].pid for call in c.inspector.scan.call_args_list])+ self.assertEqual([r.pid for r in snap.procs[:2]], [24820, big_pid])+ self.assertEqual(snap.procs[1].kind, "LLM") # classification follows the scan+ self.assertEqual(view.call_args.args[1], 8 * GiB)+ self.assertTrue(view.call_args.kwargs["attributed"])+ self.assertEqual(snap.attributed, total)+ c.scan_budget = 0.0+ c.inspector.scan.reset_mock()+ c.sample()+ c.inspector.scan.assert_not_called()++ def test_order_guess_is_marked(self):+ # an extra (foreign) context and no history: placement is a guess+ busy = line([["local usage", f"pid_24820_{LUID}", v * GiB] for v in (0.0, 3.0, 0.4)]+ + [["dedicated usage", LUID, 3.1 * GiB]], {"24820": "vmwp"})+ c = self.collector(FakeStream(parse_line(busy, now=time.time())))+ snap = self.snap()+ c._add_host_rows(snap)+ row = next(r for r in snap.procs if not r.host)+ self.assertEqual(row.gpu.inferred_by, "order")+ self.assertIn("3.00 GB?", to_text(snap))+ self.assertIn("order guess", to_text(snap))++ def test_split_reason_is_reported(self):+ snap = self.snap() # SAMPLE has 2 vmwp contexts; add Linux rows until there are more holders+ snap.procs += [ProcRow(p, "python3", "AI", ProcGpu(attributed=False), ProcDetail(p, "python3"))+ for p in (2745, 2746)]+ self.collector(FakeStream(parse_line(SAMPLE, now=time.time())))._add_host_rows(snap)+ self.assertIn("without a VM context", snap.wsl_split)+ self.assertIn("Linux VRAM split:", to_text(snap))++ def test_unmatched_adapter_adds_nothing(self):+ snap = self.snap(used=12 * GiB)+ c = self.collector(FakeStream(parse_line(SAMPLE, now=time.time())))+ c._add_host_rows(snap)+ self.assertEqual(snap.host_status, "starting") # retried for a while first+ c._host_restarted -= 60+ snap = self.snap(used=12 * GiB)+ c._add_host_rows(snap)+ self.assertTrue(snap.host_status.startswith("unmatched"))+ self.assertEqual(len(snap.procs), 1)+ self.assertIn("unmatched", to_text(snap))++ def test_adapter_match_is_kept_while_memory_moves(self):+ c = self.collector(FakeStream(parse_line(SAMPLE, now=time.time())))+ c._add_host_rows(self.snap())+ moving = self.snap(used=12 * GiB) # a model loading between the NVML and host readings+ c._add_host_rows(moving)+ self.assertEqual(moving.host_status, "ok")+ self.assertEqual(len(moving.procs), 3)++ def test_late_samples_stay_visible_and_say_so(self):+ snap = self.snap()+ self.collector(FakeStream(parse_line(SAMPLE, now=time.time() - 8)))._add_host_rows(snap)+ self.assertTrue(snap.host_data)+ self.assertIn("8s old", snap.host_status)+ self.assertEqual(len(snap.procs), 3)++ def test_ended_stream_is_restarted(self):+ dead = FakeStream(None, status="unavailable (counter stream ended)")+ c = self.collector(dead)+ c._host_restarted = time.time() - 60+ with patch.object(collector_mod, "HostStream", return_value=FakeStream(None, "starting")) as stream:+ c._add_host_rows(self.snap())+ self.assertTrue(dead.closed)+ stream.assert_called_once()++ def test_no_sample_yet_is_starting(self):+ snap = self.snap()+ self.collector(FakeStream(None, status="starting"))._add_host_rows(snap)+ self.assertEqual(snap.host_status, "starting")++ def test_interop_off_and_flag_off_never_start_powershell(self):+ with patch.object(collector_mod, "BACKEND", "wsl"), patch.object(collector_mod, "enumerate_adapters",+ return_value=[]), \+ patch.object(collector_mod, "GpuCounters"), patch.object(collector_mod, "Nvml"), \+ patch.object(collector_mod, "HostStream") as stream:+ with patch.object(collector_mod, "interop_enabled", return_value=False):+ self.assertEqual(Collector().host_status, "off (WSL interop is disabled)")+ with patch.object(collector_mod, "interop_enabled", return_value=True):+ self.assertEqual(Collector(host=False).host_status, "off (--no-host)")+ stream.assert_not_called()+ Collector()+ stream.assert_called_once()+++ if __name__ == "__main__":+ unittest.main()diff --cc tools/build_portable.pyindex 8ef8226,0a86931..0ee6831--- a/tools/build_portable.py+++ b/tools/build_portable.py@@@ -18,14 -19,27 +19,27 @@@ import zipfilROOT = Path(__file__).resolve().parents[1]- MACHINES = {"x64": 0x8664, "arm64": 0xAA64}+ MACHINES = {"x64": 0x8664, "arm64": 0xAA64} # PE machine+ ELF_MACHINES = {"x64": 0x3E, "arm64": 0xB7} # ELF e_machine-# Provided by Windows or the installed NVIDIA driver, never by the portable app.-FORBIDDEN = ("nvml.dll", "nvcuda.dll", "dxgi.dll", "pdh.dll", "libnvidia-ml.so", "libcuda.so", "libdxcore.so")++# Provided by Windows or the installed GPU driver, never by the portable app.++FORBIDDEN = ("nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll", "libnvidia-ml.so", "libcuda.so", "libdxcore.so")+ PLATFORMS = {"win-amd64": ("windows", "x64"), "win-arm64": ("windows", "arm64"),+ "linux-x86_64": ("linux", "x64"), "linux-aarch64": ("linux", "arm64")}- def architecture(platform_tag: str) -> str:- platforms = {"win-amd64": "x64", "win-arm64": "arm64"}- if platform_tag not in platforms:+ def target(platform_tag: str) -> tuple[str, str]:+ if platform_tag not in PLATFORMS:raise ValueError(f"Unsupported build interpreter: {platform_tag}")- return platforms[platform_tag]+ return PLATFORMS[platform_tag]+++ def architecture(platform_tag: str) -> str:+ return target(platform_tag)[1]+++ def is_forbidden(name: str) -> bool:+ name = name.lower()+ return any(name == f or name.startswith(f + ".") for f in FORBIDDEN)def validate_version(version: str, tag: str | None = None) -> None:@@@ -104,10 -172,10 +172,10 @@@ def add_notices(bundle: Path, system: s)(bundle / "THIRD-PARTY-NOTICES.txt").write_text("Python: see PYTHON-LICENSE.txt.\n"- "PyInstaller bootloader: see licenses\\pyinstaller (including its exception).\n"- "NVIDIA Python bindings: see licenses\\nvidia-ml-py.\n"- "Runtime package metadata and license texts are included in _internal\\*.dist-info.\n"- "GPU driver and Windows DLLs are provided by the installed driver and operating system.\n",+ f"PyInstaller bootloader: see licenses{sep}pyinstaller (including its exception).\n"+ f"NVIDIA Python bindings: see licenses{sep}nvidia-ml-py.\n"+ f"Runtime package metadata and license texts are included in _internal{sep}*.dist-info.\n"- "NVIDIA libraries are provided by the installed driver, system libraries by the operating system.\n",++ "GPU libraries are provided by the installed driver, system libraries by the operating system.\n",encoding="utf-8",)diff --cc tools/portable.specindex 68dd7a2,85154e8..9533af3--- a/tools/portable.spec+++ b/tools/portable.spec@@@ -9,12 -9,16 +9,16 @@@ analysis = Analysisdatas=collect_data_files("textual") + copy_metadata("gputaskman", recursive=True),)- # These belong to Windows or the installed GPU driver, not the portable app.- driver_dlls = {"nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll"}- analysis.binaries = [- entry for entry in analysis.binaries- if Path(entry[0]).name.lower() not in driver_dlls- ]+ # These belong to the operating system or the installed NVIDIA driver, not the portable app.-driver_libs = ("nvml.dll", "nvcuda.dll", "dxgi.dll", "pdh.dll", "libnvidia-ml.so", "libcuda.so", "libdxcore.so")++driver_libs = ("nvml.dll", "nvcuda.dll", "atiadlxx.dll", "dxgi.dll", "pdh.dll", "libnvidia-ml.so", "libcuda.so", "libdxcore.so")+++ def is_driver_lib(name):+ name = name.lower()+ return any(name == lib or name.startswith(lib + ".") for lib in driver_libs)+++ analysis.binaries = [entry for entry in analysis.binaries if not is_driver_lib(Path(entry[0]).name)]pyz = PYZ(analysis.pure)exe = EXE(