A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
Never pair a reused PID in diff
Scott Hanselman780bc9f0fc7a
diff --git a/AGENTS.md b/AGENTS.mdindex 7f345a6..cd8e24c 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -80,7 +80,7 @@ identify everything immediately; the TUI spreads scans across refreshes.| Support an app that reads and closes its model | `handy.py` as the template | Read-only settings plus a bounded, mtime-cached log tail; never read transcription content. Generalize into data-driven adapters once a second app needs one || Add a GPU vendor or backend | `dxgi.py`, `collector.py`, a new module | Pair devices by LUID or PCI bus ID, never by name or index; load vendor DLLs by absolute system path || Add a column or JSON field | `report.py` and `app.py` | Follow the null rules above; update the README tables. If it is a measured byte count, consider adding it to `diff.py` `MEASURED` |-| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero. Refuse only when both snapshots name conflicting adapters (vendor, PCI address, a non-Windows key); when they cannot be compared, warn (`name-only`) instead, because a Windows LUID changes at boot and `diff` has no override flag |+| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero. Refuse only when both snapshots name conflicting adapters (vendor, PCI address, a non-Windows key); when they cannot be compared, warn (`name-only`) instead, because a Windows LUID changes at boot and `diff` has no override flag. A PID whose two `started_at` values are both valid and more than `SAME_PROCESS_SECS` apart is a different process: no pairing rule may join it (unknown or malformed start times never conflict, so older snapshots pair as before) || Change system RAM/CPU | `host.py` | Machine memory and load as the OS reports them: never GPU bytes, never added to `Snapshot.attributed`, and `diff` ignores it. Unknown is `None`/`null` and a failure carries a reason, never zeros. The JSON key is `system` because `host` already means the WSL Windows host. Windows uses the NT process list (psutil per process takes about 6 s per pass for 1,000 processes), checked against psutil at startup; CPU rates are keyed by `(pid, create_time)` so a reused PID never inherits another process's history (on the psutil path, `is_running()` is what makes psutil drop its cached object for a reused PID; keep that call) || Change WSL matching | `wslhost.py` `ContextTracker` | Keep `tests/test_wslhost.py` replays passing; they are measured sequences |diff --git a/README.md b/README.mdindex 994888f..be560ba 100644--- a/README.md+++ b/README.md@@ -596,6 +596,15 @@ model reload often restarts the server under a new PID. A process missing from o`not listed`, not zero: it was not running or used too little GPU to be listed. Processes thatmoved by less than 16 MB are hidden from the text output but included in `--json`.+A PID is not an identity, because both Windows and Linux reuse them. Each process in `--json` carries+`started_at` (its creation time in Unix seconds, taken from data the monitor already reads). The same+PID with start times more than 2 seconds apart is a different process, so no rule pairs it: the old+one shows as `gone`, the new one as `new`, and the device totals still compare normally. A missing or+`null` `started_at` (a protected process, a WSL host row, or a snapshot from an older version) never+counts as a conflict, so those rows are paired as described above. Linux derives start times from the+boot time in whole seconds, hence the 2-second allowance; a larger clock step can make one live+process look reused, which shows as `gone` plus `new` and never as a wrong change.+Two cards of one model share a name, so `diff` also compares the adapter each snapshot recorded.It refuses a pair whose vendor ID, PCI address or (off Windows) adapter key conflicts. A Windowsadapter key changes at boot and when an adapter is re-enabled, so there it only produces a warning.diff --git a/src/gputaskman/diff.py b/src/gputaskman/diff.pyindex 479fb05..59cbdb3 100644--- a/src/gputaskman/diff.py+++ b/src/gputaskman/diff.py@@ -8,11 +8,13 @@ the process has the file, not that it is resident in GPU memory.from __future__ import annotationsimport json+import mathfrom .llm import gbMiB = 1 << 20NOISE = 16 * MiB # smaller per-process moves are allocator and compositor churn, not a model loading+SAME_PROCESS_SECS = 2.0 # one process's start time can move this much between runs (Linux boot-time rounding, clock steps)MEASURED = ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes","vram_resident_bytes", "gtt_resident_bytes") # resident: amdgpu, shared buffers included (never summed)DEVICE = (("vram_used", "dedicated used"), ("shared_used", "shared used"),@@ -52,10 +54,26 @@ def _named(p: dict) -> bool:return p.get("host") != "outside" and p.get("name") not in PLACEHOLDER_NAMES+def _started(p: dict) -> float | None:+ v = p.get("started_at")+ return v if isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v) and v > 0 else None+++def _recycled(b: dict, a: dict) -> bool:+ """One PID with two different start times is two processes: the PID was reused in between.++ Unknown or malformed start times (older snapshots, protected processes) never conflict. Rows are only+ ever paired within one host, so the PID needs no namespace check here."""+ sb, sa = _started(b), _started(a)+ return (b.get("pid") is not None and b.get("pid") == a.get("pid") and sb is not None and sa is not None+ and abs(sb - sa) > SAME_PROCESS_SECS)++def match(before: list[dict], after: list[dict]) -> list[tuple[dict | None, dict | None, str]]:"""Pair rows by PID, then by name and ports, then by name, when each pairing is unambiguous.A model reload often restarts the server under a new PID. WSL host rows have their own PID space.+ A PID that comes back with a different start time is another process and is never paired, by any rule.Every row comes back exactly once: paired, new, or gone."""keys = [(p.get("host"), p.get("pid")) for p in before]@@ -66,7 +84,7 @@ def match(before: list[dict], after: list[dict]) -> list[tuple[dict | None, dictrest: list[dict] = []for a in after:i = by_pid.get((a.get("host"), a.get("pid")))- if i is not None and i in left and left[i].get("name") == a.get("name"):+ if i is not None and i in left and left[i].get("name") == a.get("name") and not _recycled(left[i], a):pairs.append((left.pop(i), a, "pid"))else:rest.append(a)@@ -86,8 +104,10 @@ def match(before: list[dict], after: list[dict]) -> list[tuple[dict | None, dictnews.setdefault(key(a), []).append(a)for k, found in news.items():if k is not None and len(found) == 1 and len(olds.get(k, [])) == 1:- pairs.append((left.pop(olds[k][0]), found[0], how))- rest.remove(found[0])+ i = olds[k][0]+ if not _recycled(left[i], found[0]):+ pairs.append((left.pop(i), found[0], how))+ rest.remove(found[0])pairs += [(None, a, "new") for a in rest]pairs += [(b, None, "gone") for b in left.values()]return pairsdiff --git a/src/gputaskman/report.py b/src/gputaskman/report.pyindex f2d2858..ac6c8c3 100644--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@ -116,6 +116,7 @@ def to_dict(s: Snapshot) -> dict:engines = has_engine_counters(s, p)item = {"pid": p.pid, "host": p.origin if p.host else None, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,+ "started_at": round(p.detail.create_time, 3) or None, # unknown (protected process, WSL host row) is null"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,"vram_inferred_by": p.gpu.inferred_by or None,"vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed and "dedicated" in p.gpu.available else None,diff --git a/tests/test_core.py b/tests/test_core.pyindex 1e8f2b6..f92060b 100644--- a/tests/test_core.py+++ b/tests/test_core.py@@ -360,6 +360,15 @@ class MemoryTests(unittest.TestCase):s.procs[0].gpu.available.clear()self.assertIsNone(to_dict(s)["attributed"])+ def test_started_at_is_the_creation_time_and_unknown_is_null(self):+ s = snapshot()+ s.procs[0].detail.create_time = 1762680123.4567891+ s.procs.append(ProcRow(30, "browser.exe", "browser", ProcGpu(dedicated=GiB), ProcDetail(30, "browser.exe"),+ host=True, origin="windows"))+ rows = to_dict(s)["processes"]+ self.assertEqual(rows[0]["started_at"], 1762680123.457)+ self.assertEqual([r["started_at"] for r in rows[1:]], [None, None]) # no creation time (0.0) is unknown, not 1970+def test_missing_adapter_memory_is_not_reported_as_zero(self):s = snapshot()s.vram_used = Nonediff --git a/tests/test_diff.py b/tests/test_diff.pyindex 72a2f9a..286ec91 100644--- a/tests/test_diff.py+++ b/tests/test_diff.py@@ -15,13 +15,18 @@ GiB = 1 << 30MiB = 1 << 20-def proc(pid, name, vram=None, shared=0, host=None, ports=(), llm=None, files=(), inferred=None):+def proc(pid, name, vram=None, shared=0, host=None, ports=(), llm=None, files=(), inferred=None, started=...):return {"pid": pid, "host": host, "name": name, "vram_bytes": vram, "shared_bytes": shared,"committed_bytes": None, "paged_out_bytes": None, "vram_inferred_bytes": inferred,"ports": list(ports), "model_files": [{"path": f, "sources": ["mapped"]} for f in files],+ **({} if started is ... else {"started_at": started}), # absent: a snapshot from before started_at existed**({"llm": llm} if llm else {})}+def pairs(before, after):+ return {(b and b["pid"], a and a["pid"], how) for b, a, how in match(before, after)}++def snap(procs, used, attributed, taken=100.0, gpu="RTX", backend="wddm", adapter=...):return {"taken": taken, "gpu": gpu, "backend": backend, "vram_used": used, "shared_used": 0,"attributed": attributed, "processes": procs, **({} if adapter is ... else {"adapter": adapter})}@@ -65,6 +70,51 @@ class MatchTests(unittest.TestCase):self.assertIn("W:41", to_text(d))+class PidReuseTests(unittest.TestCase):+ """A PID that comes back with a different start time is another process, however alike the rows look."""++ def test_a_reused_pid_is_gone_and_new_with_or_without_ports(self):+ for ports in ([8080], []): # name+ports and unique-name fallbacks must not pair them either+ with self.subTest(ports=ports):+ before = [proc(42, "llama-server", ports=ports, started=100.0)]+ after = [proc(42, "llama-server", ports=ports, started=200.0)]+ self.assertEqual(pairs(before, after), {(None, 42, "new"), (42, None, "gone")})++ def test_one_lifetime_still_pairs_by_pid_within_the_rounding_tolerance(self):+ for later in (100.0, 101.9, 102.0, 98.0):+ with self.subTest(later=later):+ self.assertEqual(pairs([proc(42, "a.exe", started=100.0)], [proc(42, "a.exe", started=later)]),+ {(42, 42, "pid")})+ self.assertEqual(pairs([proc(42, "a.exe", started=100.0)], [proc(42, "a.exe", started=102.5)]),+ {(None, 42, "new"), (42, None, "gone")})++ def test_unknown_or_malformed_start_times_keep_the_old_heuristic_pairing(self):+ # ... means the key is absent (an older snapshot); the rest are values a snapshot could carry+ for before_value, after_value in ((..., 200.0), (100.0, ...), (None, None), (None, 200.0), (0.0, 200.0),+ (-5.0, 200.0), (True, 200.0), ("100", 200.0), ([100.0], 200.0),+ (float("nan"), 200.0), (float("inf"), 200.0)):+ with self.subTest(before=before_value, after=after_value):+ before = [proc(42, "a.exe", started=before_value)]+ after = [proc(42, "a.exe", started=after_value)]+ self.assertEqual(pairs(before, after), {(42, 42, "pid")})+ # no PID means no reuse to speak of: rows without one are still paired by name alone+ self.assertEqual({how for _, _, how in match([proc(None, "a", started=100.0)], [proc(None, "a", started=200.0)])},+ {"name"})+ before = [proc(42, "llama-server", ports=[8080], started=100.0), proc(50, "solo.exe", started=100.0)]+ after = [proc(77, "llama-server", ports=[8080], started=200.0), proc(60, "solo.exe", started=200.0)]+ self.assertEqual(pairs(before, after), {(42, 77, "name+ports"), (50, 60, "name")})++ def test_a_refused_pair_does_not_make_the_other_rows_less_ambiguous(self):+ before = [proc(42, "llama-server", started=100.0)]+ after = [proc(42, "llama-server", started=200.0), proc(77, "llama-server", started=300.0)]+ self.assertEqual(pairs(before, after), {(None, 42, "new"), (None, 77, "new"), (42, None, "gone")})++ def test_reuse_is_judged_inside_one_pid_namespace(self):+ before = [proc(42, "x", started=100.0), proc(42, "x", host="windows")]+ after = [proc(42, "x", started=200.0), proc(42, "x", host="windows")]+ self.assertEqual(pairs(before, after), {(None, 42, "new"), (42, None, "gone"), (42, 42, "pid")})++class DiffTests(unittest.TestCase):def test_measured_runtime_and_mapped_evidence_stay_separate(self):before = snap([proc(1, "dwm.exe", vram=2 * GiB),@@ -130,6 +180,17 @@ class DiffTests(unittest.TestCase):self.assertIn("(no process changed by 16 MB or more)", to_text(d))self.assertNotIn("llama (1)", to_text(d))+ def test_a_reused_pid_keeps_the_device_change_but_makes_no_process_delta(self):+ before = snap([proc(42, "llama-server", vram=4 * GiB, ports=[8080], started=100.0)], 5 * GiB, 4 * GiB)+ after = snap([proc(42, "llama-server", vram=8 * GiB, ports=[8080], started=200.0)], 9 * GiB, 8 * GiB)+ d = diff(before, after)+ self.assertEqual(d["device"]["vram_used"]["delta"], 4 * GiB) # the device-wide change is still measured+ self.assertEqual(sorted(p["match"] for p in d["processes"]), ["gone", "new"])+ self.assertTrue(all(p["measured"]["vram_bytes"]["delta"] is None for p in d["processes"]))+ rows = [line for line in to_text(d).splitlines() if line.startswith("llama-server")]+ self.assertEqual(len(rows), 2) # one row per lifetime, each absent from the other snapshot+ self.assertTrue(all("not listed" in row for row in rows))+class AdapterGuardTests(unittest.TestCase):def pair(self, before, after, backend="wddm"):