Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Never pair a reused PID in diff

Scott Hanselman780bc9f0fc7a
diff --git a/AGENTS.md b/AGENTS.md
index 7f345a6..cd8e24c 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -80,7 +80,7 @@ identify everything immediately; the TUI spreads scans across refreshes.
| Support an app that reads and closes its model | `handy.py` as the template | Read-only settings plus a bounded, mtime-cached log tail; never read transcription content. Generalize into data-driven adapters once a second app needs one |
| Add a GPU vendor or backend | `dxgi.py`, `collector.py`, a new module | Pair devices by LUID or PCI bus ID, never by name or index; load vendor DLLs by absolute system path |
| Add a column or JSON field | `report.py` and `app.py` | Follow the null rules above; update the README tables. If it is a measured byte count, consider adding it to `diff.py` `MEASURED` |
-| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero. Refuse only when both snapshots name conflicting adapters (vendor, PCI address, a non-Windows key); when they cannot be compared, warn (`name-only`) instead, because a Windows LUID changes at boot and `diff` has no override flag |
+| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero. Refuse only when both snapshots name conflicting adapters (vendor, PCI address, a non-Windows key); when they cannot be compared, warn (`name-only`) instead, because a Windows LUID changes at boot and `diff` has no override flag. A PID whose two `started_at` values are both valid and more than `SAME_PROCESS_SECS` apart is a different process: no pairing rule may join it (unknown or malformed start times never conflict, so older snapshots pair as before) |
| Change system RAM/CPU | `host.py` | Machine memory and load as the OS reports them: never GPU bytes, never added to `Snapshot.attributed`, and `diff` ignores it. Unknown is `None`/`null` and a failure carries a reason, never zeros. The JSON key is `system` because `host` already means the WSL Windows host. Windows uses the NT process list (psutil per process takes about 6 s per pass for 1,000 processes), checked against psutil at startup; CPU rates are keyed by `(pid, create_time)` so a reused PID never inherits another process's history (on the psutil path, `is_running()` is what makes psutil drop its cached object for a reused PID; keep that call) |
| Change WSL matching | `wslhost.py` `ContextTracker` | Keep `tests/test_wslhost.py` replays passing; they are measured sequences |
diff --git a/README.md b/README.md
index 994888f..be560ba 100644
--- a/README.md
+++ b/README.md
@@ -596,6 +596,15 @@ model reload often restarts the server under a new PID. A process missing from o
`not listed`, not zero: it was not running or used too little GPU to be listed. Processes that
moved by less than 16 MB are hidden from the text output but included in `--json`.
+A PID is not an identity, because both Windows and Linux reuse them. Each process in `--json` carries
+`started_at` (its creation time in Unix seconds, taken from data the monitor already reads). The same
+PID with start times more than 2 seconds apart is a different process, so no rule pairs it: the old
+one shows as `gone`, the new one as `new`, and the device totals still compare normally. A missing or
+`null` `started_at` (a protected process, a WSL host row, or a snapshot from an older version) never
+counts as a conflict, so those rows are paired as described above. Linux derives start times from the
+boot time in whole seconds, hence the 2-second allowance; a larger clock step can make one live
+process look reused, which shows as `gone` plus `new` and never as a wrong change.
+
Two cards of one model share a name, so `diff` also compares the adapter each snapshot recorded.
It refuses a pair whose vendor ID, PCI address or (off Windows) adapter key conflicts. A Windows
adapter key changes at boot and when an adapter is re-enabled, so there it only produces a warning.
diff --git a/src/gputaskman/diff.py b/src/gputaskman/diff.py
index 479fb05..59cbdb3 100644
--- a/src/gputaskman/diff.py
+++ b/src/gputaskman/diff.py
@@ -8,11 +8,13 @@ the process has the file, not that it is resident in GPU memory.
from __future__ import annotations
import json
+import math
from .llm import gb
MiB = 1 << 20
NOISE = 16 * MiB # smaller per-process moves are allocator and compositor churn, not a model loading
+SAME_PROCESS_SECS = 2.0 # one process's start time can move this much between runs (Linux boot-time rounding, clock steps)
MEASURED = ("vram_bytes", "shared_bytes", "committed_bytes", "paged_out_bytes",
"vram_resident_bytes", "gtt_resident_bytes") # resident: amdgpu, shared buffers included (never summed)
DEVICE = (("vram_used", "dedicated used"), ("shared_used", "shared used"),
@@ -52,10 +54,26 @@ def _named(p: dict) -> bool:
return p.get("host") != "outside" and p.get("name") not in PLACEHOLDER_NAMES
+def _started(p: dict) -> float | None:
+ v = p.get("started_at")
+ return v if isinstance(v, (int, float)) and not isinstance(v, bool) and math.isfinite(v) and v > 0 else None
+
+
+def _recycled(b: dict, a: dict) -> bool:
+ """One PID with two different start times is two processes: the PID was reused in between.
+
+ Unknown or malformed start times (older snapshots, protected processes) never conflict. Rows are only
+ ever paired within one host, so the PID needs no namespace check here."""
+ sb, sa = _started(b), _started(a)
+ return (b.get("pid") is not None and b.get("pid") == a.get("pid") and sb is not None and sa is not None
+ and abs(sb - sa) > SAME_PROCESS_SECS)
+
+
def match(before: list[dict], after: list[dict]) -> list[tuple[dict | None, dict | None, str]]:
"""Pair rows by PID, then by name and ports, then by name, when each pairing is unambiguous.
A model reload often restarts the server under a new PID. WSL host rows have their own PID space.
+ A PID that comes back with a different start time is another process and is never paired, by any rule.
Every row comes back exactly once: paired, new, or gone.
"""
keys = [(p.get("host"), p.get("pid")) for p in before]
@@ -66,7 +84,7 @@ def match(before: list[dict], after: list[dict]) -> list[tuple[dict | None, dict
rest: list[dict] = []
for a in after:
i = by_pid.get((a.get("host"), a.get("pid")))
- if i is not None and i in left and left[i].get("name") == a.get("name"):
+ if i is not None and i in left and left[i].get("name") == a.get("name") and not _recycled(left[i], a):
pairs.append((left.pop(i), a, "pid"))
else:
rest.append(a)
@@ -86,8 +104,10 @@ def match(before: list[dict], after: list[dict]) -> list[tuple[dict | None, dict
news.setdefault(key(a), []).append(a)
for k, found in news.items():
if k is not None and len(found) == 1 and len(olds.get(k, [])) == 1:
- pairs.append((left.pop(olds[k][0]), found[0], how))
- rest.remove(found[0])
+ i = olds[k][0]
+ if not _recycled(left[i], found[0]):
+ pairs.append((left.pop(i), found[0], how))
+ rest.remove(found[0])
pairs += [(None, a, "new") for a in rest]
pairs += [(b, None, "gone") for b in left.values()]
return pairs
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index f2d2858..ac6c8c3 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -116,6 +116,7 @@ def to_dict(s: Snapshot) -> dict:
engines = has_engine_counters(s, p)
item = {
"pid": p.pid, "host": p.origin if p.host else None, "name": p.name, "kind": p.kind, "runtimes": p.detail.runtimes,
+ "started_at": round(p.detail.create_time, 3) or None, # unknown (protected process, WSL host row) is null
"vram_attributed": p.gpu.attributed, "vram_inferred_bytes": p.gpu.inferred,
"vram_inferred_by": p.gpu.inferred_by or None,
"vram_bytes": int(p.gpu.dedicated) if p.gpu.attributed and "dedicated" in p.gpu.available else None,
diff --git a/tests/test_core.py b/tests/test_core.py
index 1e8f2b6..f92060b 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -360,6 +360,15 @@ class MemoryTests(unittest.TestCase):
s.procs[0].gpu.available.clear()
self.assertIsNone(to_dict(s)["attributed"])
+ def test_started_at_is_the_creation_time_and_unknown_is_null(self):
+ s = snapshot()
+ s.procs[0].detail.create_time = 1762680123.4567891
+ s.procs.append(ProcRow(30, "browser.exe", "browser", ProcGpu(dedicated=GiB), ProcDetail(30, "browser.exe"),
+ host=True, origin="windows"))
+ rows = to_dict(s)["processes"]
+ self.assertEqual(rows[0]["started_at"], 1762680123.457)
+ self.assertEqual([r["started_at"] for r in rows[1:]], [None, None]) # no creation time (0.0) is unknown, not 1970
+
def test_missing_adapter_memory_is_not_reported_as_zero(self):
s = snapshot()
s.vram_used = None
diff --git a/tests/test_diff.py b/tests/test_diff.py
index 72a2f9a..286ec91 100644
--- a/tests/test_diff.py
+++ b/tests/test_diff.py
@@ -15,13 +15,18 @@ GiB = 1 << 30
MiB = 1 << 20
-def proc(pid, name, vram=None, shared=0, host=None, ports=(), llm=None, files=(), inferred=None):
+def proc(pid, name, vram=None, shared=0, host=None, ports=(), llm=None, files=(), inferred=None, started=...):
return {"pid": pid, "host": host, "name": name, "vram_bytes": vram, "shared_bytes": shared,
"committed_bytes": None, "paged_out_bytes": None, "vram_inferred_bytes": inferred,
"ports": list(ports), "model_files": [{"path": f, "sources": ["mapped"]} for f in files],
+ **({} if started is ... else {"started_at": started}), # absent: a snapshot from before started_at existed
**({"llm": llm} if llm else {})}
+def pairs(before, after):
+ return {(b and b["pid"], a and a["pid"], how) for b, a, how in match(before, after)}
+
+
def snap(procs, used, attributed, taken=100.0, gpu="RTX", backend="wddm", adapter=...):
return {"taken": taken, "gpu": gpu, "backend": backend, "vram_used": used, "shared_used": 0,
"attributed": attributed, "processes": procs, **({} if adapter is ... else {"adapter": adapter})}
@@ -65,6 +70,51 @@ class MatchTests(unittest.TestCase):
self.assertIn("W:41", to_text(d))
+class PidReuseTests(unittest.TestCase):
+ """A PID that comes back with a different start time is another process, however alike the rows look."""
+
+ def test_a_reused_pid_is_gone_and_new_with_or_without_ports(self):
+ for ports in ([8080], []): # name+ports and unique-name fallbacks must not pair them either
+ with self.subTest(ports=ports):
+ before = [proc(42, "llama-server", ports=ports, started=100.0)]
+ after = [proc(42, "llama-server", ports=ports, started=200.0)]
+ self.assertEqual(pairs(before, after), {(None, 42, "new"), (42, None, "gone")})
+
+ def test_one_lifetime_still_pairs_by_pid_within_the_rounding_tolerance(self):
+ for later in (100.0, 101.9, 102.0, 98.0):
+ with self.subTest(later=later):
+ self.assertEqual(pairs([proc(42, "a.exe", started=100.0)], [proc(42, "a.exe", started=later)]),
+ {(42, 42, "pid")})
+ self.assertEqual(pairs([proc(42, "a.exe", started=100.0)], [proc(42, "a.exe", started=102.5)]),
+ {(None, 42, "new"), (42, None, "gone")})
+
+ def test_unknown_or_malformed_start_times_keep_the_old_heuristic_pairing(self):
+ # ... means the key is absent (an older snapshot); the rest are values a snapshot could carry
+ for before_value, after_value in ((..., 200.0), (100.0, ...), (None, None), (None, 200.0), (0.0, 200.0),
+ (-5.0, 200.0), (True, 200.0), ("100", 200.0), ([100.0], 200.0),
+ (float("nan"), 200.0), (float("inf"), 200.0)):
+ with self.subTest(before=before_value, after=after_value):
+ before = [proc(42, "a.exe", started=before_value)]
+ after = [proc(42, "a.exe", started=after_value)]
+ self.assertEqual(pairs(before, after), {(42, 42, "pid")})
+ # no PID means no reuse to speak of: rows without one are still paired by name alone
+ self.assertEqual({how for _, _, how in match([proc(None, "a", started=100.0)], [proc(None, "a", started=200.0)])},
+ {"name"})
+ before = [proc(42, "llama-server", ports=[8080], started=100.0), proc(50, "solo.exe", started=100.0)]
+ after = [proc(77, "llama-server", ports=[8080], started=200.0), proc(60, "solo.exe", started=200.0)]
+ self.assertEqual(pairs(before, after), {(42, 77, "name+ports"), (50, 60, "name")})
+
+ def test_a_refused_pair_does_not_make_the_other_rows_less_ambiguous(self):
+ before = [proc(42, "llama-server", started=100.0)]
+ after = [proc(42, "llama-server", started=200.0), proc(77, "llama-server", started=300.0)]
+ self.assertEqual(pairs(before, after), {(None, 42, "new"), (None, 77, "new"), (42, None, "gone")})
+
+ def test_reuse_is_judged_inside_one_pid_namespace(self):
+ before = [proc(42, "x", started=100.0), proc(42, "x", host="windows")]
+ after = [proc(42, "x", started=200.0), proc(42, "x", host="windows")]
+ self.assertEqual(pairs(before, after), {(None, 42, "new"), (42, None, "gone"), (42, 42, "pid")})
+
+
class DiffTests(unittest.TestCase):
def test_measured_runtime_and_mapped_evidence_stay_separate(self):
before = snap([proc(1, "dwm.exe", vram=2 * GiB),
@@ -130,6 +180,17 @@ class DiffTests(unittest.TestCase):
self.assertIn("(no process changed by 16 MB or more)", to_text(d))
self.assertNotIn("llama (1)", to_text(d))
+ def test_a_reused_pid_keeps_the_device_change_but_makes_no_process_delta(self):
+ before = snap([proc(42, "llama-server", vram=4 * GiB, ports=[8080], started=100.0)], 5 * GiB, 4 * GiB)
+ after = snap([proc(42, "llama-server", vram=8 * GiB, ports=[8080], started=200.0)], 9 * GiB, 8 * GiB)
+ d = diff(before, after)
+ self.assertEqual(d["device"]["vram_used"]["delta"], 4 * GiB) # the device-wide change is still measured
+ self.assertEqual(sorted(p["match"] for p in d["processes"]), ["gone", "new"])
+ self.assertTrue(all(p["measured"]["vram_bytes"]["delta"] is None for p in d["processes"]))
+ rows = [line for line in to_text(d).splitlines() if line.startswith("llama-server")]
+ self.assertEqual(len(rows), 2) # one row per lifetime, each absent from the other snapshot
+ self.assertTrue(all("not listed" in row for row in rows))
+
class AdapterGuardTests(unittest.TestCase):
def pair(self, before, after, backend="wddm"):