Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Harden the system RAM and CPU panel: OS-reported, never GPU bytes

Scott Hanselman76ecd74633bd
diff --git a/AGENTS.md b/AGENTS.md
index 0dc492b..7192eb6 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -58,6 +58,7 @@ cli.py ──► Collector.sample() (collector.py)
├─ LLM detail llm.py: buffer logs, ServerProber (local APIs), build_view()
└─ app adapters handy.py (settings + bounded log tail)
──► Snapshot ──► app.py (Textual TUI) / report.py (text and JSON)
+cli.py ──► HostSampler.sample() (host.py: system RAM/CPU, OS-reported; sampled beside the Collector, not in Snapshot)
cli.py diff ──► diff.py: compares two saved --json snapshots (no collection)
```
@@ -80,6 +81,7 @@ identify everything immediately; the TUI spreads scans across refreshes.
| Add a GPU vendor or backend | `dxgi.py`, `collector.py`, a new module | Pair devices by LUID or PCI bus ID, never by name or index; load vendor DLLs by absolute system path |
| Add a column or JSON field | `report.py` and `app.py` | Follow the null rules above; update the README tables. If it is a measured byte count, consider adding it to `diff.py` `MEASURED` |
| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero |
+| Change system RAM/CPU | `host.py` | Machine memory and load as the OS reports them: never GPU bytes, never added to `Snapshot.attributed`, and `diff` ignores it. Unknown is `None`/`null` and a failure carries a reason, never zeros. The JSON key is `system` because `host` already means the WSL Windows host. Windows uses the NT process list (psutil per process takes about 6 s per pass for 1,000 processes), checked against psutil at startup; CPU rates are keyed by `(pid, create_time)` so a reused PID never inherits another process's history |
| Change WSL matching | `wslhost.py` `ContextTracker` | Keep `tests/test_wslhost.py` replays passing; they are measured sequences |
## Verify
diff --git a/README.md b/README.md
index d0b3b44..6ff5caf 100644
--- a/README.md
+++ b/README.md
@@ -388,6 +388,8 @@ for a privacy consent prompt.
video-engine clock, and responsive history charts with an all-charts or single-metric view.
- **Process view:** local GPU memory, GPU-accessible system RAM, estimated non-resident commit,
engine activity, and detected runtimes such as CUDA, HIP, Vulkan, Direct3D, ggml and ONNX Runtime.
+- **System RAM and CPU:** a panel plus RAM and CPU% columns, as the OS reports them: machine memory
+ and load, never GPU memory (see [System RAM and CPU](#system-ram-and-cpu)).
- **Model inspector:** which model each process holds (LLM, decision, embedding, speech, or diffusion), how
it was identified, architecture, quantization, layer count, weights, estimated KV cost, and
buffer sizes from compatible load logs.
@@ -423,13 +425,14 @@ uv run --locked gputaskman --once # one text snapshot
uv run --locked gputaskman --json # one JSON snapshot; implies --once
uv run --locked gputaskman diff before.json after.json # compare two --json snapshots
uv run --locked gputaskman --no-host # WSL: don't read Windows host counters through interop
+uv run --locked gputaskman --no-system # don't sample system RAM/CPU: no panel, columns, or JSON section
uv run --locked gputaskman --help
```
| Key | Action |
|---|---|
| Up / Down | Select a process and inspect it |
-| `s` | Cycle sort: VRAM, GPU%, spill, paged, name |
+| `s` | Cycle sort: VRAM, GPU%, spill, paged, name (then RAM, CPU% unless `--no-system`) |
| `c` | Cycle all charts or a single metric (also available in the **Charts** dropdown) |
| `a` | Switch between top and all collected processes |
| `m` | Toggle full diagnostics / slim monitoring |
@@ -534,6 +537,41 @@ different adapter/render GUID from DXGI's `0x4BB7`; it was not a proven alias fo
The portable build's `--once` explanation and `--json` unmatched status were verified in the guest.
Other Hyper-V GPU-PV configurations have not been validated.
+### System RAM and CPU
+
+A **System** panel under the GPU header, and **RAM** and **CPU%** columns in the process table, show what
+each process costs the *machine*, as the OS reports it. The numbers come from psutil; on Windows the process
+list comes from `NtQuerySystemInformation`, because psutil's per-process calls take about 6 s per pass for
+~1,000 processes there. `s` sorts by them. Full diagnostics also shows a per-thread strip, 60-sample CPU and
+RAM histories, and the top processes by RAM and CPU; slim monitoring keeps two lines (the RAM and CPU bars).
+`--no-system` removes the panel, the columns and the sampling.
+
+These are system-memory figures, never GPU bytes. They are not added to any GPU total or to the attributed VRAM:
+
+- **RAM** is the process's working set (Linux: resident set), which includes pages shared with other processes,
+ so it will not match Task Manager's default *Memory (active private working set)* column. `PRIVATE` in
+ `--once` (`private` in `--json`) is the Windows commit charged to the process; it is `n/a` (`null`) elsewhere.
+- **CPU%** is a share of the whole machine, like Task Manager: one busy thread on a 32-thread machine is about
+ 3%. It is a rate between two samples, so a new process (or one whose PID was reused) shows `…` for one
+ refresh. Below 0.5% it is blank.
+- **`n/a`** means the OS would not say (a protected process), or the row is a Windows process listed from WSL,
+ whose PID is not a Linux PID.
+- Inside WSL the panel describes the Linux VM, and its border says so.
+- When the numbers cannot be read, the panel, `--once` and `--json` say why, and the GPU view is unaffected.
+ That covers 32-bit Python on Windows and a Windows build whose process-list layout does not match: the layout
+ is checked against psutil for gputaskman's own process at startup instead of being trusted.
+
+`--json` adds a `system` section: `cpu`, `ram` (with swap or pagefile), the top 10 processes machine-wide by
+`rss` and by `cpu_pct`, `denied` (processes the OS would not describe) and `cost_ms`. Unknown values are `null`;
+if sampling failed the section is `{"error": "<reason>"}`. `--once` and `--json` take their CPU readings inside
+the second they already wait for GPU engine rates. `gputaskman diff` compares GPU memory only and ignores this
+section.
+
+Live checks: Windows 11 on the RTX 4080 SUPER machine with 995 processes (the process list matched psutil for the
+same process; the TUI and `--json` ran), and Linux under WSL2 on the same machine (totals matched `/proc/meminfo`,
+gputaskman's own resident size matched `VmRSS` within 1%, and a single busy thread read 3.12% of the machine
+against the expected 100/32 = 3.1%). **Not validated on hardware:** macOS, Windows on ARM64, and bare-metal Linux.
+
### Comparing snapshots
To see which memory bucket actually grew when you load a model or change the context length, save
@@ -549,7 +587,8 @@ uv run --locked gputaskman diff before.json after.json # add --json for
The device and process tables compare **measured** counters only. Runtime-reported segments
(llama.cpp buffer logs) and estimates (KV cache from GGUF metadata and `-c`) are listed separately
under each process and are never added to measured totals. A model file that appears in a process
-is listed as evidence that the process opened or named it, not as GPU growth.
+is listed as evidence that the process opened or named it, not as GPU growth. The `system` section
+(machine RAM and CPU) is not compared.
Processes are paired by PID, then by name and listening ports, then by a unique name, because a
model reload often restarts the server under a new PID. A process missing from one snapshot shows
@@ -709,7 +748,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:
| macOS: `ioreg` device stats + Metal clients | ~10 ms | Every refresh |
| macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above |
| macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) |
-| System RAM/CPU panel (fork): Windows `NtQuerySystemInformation` process list + psutil totals | ~6 ms for ~480 processes (Windows 11, i9-13900K) | Every refresh; `--no-system` turns it off. psutil per-process `memory_info`/`cpu_times` took ~1 s per pass here, so it is only the non-Windows fallback |
+| System RAM/CPU: process list + psutil totals | Windows (`NtQuerySystemInformation`): ~28 ms median for 995 processes (Windows 11, RTX 4080 SUPER machine, 32 threads); the contributor measured ~6 ms for ~480 processes (Windows 11, i9-13900K). Linux under WSL2 (psutil): ~3.4 ms median for 44 processes, ~9.5 ms for the first sample of 123 | Every refresh; `--no-system` turns it off. psutil's per-process `memory_info`/`cpu_times` took ~6 s per pass for 995 processes on Windows, so it is only the non-Windows path. macOS unmeasured |
With two llama-servers running, a steady refresh costs roughly 15-30 ms of CPU (about 340 ms before
process details were cached).
@@ -861,6 +900,7 @@ downloads are bundled in this repository.
| `stores.py` | Friendly names from Ollama, LM Studio, and Hugging Face model stores |
| `gguf.py` | Model metadata, kind, and size estimates |
| `llm.py` | Buffer logs, local server probing, and breakdowns |
+| `host.py` | System RAM and CPU as the OS reports them: psutil totals, per-process RAM and CPU%, and the Windows process-list reader |
| `collector.py` | Shared snapshot collection |
| `app.py` / `report.py` / `cli.py` | TUI, text/JSON, and command-line entry points |
| `diff.py` | Before/after comparison of two `--json` snapshots |
diff --git a/src/gputaskman/app.py b/src/gputaskman/app.py
index 7b32d7e..412c933 100644
--- a/src/gputaskman/app.py
+++ b/src/gputaskman/app.py
@@ -17,8 +17,8 @@ from textual.screen import ModalScreen
from textual.widgets import DataTable, Footer, Markdown, Select, Static
from textual.worker import get_current_worker
-from .host import HostSample, HostSampler
from .collector import BACKEND, Collector, ProcRow, Snapshot, enumerate_adapters, ordered_adapters
+from .host import HostSample, HostSampler
from .llm import gb, redact_cmdline
from .nvml import NvDevice, format_metric
from .recovery import LiveCollectors
@@ -128,6 +128,13 @@ not necessarily current residency. A game's textures cannot be enumerated here.
on the **3D** engine. AMD device GPU charts show graphics-engine activity, which is not
necessarily equivalent to NVML GPU utilization. Vulkan does not expose another process's HIP internals.
+**System panel, RAM and CPU% columns** are reported by the OS (`NtQuerySystemInformation` on Windows,
+psutil elsewhere), not by the GPU. RAM is a process's working set, shared pages included, so rows can add up
+to more than the machine uses; CPU% is its share of the whole machine, like Task Manager. This is system memory:
+it is never GPU bytes and never added to a GPU total. `n/a` means the OS wouldn't say (or it is a Windows-side
+row seen from WSL); `…` means one more sample is needed. Inside WSL they describe the Linux VM.
+`--no-system` hides them.
+
Keys: `↑/↓` select · `s` sort · `c` chart · `a` show all · `p` pause · `?` this help · `q` quit
All eight trend charts are visible by default. They stack in narrow terminals and
@@ -204,9 +211,9 @@ class GpuTaskManager(App):
Binding("question_mark", "help", "Explain"),
]
SORTS = [("VRAM", lambda r: -(r.gpu.shown_vram or 0)), ("GPU%", lambda r: -r.gpu.gpu_pct),
- ("RAM", lambda r: -(r.ram or 0)), ("CPU%", lambda r: -(r.cpu or 0)),
("spill", lambda r: -r.gpu.shared), ("paged", lambda r: -r.gpu.paged_out),
("name", lambda r: r.name.lower())]
+ SYSTEM_SORTS = [("RAM", lambda r: -(r.ram or 0)), ("CPU%", lambda r: -(r.cpu or 0))]
CHART_COLORS = {
"GPU": "#4ade80",
"SM clock": "#fb923c",
@@ -223,6 +230,7 @@ class GpuTaskManager(App):
system: bool = True) -> None:
super().__init__()
self.system = system
+ self.sorts = self.SORTS + self.SYSTEM_SORTS if system else self.SORTS
self.host_sample: HostSample | None = None
self.host_error: str | None = None
self._sampler: HostSampler | None = None
@@ -264,9 +272,9 @@ class GpuTaskManager(App):
def compose(self) -> ComposeResult:
with VerticalScroll(id="dashboard"):
+ yield Static("Sampling GPU counters...", id="device")
if self.system:
yield Static("Sampling RAM and CPU...", id="system")
- yield Static("Sampling GPU counters...", id="device")
with Vertical(id="history"):
with Horizontal(id="chart-controls"):
yield Static("Charts", id="chart-label")
@@ -352,13 +360,12 @@ class GpuTaskManager(App):
def apply_all(self, snapshots: list[Snapshot], host: HostSample | None = None) -> None:
if not self.is_running:
return # A worker may finish sampling after shutdown has removed the widgets.
- if host is not None:
- self.host_sample = host
- self.hist_cpu.append(host.cpu_pct)
- self.hist_ram.append(100 * host.ram_used / host.ram_total if host.ram_total else None)
+ self.host_sample = host # None after a failed sample: a stale reading is not shown as current
+ self.hist_cpu.append(host.cpu_pct if host else None)
+ self.hist_ram.append(100 * host.ram_used / host.ram_total if host and host.ram_total else None)
for snap in snapshots:
for row in snap.procs: # Windows (W:) / outside (H:) rows have no local pid: RAM/CPU stay unknown
- hp = self.host_sample.proc(row.pid) if self.host_sample and not row.host else None
+ hp = host.proc(row.pid) if host and not row.host else None
row.ram, row.cpu = (hp.rss, hp.cpu_pct) if hp else (None, None)
multi = len(snapshots) > 1
if self.snapshots and multi != (len(self.snapshots) > 1) and self.selected_key is not None:
@@ -434,8 +441,13 @@ class GpuTaskManager(App):
if len(self.snapshots) > 1:
devices.append(Text(f"GPU {index}", style="bold"))
devices.append(self.render_device(snap))
- if self.system and self.host_sample:
- self.query_one("#system", Static).update(self.render_system(self.host_sample))
+ if self.system:
+ system = self.query_one("#system", Static)
+ system.border_title = "System (WSL VM, OS-reported)" if BACKEND == "wsl" else "System (OS-reported)"
+ if self.host_sample:
+ system.update(self.render_system(self.host_sample))
+ elif self.host_error:
+ system.update(Text(f"RAM/CPU unavailable: {self.host_error}", style="yellow"))
device = self.query_one("#device", Static)
device.border_title = "Slim monitoring" if self.monitoring else "Full diagnostics"
device.update(
@@ -443,39 +455,60 @@ class GpuTaskManager(App):
self.render_charts()
def render_system(self, h: HostSample) -> Group:
- """System RAM + CPU (fork addition). All values are OS-reported via psutil / NtQuerySystemInformation."""
- width = max(min(self.size.width - 50, 80), 16)
+ """System RAM and CPU as the OS reports them: system memory, never GPU bytes and never in a GPU total."""
+ min_bar = 8
+ room = self.size.width - 12 # what a line has left once the label, panel border, padding and scrollbar are counted
+
+ def line(text: str = "") -> Text:
+ return Text(text, no_wrap=True, overflow="ellipsis") # never wraps onto a second row
+
ram_pct = 100 * h.ram_used / h.ram_total if h.ram_total else 0
- ram = Text("RAM ")
- ram.append_text(stacked_bar([(h.ram_used, "dodger_blue2")], h.ram_total, width))
- ram.append(f" {gb(h.ram_used)} / {gb(h.ram_total)}", style="bold")
- ram.append(f" {ram_pct:.0f}% free {gb(h.ram_available)}", style="grey62")
+ ram_tail = line()
+ ram_tail.append(f" {gb(h.ram_used)} / {gb(h.ram_total)}", style="bold")
+ ram_tail.append(f" {ram_pct:.0f}%", style="grey62")
+ optional = [(f" free {gb(h.ram_available)}", "grey62")]
if h.swap_total:
- ram.append(f" pagefile {gb(h.swap_used)}/{gb(h.swap_total)}",
- style="yellow" if h.swap_used > h.swap_total / 4 else "grey62")
- cpu = Text("CPU ")
+ optional.append((f" {'pagefile' if os.name == 'nt' else 'swap'} {gb(h.swap_used)}/{gb(h.swap_total)}",
+ "yellow" if h.swap_used > h.swap_total / 4 else "grey62"))
+ for extra, style in optional: # detail is dropped, not wrapped, in a narrow window
+ if ram_tail.cell_len + len(extra) + min_bar <= room:
+ ram_tail.append(extra, style=style)
+ cpu_tail = line()
+ cpu_tail.append(f" {h.cpu_pct:4.0f}%", style="bold")
+ threads = f" {h.logical} threads" + (f"/{h.physical} cores" if h.physical else "")
+ if not self.monitoring and cpu_tail.cell_len + len(threads) + min_bar <= room:
+ cpu_tail.append(threads, style="grey62")
+ # Both bars share one width, sized so the longer line still fits on a single row.
+ width = max(min_bar, min(60, room - max(ram_tail.cell_len, cpu_tail.cell_len)))
+ ram = line("RAM ")
+ ram.append_text(stacked_bar([(h.ram_used, "dodger_blue2")], h.ram_total, width))
+ ram.append_text(ram_tail)
+ cpu = line("CPU ")
cpu.append_text(stacked_bar([(h.cpu_pct, "green3")], 100, width))
- cpu.append(f" {h.cpu_pct:4.0f}%", style="bold")
- cpu.append(f" {h.logical} threads" + (f"/{h.physical} cores" if h.physical else "")
- + (f" {h.cpu_mhz:.0f} MHz" if h.cpu_mhz else ""), style="grey62")
- cores = Text(" ")
+ cpu.append_text(cpu_tail)
+ if self.monitoring:
+ return Group(ram, cpu)
+ cores = line(" ")
for v in h.per_core:
cores.append(SPARK[min(int(v / 100 * (len(SPARK) - 1) + 0.5), len(SPARK) - 1)],
style="red" if v >= 90 else "yellow" if v >= 60 else "green3")
cores.append(" per-thread", style="grey50")
- hist = Text(" ")
- hist.append(spark(self.hist_cpu, 30), style="green3")
+ history = max(8, min(30, (room - 18) // 2)) # each sparkline's width: two of them plus their labels share a row
+ hist = line(" ")
+ hist.append(spark(self.hist_cpu, history), style="green3")
hist.append(" CPU 60s ", style="grey50")
- hist.append(spark(self.hist_ram, 30), style="dodger_blue2")
+ hist.append(spark(self.hist_ram, history), style="dodger_blue2")
hist.append(" RAM 60s", style="grey50")
- lines = [ram, cpu, cores, hist]
- if not self.monitoring:
- top = Text(" top RAM: ", style="grey62")
- top.append(" ".join(f"{p.name} {gb(p.rss)}" for p in h.top("rss", 5)))
- top.append("\n top CPU: ", style="grey62")
- top.append(" ".join(f"{p.name} {p.cpu_pct:.0f}%" for p in h.top("cpu_pct", 5) if p.cpu_pct >= 0.5) or "idle")
- lines.append(top)
- return Group(*lines)
+ top_ram = line()
+ top_ram.append(" top RAM: ", style="grey62")
+ top_ram.append(" ".join(f"{p.name} {gb(p.rss)}" for p in h.top("rss", 5)))
+ top_cpu = line()
+ top_cpu.append(" top CPU: ", style="grey62")
+ rated = h.top("cpu_pct", 5)
+ busy = [p for p in rated if p.cpu_pct >= 0.5]
+ # "…" until some process has a CPU reading: no reading yet is not the same as idle
+ top_cpu.append(" ".join(f"{p.name} {p.cpu_pct:.0f}%" for p in busy) if busy else "idle" if rated else "…")
+ return Group(ram, cpu, cores, hist, top_ram, top_cpu)
def render_device(self, s: Snapshot) -> Group:
nv = s.nv
@@ -653,9 +686,9 @@ class GpuTaskManager(App):
if not self.show_all:
rows = [r for r in rows if not r.gpu.attributed or r.gpu.dedicated >= 32 << 20 or r.gpu.gpu_pct >= 1 or r.gpu.shared >= 64 << 20
or r.gpu.paged_out >= 128 << 20 or r.llm]
- rows = sorted(rows, key=self.SORTS[self.sort_idx][1])
+ rows = sorted(rows, key=self.sorts[self.sort_idx][1])
vm = next((i for i, r in enumerate(rows) if r.kind == "wsl-vm"), None)
- if vm is not None and self.SORTS[self.sort_idx][0] == "VRAM":
+ if vm is not None and self.sorts[self.sort_idx][0] == "VRAM":
# Linux rows share the WSL VM row's total: keep them under it, ordered by their inferred sizes.
linux = [r for r in rows if not r.gpu.attributed and not r.host]
rest = [r for r in rows if r not in linux]
@@ -670,7 +703,7 @@ class GpuTaskManager(App):
rows = [r for r in rows if not (
self._row_snapshots[r.key].backend == "wsl" and not r.host and r.gpu_index != 0)]
if len(self.snapshots) > 1 and not any(snap.backend == "wsl" for snap in self.snapshots):
- rows.sort(key=self.SORTS[self.sort_idx][1])
+ rows.sort(key=self.sorts[self.sort_idx][1])
keep = self.selected_key
t.clear()
@@ -693,6 +726,11 @@ class GpuTaskManager(App):
spill = Text("n/a", justify="right", style="grey62")
if wddm and not r.gpu.paged_available:
paged = Text("n/a", justify="right", style="grey62")
+ system_cells = []
+ if self.system:
+ known = r.ram is not None # "…": readable, CPU% needs one more sample; "n/a": the OS wouldn't say
+ system_cells = [Text(gb(r.ram) if known else "n/a", justify="right", style="" if known else "grey62"),
+ Text("…", justify="right", style="grey62") if known and r.cpu is None else pct(r.cpu)]
t.add_row(
Text(r.pid_label, justify="right", style="grey62" if r.host else ""),
Text(r.name),
@@ -703,10 +741,7 @@ class GpuTaskManager(App):
pct(e.get("3d", 0) + e.get("compute", 0) if split else None),
pct(e.get("copy", 0) if split else None),
pct(e.get("decode", 0) + e.get("encode", 0) + e.get("video", 0) if split else None),
- *([Text(gb(r.ram) if r.ram is not None else "n/a", justify="right",
- style="" if r.ram is not None else "grey62"),
- pct(r.cpu) if r.cpu is not None or r.host else Text("…", justify="right", style="grey62")]
- if self.system else []),
+ *system_cells,
Text(r.inside, style="magenta" if r.llm else "grey62", overflow="ellipsis", no_wrap=True),
*([Text("?" if s.backend == "wsl" and not r.host else str(r.gpu_index))]
if len(self.snapshots) > 1 else []),
@@ -717,7 +752,7 @@ class GpuTaskManager(App):
t.move_cursor(row=t.get_row_index(keep), animate=False, scroll=False)
except Exception:
pass
- t.border_subtitle = f"sort: {self.SORTS[self.sort_idx][0]} · {'all' if self.show_all else 'top'} processes"
+ t.border_subtitle = f"sort: {self.sorts[self.sort_idx][0]} · {'all' if self.show_all else 'top'} processes"
if s.backend == "wsl" and not any(not r.host for r in s.procs):
t.border_subtitle = WSL_EMPTY
@@ -856,7 +891,7 @@ class GpuTaskManager(App):
self.render_inspector()
def action_cycle_sort(self) -> None:
- self.sort_idx = (self.sort_idx + 1) % len(self.SORTS)
+ self.sort_idx = (self.sort_idx + 1) % len(self.sorts)
if self.snap:
self.render_table(self.snap)
diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.py
index b7ab8d2..df2b606 100644
--- a/src/gputaskman/cli.py
+++ b/src/gputaskman/cli.py
@@ -38,7 +38,8 @@ def main() -> None:
ap.add_argument("--list-gpus", action="store_true", help="list selectable GPU adapters and exit")
ap.add_argument("--no-host", action="store_true",
help="WSL: don't read Windows host GPU counters through interop (WSL VM total, Windows apps)")
- ap.add_argument("--no-system", action="store_true", help="hide the system RAM/CPU panel and columns")
+ ap.add_argument("--no-system", action="store_true",
+ help="don't sample system RAM/CPU: no TUI panel or RAM/CPU%% columns, no system section in --once/--json")
ap.add_argument("-i", "--interval", type=float, default=1.0, help="refresh seconds (default 1)")
args = ap.parse_args()
if args.gpu is not None and args.gpu < 0:
@@ -78,11 +79,21 @@ def main() -> None:
c.host.wait(15.0) # WSL: the first Windows counter sample takes several seconds
if hasattr(c.counters, "warm_up"):
c.counters.warm_up(15.0) # macOS: per-process GPU memory comes from a background footprint pass
- hs = None if args.no_system else HostSampler()
- if hs:
- hs.sample() # CPU% is a rate too; this primes it
+ sampler, system, system_error = None, None, None
+ if not args.no_system:
+ try:
+ sampler = HostSampler()
+ sampler.sample() # per-process CPU% is a rate too; this primes it
+ except Exception as error: # system RAM/CPU is extra: its failure must not cost the GPU report
+ sampler, system_error = None, f"{type(error).__name__}: {error}"
time.sleep(1.0) # engine utilization is a rate; needs two samples
- print(dump(c.sample(), args.json, hs.sample() if hs else None))
+ snapshot = c.sample()
+ if sampler:
+ try:
+ system = sampler.sample()
+ except Exception as error:
+ system_error = f"{type(error).__name__}: {error}"
+ print(dump(snapshot, args.json, system, system_error))
finally:
c.close()
return
diff --git a/src/gputaskman/host.py b/src/gputaskman/host.py
index ffeaff8..c53a91c 100644
--- a/src/gputaskman/host.py
+++ b/src/gputaskman/host.py
@@ -1,7 +1,9 @@
"""System RAM and CPU, alongside the GPU view.
-Every number here is reported by the OS through psutil (measured, not inferred). Unknown stays None:
-a process we may not inspect has ``rss=None`` / ``cpu_pct=None``, never 0. Fork addition, not upstream.
+Every number here is reported by the OS (psutil, or NtQuerySystemInformation for processes on Windows):
+measured, not inferred. Unknown stays None: a process we may not inspect has ``rss=None`` /
+``cpu_pct=None``, never 0. These are system-memory figures: never GPU bytes, never part of
+``Snapshot.attributed``.
"""
from __future__ import annotations
@@ -20,14 +22,13 @@ class HostProc:
name: str
rss: int | None # resident / working set bytes
private: int | None # Windows private bytes (commit charged to the process); None elsewhere
- cpu_pct: float | None # share of the WHOLE machine (0-100), like Task Manager; None on the first sample
+ cpu_pct: float | None # share of the WHOLE machine (0-100), like Task Manager; None until a second reading
@dataclass
class HostSample:
cpu_pct: float # whole machine, 0-100
per_core: list[float]
- cpu_mhz: float | None
logical: int
physical: int | None
ram_total: int
@@ -50,72 +51,65 @@ class HostSample:
class HostSampler:
- """Keeps psutil.Process objects between samples: per-process CPU% is a rate and needs the previous reading."""
+ """Per-process CPU% is a rate: the previous reading is kept per (pid, creation time), because pids are reused."""
def __init__(self) -> None:
- self._procs: dict[int, psutil.Process] = {}
- self._rated: set[int] = set() # pids with a previous CPU reading
self._logical = psutil.cpu_count() or 1
self._physical = psutil.cpu_count(logical=False)
- self._last_cpu: dict[int, int] = {}
+ self._rated: set[tuple[int, float]] = set() # psutil path: processes with a previous CPU reading
+ self._last_cpu: dict[tuple[int, float], int] = {} # NT path: CPU time (100 ns) at the previous sample
self._last_wall = 0.0
+ if _nt_query is not None:
+ _check_nt_layout(_nt_query()) # raises rather than show numbers parsed from a layout we don't recognise
psutil.cpu_percent(percpu=True) # prime the system-wide rate
def sample(self) -> HostSample:
t0 = time.perf_counter()
per_core = psutil.cpu_percent(percpu=True)
vm, sw = psutil.virtual_memory(), psutil.swap_memory()
- try:
- freq = psutil.cpu_freq()
- mhz = freq.current if freq and freq.current else None
- except (OSError, NotImplementedError): # unavailable on some platforms/containers
- mhz = None
if _nt_query is not None:
- procs, denied = self._sample_nt(), 0
- return self._finish(t0, per_core, mhz, vm, sw, procs, denied)
+ return self._finish(t0, per_core, vm, sw, self._sample_nt(), 0)
procs, denied, seen = [], 0, set()
- for p in psutil.process_iter():
- pid = p.pid
- seen.add(pid)
- p = self._procs.setdefault(pid, p)
+ for p in psutil.process_iter(): # psutil keeps these objects between calls and replaces one whose pid was reused
try:
with p.oneshot():
name = p.name()
mi = p.memory_info()
cpu = p.cpu_percent(None)
+ ident = (p.pid, p.create_time())
except psutil.NoSuchProcess:
continue
except psutil.AccessDenied:
denied += 1
- procs.append(HostProc(pid, _safe_name(p), None, None, None))
+ procs.append(HostProc(p.pid, _safe_name(p), None, None, None))
continue
- first = pid not in self._rated
- self._rated.add(pid)
- procs.append(HostProc(pid, name, mi.rss, getattr(mi, "private", None),
- None if first else cpu / self._logical))
- for gone in self._procs.keys() - seen:
- del self._procs[gone]
- self._rated &= seen
- return self._finish(t0, per_core, mhz, vm, sw, procs, denied)
+ seen.add(ident)
+ procs.append(HostProc(p.pid, name, mi.rss, getattr(mi, "private", None),
+ cpu / self._logical if ident in self._rated else None))
+ self._rated = seen
+ return self._finish(t0, per_core, vm, sw, procs, denied)
def _sample_nt(self) -> list[HostProc]:
now = time.perf_counter()
rows = _nt_query()
wall = now - self._last_wall if self._last_wall else None
- procs = []
- for pid, (name, ws, private, cpu100ns) in rows.items():
- prev = self._last_cpu.get(pid)
+ procs, last = [], {}
+ for pid, (name, ws, private, cpu100ns, created) in rows.items():
+ if pid == 0: # System Idle Process: its "CPU" is idle time
+ continue
+ ident = (pid, created)
+ last[ident] = cpu100ns
+ prev = self._last_cpu.get(ident) # a reused pid has a new creation time: no previous reading
cpu = None
- if wall and prev is not None and pid != 0: # pid 0 = System Idle Process: its "CPU" is idle time
- cpu = max(0.0, (cpu100ns - prev) / 1e7 / wall / self._logical * 100)
+ if wall and prev is not None and cpu100ns >= prev:
+ cpu = (cpu100ns - prev) / 1e7 / wall / self._logical * 100
procs.append(HostProc(pid, name, ws, private, cpu))
- self._last_cpu = {pid: r[3] for pid, r in rows.items()}
- self._last_wall = now
- return [p for p in procs if p.pid != 0]
+ self._last_cpu, self._last_wall = last, now
+ return procs
- def _finish(self, t0, per_core, mhz, vm, sw, procs, denied) -> HostSample:
+ def _finish(self, t0, per_core, vm, sw, procs, denied) -> HostSample:
return HostSample(
- cpu_pct=sum(per_core) / len(per_core) if per_core else 0.0, per_core=per_core, cpu_mhz=mhz,
+ cpu_pct=sum(per_core) / len(per_core) if per_core else 0.0, per_core=per_core,
logical=self._logical, physical=self._physical,
ram_total=vm.total, ram_used=vm.total - vm.available, ram_available=vm.available,
swap_total=sw.total, swap_used=sw.used, procs=procs, denied=denied,
@@ -123,17 +117,27 @@ class HostSampler:
# Windows: one NtQuerySystemInformation(SystemProcessInformation) call returns every process's working set,
-# private bytes and CPU time. psutil opens each process instead: measured 2026-11-08 on a 483-process
-# Windows 11 box, memory_info ~1.1 s and cpu_times ~1.0 s per pass, versus a few ms here. Same numbers:
-# WorkingSetSize = psutil rss, PagefileUsage = psutil private. No process handle needed, so no AccessDenied.
-_nt_query = None
+# private bytes, creation time and CPU time. psutil opens each process instead: measured 2026-11-08 on a
+# 995-process Windows 11 box, ~6 s per pass versus ~28 ms here. Same numbers: WorkingSetSize = psutil rss,
+# PagefileUsage = psutil private. No process handle needed, so no AccessDenied. The structure is read by
+# byte offset (winternl.h, x64): HostSampler checks those offsets against our own process before trusting a row.
+_FILETIME_UNIX_EPOCH = 11644473600 # seconds from 1601-01-01 (FILETIME) to 1970-01-01
+
+
+def _nt_unavailable() -> dict:
+ # psutil's per-process path takes seconds per pass on Windows, which would stall every GPU refresh.
+ raise OSError("system RAM/CPU needs 64-bit Python on Windows")
+
+
+_nt_query = _nt_unavailable if sys.platform == "win32" else None
if sys.platform == "win32" and struct.calcsize("P") == 8:
import ctypes
_ntdll = ctypes.WinDLL("ntdll")
_ntdll.NtQuerySystemInformation.restype = ctypes.c_long
- def _nt_query_impl() -> dict[int, tuple[str, int, int, int]]:
+ def _nt_query_impl() -> dict[int, tuple[str, int, int, int, float]]:
+ """pid -> (image name, working set, private bytes, CPU time in 100 ns, creation time in unix seconds)."""
size = 1 << 20
while True:
buf = ctypes.create_string_buffer(size)
@@ -148,16 +152,17 @@ if sys.platform == "win32" and struct.calcsize("P") == 8:
raw, base, out, off = buf.raw, ctypes.addressof(buf), {}, 0
while True:
nxt = struct.unpack_from("<I", raw, off)[0]
- user, kernel = struct.unpack_from("<qq", raw, off + 40)
+ created, user, kernel = struct.unpack_from("<qqq", raw, off + 32)
name_len, name_ptr = struct.unpack_from("<H6xQ", raw, off + 56)
pid = struct.unpack_from("<Q", raw, off + 80)[0]
ws, = struct.unpack_from("<Q", raw, off + 144)
private, = struct.unpack_from("<Q", raw, off + 184)
- if name_ptr:
- name = raw[name_ptr - base:name_ptr - base + name_len].decode("utf-16-le", "replace")
+ start = name_ptr - base
+ if name_ptr and 0 <= start and start + name_len <= len(raw):
+ name = raw[start:start + name_len].decode("utf-16-le", "replace")
else:
name = "System Idle Process" if pid == 0 else "?"
- out[pid] = (name, ws, private, user + kernel)
+ out[pid] = (name, ws, private, user + kernel, created / 1e7 - _FILETIME_UNIX_EPOCH)
if not nxt:
return out
off += nxt
@@ -165,6 +170,18 @@ if sys.platform == "win32" and struct.calcsize("P") == 8:
_nt_query = _nt_query_impl
+def _check_nt_layout(rows: dict) -> None:
+ """Our own process, as the NT process list reports it, must match what psutil says about it."""
+ me = psutil.Process()
+ row = rows.get(me.pid)
+ if row is not None:
+ name, ws, _private, _cpu, created = row
+ rss = me.memory_info().rss
+ if name.lower() == me.name().lower() and abs(created - me.create_time()) < 2 and rss / 2 <= ws <= rss * 2:
+ return
+ raise OSError("NtQuerySystemInformation process layout not recognised on this Windows build")
+
+
def _safe_name(p: psutil.Process) -> str:
try:
return p.name()
@@ -172,15 +189,20 @@ def _safe_name(p: psutil.Process) -> str:
return "?"
+def _proc_dict(p: HostProc) -> dict:
+ return {"pid": p.pid, "name": p.name, "rss": p.rss, "private": p.private,
+ "cpu_pct": None if p.cpu_pct is None else round(p.cpu_pct, 1)}
+
+
def to_dict(h: HostSample) -> dict:
return {
- "source": "psutil (OS-reported)",
- "cpu": {"pct": round(h.cpu_pct, 1), "per_core": h.per_core, "mhz": h.cpu_mhz,
+ "source": "OS-reported (psutil; NtQuerySystemInformation for the process list on Windows)",
+ "cpu": {"pct": round(h.cpu_pct, 1), "per_core": h.per_core,
"logical": h.logical, "physical": h.physical},
"ram": {"total": h.ram_total, "used": h.ram_used, "available": h.ram_available,
"swap_total": h.swap_total, "swap_used": h.swap_used},
- "top_ram": [vars(p) for p in h.top("rss", 10)],
- "top_cpu": [vars(p) for p in h.top("cpu_pct", 10)],
+ "top_ram": [_proc_dict(p) for p in h.top("rss", 10)],
+ "top_cpu": [_proc_dict(p) for p in h.top("cpu_pct", 10)],
"denied": h.denied,
"cost_ms": round(h.cost_ms, 1),
}
@@ -189,11 +211,11 @@ def to_dict(h: HostSample) -> dict:
def to_text(h: HostSample) -> str:
from .llm import gb
- lines = [f"CPU {h.cpu_pct:.0f}% {h.logical} threads" + (f" / {h.physical} cores" if h.physical else "")
- + (f" {h.cpu_mhz:.0f} MHz" if h.cpu_mhz else ""),
+ lines = ["System RAM and CPU (OS-reported; not GPU memory)",
+ f"CPU {h.cpu_pct:.0f}% {h.logical} threads" + (f" / {h.physical} cores" if h.physical else ""),
f"RAM {gb(h.ram_used)} / {gb(h.ram_total)} available {gb(h.ram_available)}"
f" swap/pagefile {gb(h.swap_used)} / {gb(h.swap_total)}",
- "", f"{'PID':>7} {'PROCESS':<28} {'RAM':>9} {'PRIVATE':>9} {'CPU%':>5}"]
+ "", "Top processes by RAM, whole machine:", f"{'PID':>7} {'PROCESS':<28} {'RAM':>9} {'PRIVATE':>9} {'CPU%':>5}"]
for p in h.top("rss", 10):
lines.append(f"{p.pid:>7} {p.name[:28]:<28} {gb(p.rss):>9} {gb(p.private) if p.private is not None else 'n/a':>9} "
f"{f'{p.cpu_pct:.0f}' if p.cpu_pct is not None else 'n/a':>5}")
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index e1ef542..f2d2858 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -272,13 +272,18 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
return "\n".join(out)
-def dump(s: Snapshot, as_json: bool, host=None) -> str:
- """host: an optional host.HostSample (system RAM/CPU), added as a "host" section."""
+def dump(s: Snapshot, as_json: bool, system=None, system_error: str | None = None) -> str:
+ """system: an optional host.HostSample (OS-reported system RAM/CPU); system_error: why it is missing."""
from . import host as hostmod
if as_json:
d = to_dict(s)
- if host is not None:
- d["host"] = hostmod.to_dict(host)
+ if system is not None:
+ d["system"] = hostmod.to_dict(system)
+ elif system_error:
+ d["system"] = {"error": system_error}
return json.dumps(d, indent=2, default=str)
- return to_text(s) + ("\n\n" + hostmod.to_text(host) if host is not None else "")
+ text = to_text(s)
+ if system is not None:
+ return text + "\n\n" + hostmod.to_text(system)
+ return text + (f"\n\nSystem RAM/CPU unavailable: {system_error}" if system_error else "")
diff --git a/tests/test_core.py b/tests/test_core.py
index 95d7e46..384d241 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -28,6 +28,7 @@ from gputaskman.collector import Collector, ProcRow, Snapshot, select_adapter, s
from gputaskman.dxgi import Adapter
from gputaskman.gguf import GgufInfo, read_gguf
from gputaskman.handy import HandyEvidence, inspect_handy, _log_cache, _settings_cache
+from gputaskman.host import HostSample
from gputaskman.llm import (LlmView, ServerInfo, ServerProber, _get, api_key, build_view, is_gpu_device,
llama_server_models, lmstudio_models, parse_flags, parse_load_log, probe_server,
redact_cmdline)
@@ -1019,6 +1020,12 @@ class UiTests(unittest.IsolatedAsyncioTestCase):
backend = patch("gputaskman.app.BACKEND", "wddm")
backend.start()
self.addCleanup(backend.stop)
+ # The monitor must not query this machine's processes: a fixed sample keeps these tests hardware-independent.
+ sampler = patch("gputaskman.app.HostSampler")
+ sampler.start().return_value.sample.return_value = HostSample(
+ cpu_pct=10.0, per_core=[10.0], logical=1, physical=1, ram_total=16 * GiB, ram_used=4 * GiB,
+ ram_available=12 * GiB, swap_total=0, swap_used=0)
+ self.addCleanup(sampler.stop)
def chart_text(self, app):
widget = app.query_one("#charts", Static)
@@ -1533,7 +1540,7 @@ class UiTests(unittest.IsolatedAsyncioTestCase):
app.apply(snapshot())
await pilot.pause()
self.assertEqual(app.selected_pid, 20)
- for _ in app.SORTS:
+ for _ in app.sorts:
await pilot.press("s")
self.assertEqual(app.sort_idx, 0)
await pilot.press("a", "p")
diff --git a/tests/test_host.py b/tests/test_host.py
index a18053e..9bb5c37 100644
--- a/tests/test_host.py
+++ b/tests/test_host.py
@@ -1,30 +1,203 @@
+import contextlib
import os
+import struct
import sys
import unittest
+from unittest.mock import patch
import psutil
from gputaskman import host
+SECOND = 10_000_000 # one second of CPU time, in the 100 ns units NtQuerySystemInformation reports
-class HostTests(unittest.TestCase):
+
+def sample(**override) -> host.HostSample:
+ fields = dict(cpu_pct=5, per_core=[5], logical=1, physical=1, ram_total=100, ram_used=40,
+ ram_available=60, swap_total=0, swap_used=0, procs=[])
+ return host.HostSample(**{**fields, **override})
+
+
+class Clock:
+ """Stands in for the time module inside host.py, so rates come out exact."""
+
+ def __init__(self) -> None:
+ self.now = 1000.0
+
+ def perf_counter(self) -> float:
+ return self.now
+
+
+class SampleTests(unittest.TestCase):
def test_unknown_is_not_zero(self):
- h = host.HostSample(cpu_pct=5, per_core=[5], cpu_mhz=None, logical=1, physical=1, ram_total=100,
- ram_used=40, ram_available=60, swap_total=0, swap_used=0,
- procs=[host.HostProc(1, "a", None, None, None), host.HostProc(2, "b", 10, 5, 0.0)])
+ h = sample(procs=[host.HostProc(1, "a", None, None, None), host.HostProc(2, "b", 10, 5, 0.0)])
self.assertEqual([p.pid for p in h.top("rss")], [2]) # the unreadable process is not ranked as 0
+ self.assertEqual([p.pid for p in h.top("cpu_pct")], [2]) # a measured 0.0 is ranked, unknown is not
self.assertIsNone(h.proc(1).rss)
- self.assertIsNone(host.to_dict(h)["cpu"]["mhz"])
+ self.assertIsNone(h.proc(None))
+ d = host.to_dict(h)
+ self.assertEqual([p["pid"] for p in d["top_ram"]], [2])
+ self.assertEqual(d["top_cpu"][0]["cpu_pct"], 0.0)
+ self.assertNotIn("mhz", d["cpu"]) # psutil reports a nominal frequency, not a measurement
+
+ def test_json_keeps_unknown_as_null_and_rounds_rates(self):
+ h = sample(procs=[host.HostProc(2, "b", 10, None, 12.3456)])
+ d = host.to_dict(h)
+ self.assertEqual(d["top_ram"], [{"pid": 2, "name": "b", "rss": 10, "private": None, "cpu_pct": 12.3}])
+
+ def test_text_labels_the_source_and_unknowns(self):
+ h = sample(procs=[host.HostProc(2, "b", 10, None, None)], denied=3)
+ text = host.to_text(h)
+ self.assertIn("OS-reported; not GPU memory", text)
+ self.assertIn("n/a", text)
+ self.assertIn("3 processes not readable", text)
+
+
+class NtPathTests(unittest.TestCase):
+ """The Windows process-list path, driven by canned rows so every OS runs it."""
+
+ def setUp(self):
+ self.rows: dict = {}
+ self.clock = Clock()
+ for target in (patch.object(host, "_nt_query", lambda: dict(self.rows)),
+ patch.object(host, "_check_nt_layout"),
+ patch.object(host, "time", self.clock)):
+ target.start()
+ self.addCleanup(target.stop)
+ self.sampler = host.HostSampler()
+ self.sampler._logical = 4
+
+ def read(self, rows: dict, seconds: float = 0) -> host.HostSample:
+ self.rows = rows
+ self.clock.now += seconds
+ return self.sampler.sample()
+
+ def test_cpu_is_a_rate_that_needs_two_readings(self):
+ first = self.read({7: ("a.exe", 100, 50, 10 * SECOND, 5000.0)})
+ self.assertIsNone(first.proc(7).cpu_pct)
+ second = self.read({7: ("a.exe", 100, 50, 12 * SECOND, 5000.0)}, seconds=4)
+ # 2 s of CPU time in 4 s of wall clock on 4 logical CPUs: 12.5% of the whole machine
+ self.assertAlmostEqual(second.proc(7).cpu_pct, 12.5)
+ self.assertEqual((second.proc(7).rss, second.proc(7).private), (100, 50))
+ self.assertEqual(second.denied, 0) # no process handle is opened, so nothing is denied
+
+ def test_new_and_reused_pids_have_no_rate(self):
+ self.read({7: ("a.exe", 100, 50, 10 * SECOND, 5000.0)})
+ second = self.read({7: ("b.exe", 100, 50, 11 * SECOND, 6000.0), # pid 7 reused: new creation time
+ 8: ("c.exe", 100, 50, SECOND, 6000.0)}, seconds=1) # pid 8 appeared
+ self.assertIsNone(second.proc(7).cpu_pct)
+ self.assertIsNone(second.proc(8).cpu_pct)
+ third = self.read({7: ("b.exe", 100, 50, 12 * SECOND, 6000.0)}, seconds=1)
+ self.assertAlmostEqual(third.proc(7).cpu_pct, 25.0) # the new process is rated against itself
+
+ def test_cpu_time_going_backwards_is_unknown(self):
+ self.read({7: ("a.exe", 100, 50, 10 * SECOND, 5000.0)})
+ second = self.read({7: ("a.exe", 100, 50, 9 * SECOND, 5000.0)}, seconds=1)
+ self.assertIsNone(second.proc(7).cpu_pct)
+
+ def test_idle_process_is_not_a_process(self):
+ h = self.read({0: ("System Idle Process", 8, 0, 900 * SECOND, 0.0), 7: ("a.exe", 100, 50, SECOND, 5000.0)})
+ self.assertIsNone(h.proc(0))
+ self.assertEqual([p.pid for p in h.procs], [7])
+
+ def test_machine_without_a_usable_query_says_why(self):
+ with patch.object(host, "_nt_query", host._nt_unavailable):
+ with self.assertRaisesRegex(OSError, "64-bit Python on Windows"):
+ host.HostSampler()
+
+
+class LayoutCheckTests(unittest.TestCase):
+ """Rows are parsed by byte offset, so they are checked against our own process before anything is shown."""
+
+ def row(self, **change) -> dict:
+ me = psutil.Process()
+ fields = {"name": me.name(), "ws": me.memory_info().rss, "created": me.create_time(), **change}
+ return {me.pid: (fields["name"], fields["ws"], 0, 0, fields["created"])}
+
+ def test_a_row_that_matches_psutil_is_accepted(self):
+ host._check_nt_layout(self.row())
+
+ def test_a_row_that_disagrees_with_psutil_is_rejected(self):
+ me = psutil.Process()
+ for change in ({"name": "other.exe"}, {"created": me.create_time() + 60}, {"ws": 0},
+ {"ws": me.memory_info().rss * 10}):
+ with self.subTest(change=change), self.assertRaisesRegex(OSError, "layout not recognised"):
+ host._check_nt_layout(self.row(**change))
+
+ def test_missing_own_process_is_rejected(self):
+ with self.assertRaisesRegex(OSError, "layout not recognised"):
+ host._check_nt_layout({})
+
+
+class FakeProc:
+ def __init__(self, pid, name="p.exe", created=1.0, rss=10, cpu=40.0, denied=False, gone=False):
+ self.pid, self._name, self._created, self.rss = pid, name, created, rss
+ self.cpu, self.denied, self.gone = cpu, denied, gone
+
+ def oneshot(self):
+ return contextlib.nullcontext()
+
+ def name(self):
+ if self.gone:
+ raise psutil.NoSuchProcess(self.pid)
+ return self._name
+
+ def memory_info(self):
+ if self.denied:
+ raise psutil.AccessDenied(self.pid)
+ return type("MemoryInfo", (), {"rss": self.rss})()
+
+ def cpu_percent(self, interval=None):
+ return self.cpu
+
+ def create_time(self):
+ return self._created
+
+
+class PsutilPathTests(unittest.TestCase):
+ """Linux, macOS and WSL read each process through psutil."""
+
+ def setUp(self):
+ self.procs: list = []
+ for target in (patch.object(host, "_nt_query", None),
+ patch.object(host.psutil, "process_iter", lambda: iter(self.procs))):
+ target.start()
+ self.addCleanup(target.stop)
+ self.sampler = host.HostSampler()
+ self.sampler._logical = 4
+
+ def test_cpu_is_a_share_of_the_machine_after_the_second_reading(self):
+ self.procs = [FakeProc(1, cpu=100.0)]
+ self.assertIsNone(self.sampler.sample().proc(1).cpu_pct)
+ self.assertAlmostEqual(self.sampler.sample().proc(1).cpu_pct, 25.0) # psutil's 100% is one CPU of 4
+
+ def test_a_reused_pid_is_not_rated_against_the_previous_process(self):
+ self.procs = [FakeProc(1, created=1.0, cpu=100.0)]
+ self.sampler.sample()
+ self.procs = [FakeProc(1, created=2.0, cpu=100.0)]
+ self.assertIsNone(self.sampler.sample().proc(1).cpu_pct)
+
+ def test_protected_and_vanished_processes_are_unknown_or_skipped(self):
+ self.procs = [FakeProc(1, denied=True), FakeProc(2, gone=True), FakeProc(3, rss=7)]
+ h = self.sampler.sample()
+ self.assertEqual(h.denied, 1)
+ self.assertIsNone(h.proc(1).rss) # not readable: unknown, not 0
+ self.assertIsNone(h.proc(2))
+ self.assertEqual(h.proc(3).rss, 7)
+
- @unittest.skipUnless(sys.platform == "win32", "NtQuerySystemInformation is Windows-only")
+class LiveTests(unittest.TestCase):
+ @unittest.skipUnless(sys.platform == "win32" and struct.calcsize("P") == 8,
+ "NtQuerySystemInformation is 64-bit Windows only")
def test_nt_query_matches_psutil(self):
me = psutil.Process(os.getpid())
- name, ws, private, cpu = host._nt_query()[os.getpid()]
+ name, ws, private, cpu, created = host._nt_query()[os.getpid()]
mi = me.memory_info()
self.assertEqual(name.lower(), me.name().lower())
self.assertLess(abs(private - mi.private), 8 << 20)
self.assertLess(abs(ws - mi.rss), 16 << 20)
self.assertGreater(cpu, 0)
+ self.assertLess(abs(created - me.create_time()), 2)
def test_sampler_rates_cpu_after_second_sample(self):
s = host.HostSampler()
@@ -33,7 +206,8 @@ class HostTests(unittest.TestCase):
second = s.sample()
self.assertTrue(any(p.cpu_pct is not None for p in second.procs))
self.assertTrue(0 <= second.cpu_pct <= 100)
- self.assertGreater(second.ram_total, second.ram_used > 0)
+ self.assertGreater(second.ram_used, 0)
+ self.assertGreaterEqual(second.ram_total, second.ram_used)
if __name__ == "__main__":
diff --git a/tests/test_system.py b/tests/test_system.py
new file mode 100644
index 0000000..a8fa1e6
--- /dev/null
+++ b/tests/test_system.py
@@ -0,0 +1,292 @@
+"""System RAM and CPU: the panel, the table columns, the one-shot report, and the rules they must keep.
+
+These are OS-reported system-memory figures. They are never GPU bytes and never part of Snapshot.attributed.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import io
+import json
+import unittest
+from unittest.mock import patch
+
+from textual.widgets import Static
+
+from gputaskman.app import GpuTaskManager
+from gputaskman.cli import main
+from gputaskman.collector import ProcRow, Snapshot
+from gputaskman.diff import diff
+from gputaskman.host import HostProc, HostSample
+from gputaskman.pdh import ProcGpu
+from gputaskman.procinfo import ProcDetail
+from gputaskman.report import dump, to_dict
+
+GiB = 1 << 30
+MiB = 1 << 20
+
+
+def row(pid: int, name: str, kind: str = "other", dedicated: int = GiB, **extra) -> ProcRow:
+ return ProcRow(pid, name, kind, ProcGpu(dedicated=dedicated), ProcDetail(pid, name), **extra)
+
+
+def snapshot() -> Snapshot:
+ return Snapshot(taken=1, adapter=None, adapters=[], nv=None, driver="", cuda="",
+ vram_total=16 * GiB, vram_used=10 * GiB, shared_used=GiB,
+ procs=[row(10, "llama-server.exe", "LLM", 8 * GiB), row(20, "dwm.exe", "compositor", 2 * GiB)])
+
+
+def host_sample(**override) -> HostSample:
+ fields = dict(cpu_pct=37.0, per_core=[10.0, 90.0], logical=2, physical=2, ram_total=32 * GiB,
+ ram_used=12 * GiB, ram_available=20 * GiB, swap_total=4 * GiB, swap_used=GiB,
+ procs=[HostProc(10, "llama-server.exe", 6 * GiB, 5 * GiB, 12.0),
+ HostProc(20, "dwm.exe", 300 * MiB, 200 * MiB, None)])
+ return HostSample(**{**fields, **override})
+
+
+class SystemUiTests(unittest.IsolatedAsyncioTestCase):
+ def setUp(self):
+ backend = patch("gputaskman.app.BACKEND", "wddm") # these are WDDM snapshots, rendered the same on every OS
+ sampler = patch("gputaskman.app.HostSampler")
+ backend.start()
+ self.sampler = sampler.start()
+ self.sampler.return_value.sample.return_value = host_sample()
+ self.addCleanup(backend.stop)
+ self.addCleanup(sampler.stop)
+
+ def panel(self, app) -> list[str]:
+ content = app.query_one("#system", Static).content
+ return [text.plain for text in getattr(content, "renderables", [content])]
+
+ async def test_panel_and_columns_show_os_reported_values(self):
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([snapshot()], host_sample())
+ await pilot.pause()
+ table = app.query_one("#procs")
+ self.assertEqual(table.get_cell("10", "RAM").plain, "6.00 GB")
+ self.assertEqual(table.get_cell("10", "CPU%").plain, "12")
+ self.assertEqual(table.get_cell("20", "RAM").plain, "300 MB")
+ self.assertEqual(table.get_cell("20", "CPU%").plain, "…") # readable, but one more sample is needed
+ self.assertEqual(str(app.query_one("#system", Static).border_title), "System (OS-reported)")
+ text = "\n".join(self.panel(app))
+ for shown in ("12.00 GB / 32.00 GB", "free 20.00 GB", "1.00 GB/4.00 GB", "37%",
+ "2 threads/2 cores", "top RAM: llama-server.exe 6.00 GB", "top CPU: llama-server.exe 12%"):
+ self.assertIn(shown, text)
+
+ async def test_system_memory_is_never_added_to_gpu_totals(self):
+ s = snapshot()
+ attributed = s.attributed
+ self.assertEqual(attributed, 10 * GiB)
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([s], host_sample())
+ await pilot.pause()
+ self.assertEqual(s.attributed, attributed)
+ self.assertEqual(s.vram_used, 10 * GiB)
+
+ async def test_other_pid_spaces_never_borrow_a_local_process_reading(self):
+ s = snapshot()
+ foreign = row(10, "foreign.exe", host=True, origin="windows") # a Windows pid that equals a local pid
+ s.procs.append(foreign)
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([s], host_sample())
+ await pilot.pause()
+ table = app.query_one("#procs")
+ self.assertEqual(table.get_cell(foreign.key, "RAM").plain, "n/a")
+ self.assertEqual(table.get_cell(foreign.key, "CPU%").plain, "n/a")
+ self.assertEqual(table.get_cell("10", "RAM").plain, "6.00 GB")
+
+ async def test_processes_the_os_would_not_describe_show_n_a(self):
+ s = snapshot()
+ s.procs.append(row(30, "protected.exe")) # not in the host sample
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([s], host_sample())
+ await pilot.pause()
+ table = app.query_one("#procs")
+ self.assertEqual(table.get_cell("30", "RAM").plain, "n/a")
+ self.assertEqual(table.get_cell("30", "CPU%").plain, "n/a")
+
+ async def test_failed_sample_keeps_the_gpu_view_and_says_why(self):
+ self.sampler.return_value.sample.side_effect = OSError("denied")
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ host = app._sample_host()
+ self.assertIsNone(host)
+ app.apply_all([snapshot()], host)
+ await pilot.pause()
+ self.assertIn("RAM/CPU unavailable: OSError: denied", "\n".join(self.panel(app)))
+ table = app.query_one("#procs")
+ self.assertEqual(table.row_count, 2)
+ self.assertEqual(table.get_cell("10", "RAM").plain, "n/a")
+ self.assertIn("10.00 GB / 16.00 GB",
+ "\n".join(t.plain for t in app.query_one("#device", Static).content.renderables))
+ self.sampler.return_value.sample.side_effect = None # a later sample recovers
+ host = app._sample_host()
+ self.assertIsNotNone(host)
+ self.assertIsNone(app.host_error)
+ app.apply_all([snapshot()], host)
+ await pilot.pause()
+ self.assertEqual(table.get_cell("10", "RAM").plain, "6.00 GB")
+
+ async def test_a_sampler_that_cannot_start_says_why(self):
+ self.sampler.side_effect = OSError("layout not recognised")
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)):
+ self.assertIsNone(app._sample_host())
+ self.assertEqual(app.host_error, "OSError: layout not recognised")
+
+ async def test_no_system_removes_the_panel_columns_and_sorts(self):
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager(system=False)
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply(snapshot())
+ await pilot.pause()
+ self.assertEqual(len(app.query("#system")), 0)
+ for column in ("RAM", "CPU%"):
+ self.assertNotIn(column, app.query_one("#procs").columns)
+ self.assertEqual(app.sorts, GpuTaskManager.SORTS)
+ self.assertIsNone(app._sample_host())
+ self.sampler.assert_not_called()
+
+ async def test_ram_and_cpu_sorts_rank_unknown_last(self):
+ s = snapshot()
+ s.procs.insert(0, row(30, "protected.exe")) # not in the host sample
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([s], host_sample())
+ await pilot.pause()
+ for _ in GpuTaskManager.SORTS:
+ await pilot.press("s")
+ self.assertEqual(app.query_one("#procs").border_subtitle.split(" ·")[0], "sort: RAM")
+ self.assertEqual([r.pid for r in app.visible_rows(s)], [10, 20, 30])
+ await pilot.press("s")
+ self.assertEqual(app.query_one("#procs").border_subtitle.split(" ·")[0], "sort: CPU%")
+ self.assertEqual(app.visible_rows(s)[0].pid, 10)
+
+ async def test_panel_title_says_when_it_describes_the_wsl_vm(self):
+ with patch.object(GpuTaskManager, "poll"), patch("gputaskman.app.BACKEND", "wsl"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([snapshot()], host_sample())
+ await pilot.pause()
+ self.assertEqual(str(app.query_one("#system", Static).border_title), "System (WSL VM, OS-reported)")
+
+ async def test_gpu_panel_stays_above_the_system_panel_in_a_small_window(self):
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(100, 24)) as pilot:
+ app.apply_all([snapshot()], host_sample())
+ await pilot.pause()
+ device, system = app.query_one("#device"), app.query_one("#system")
+ self.assertEqual(device.region.y, app.query_one("#dashboard").region.y)
+ self.assertLess(device.region.y, system.region.y)
+
+ async def test_slim_mode_keeps_two_lines_and_full_mode_keeps_the_detail(self):
+ with patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(140, 42)) as pilot:
+ app.apply_all([snapshot()], host_sample())
+ await pilot.pause()
+ self.assertEqual(len(self.panel(app)), 6)
+ await pilot.press("m")
+ await pilot.pause()
+ slim = self.panel(app)
+ self.assertEqual([line[:3] for line in slim], ["RAM", "CPU"])
+ self.assertIn("12.00 GB / 32.00 GB", slim[0])
+ self.assertNotIn("threads", slim[1])
+
+ async def test_lines_never_wrap_at_any_width(self):
+ # 128 threads: the per-thread strip alone is wider than any of these terminals
+ sample = host_sample(per_core=[50.0] * 128, logical=128, physical=64)
+ for width in (60, 80, 100, 140, 200):
+ for slim in (False, True):
+ with self.subTest(width=width, slim=slim), patch.object(GpuTaskManager, "poll"):
+ app = GpuTaskManager()
+ async with app.run_test(size=(width, 42)) as pilot:
+ if slim:
+ await pilot.press("m")
+ app.apply_all([snapshot()], sample)
+ await pilot.pause()
+ panel = app.query_one("#system", Static)
+ lines = list(panel.content.renderables)
+ self.assertEqual(panel.content_size.height, len(lines)) # one row per line: nothing wrapped
+ # the RAM and CPU bars and the sparkline row are sized to fit whole, not cut off
+ for index in (0, 1, 3)[:len(lines)]:
+ self.assertLessEqual(lines[index].cell_len, panel.content_size.width, lines[index].plain)
+
+
+class SystemReportTests(unittest.TestCase):
+ def run_cli(self, *args) -> str:
+ output = io.StringIO()
+ with patch("sys.argv", ["gputaskman", *args]), patch("gputaskman.cli.time.sleep"), \
+ patch("gputaskman.collector.Collector") as collector, contextlib.redirect_stdout(output):
+ collector.return_value.sample.return_value = snapshot()
+ main()
+ return output.getvalue()
+
+ def test_json_carries_a_system_section_with_unknowns_as_null(self):
+ h = host_sample(procs=[HostProc(1, "a.exe", 10, None, 3.14159)])
+ text = dump(snapshot(), True, h)
+ system = json.loads(text)["system"]
+ self.assertEqual(system["ram"]["total"], 32 * GiB)
+ self.assertIn("OS-reported", system["source"])
+ self.assertEqual(system["top_ram"], [{"pid": 1, "name": "a.exe", "rss": 10, "private": None, "cpu_pct": 3.1}])
+ self.assertEqual(json.loads(text)["processes"], to_dict(snapshot())["processes"]) # the GPU part is untouched
+
+ def test_text_report_appends_the_system_section(self):
+ text = dump(snapshot(), False, host_sample())
+ self.assertIn("System RAM and CPU (OS-reported; not GPU memory)", text)
+ self.assertIn("llama-server.exe", text)
+
+ def test_unavailable_system_figures_say_why_in_json_and_text(self):
+ self.assertEqual(json.loads(dump(snapshot(), True, None, "OSError: boom"))["system"], {"error": "OSError: boom"})
+ self.assertIn("System RAM/CPU unavailable: OSError: boom", dump(snapshot(), False, None, "OSError: boom"))
+ self.assertNotIn("system", json.loads(dump(snapshot(), True))) # not asked for: no section at all
+
+ def test_json_one_shot_samples_twice_so_cpu_has_a_rate(self):
+ with patch("gputaskman.host.HostSampler") as factory:
+ factory.return_value.sample.return_value = host_sample()
+ output = self.run_cli("--json")
+ self.assertEqual(factory.return_value.sample.call_count, 2) # one primes the rates, one is reported
+ self.assertEqual(json.loads(output)["system"]["cpu"]["pct"], 37.0)
+
+ def test_no_system_flag_skips_the_sampler_and_the_section(self):
+ with patch("gputaskman.host.HostSampler") as factory:
+ output = self.run_cli("--json", "--no-system")
+ factory.assert_not_called()
+ self.assertNotIn("system", json.loads(output))
+
+ def test_a_failing_sampler_does_not_cost_the_gpu_report(self):
+ with patch("gputaskman.host.HostSampler") as factory:
+ factory.side_effect = OSError("layout not recognised")
+ output = self.run_cli("--json")
+ data = json.loads(output)
+ self.assertEqual(data["system"], {"error": "OSError: layout not recognised"})
+ self.assertEqual(len(data["processes"]), 2)
+
+ def test_a_sampler_that_fails_later_does_not_cost_the_gpu_report(self):
+ with patch("gputaskman.host.HostSampler") as factory:
+ factory.return_value.sample.side_effect = [host_sample(), OSError("gone")]
+ output = self.run_cli("--once")
+ self.assertIn("System RAM/CPU unavailable: OSError: gone", output)
+ self.assertIn("llama-server.exe", output)
+
+ def test_diff_ignores_the_system_section(self):
+ before, after = to_dict(snapshot()), to_dict(snapshot())
+ plain = diff(before, after)
+ with_system = diff({**before, "system": {"ram": {"used": 1}}}, {**after, "system": {"ram": {"used": 2}}})
+ self.assertEqual(with_system, plain)
+
+
+if __name__ == "__main__":
+ unittest.main()