Skip to content

A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).

Python

Add a record command that appends one JSON snapshot per interval

Scott Hanselmane2d7ddb80439
diff --git a/AGENTS.md b/AGENTS.md
index 352e9dc..41e73ce 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -60,6 +60,7 @@ cli.py ──► Collector.sample() (collector.py)
──► Snapshot ──► app.py (Textual TUI) / report.py (text and JSON)
cli.py ──► HostSampler.sample() (host.py: system RAM/CPU, OS-reported; sampled beside the Collector, not in Snapshot)
cli.py diff ──► diff.py: compares two saved --json snapshots (no collection)
+cli.py record ──► record.py: appends one --json snapshot per interval (Collector, or LiveCollectors on Windows, sampled as the TUI does)
```
The backend is chosen at import time in `collector.py`: `BACKEND` is `wddm` on Windows, `nvml` on
@@ -83,6 +84,7 @@ identify everything immediately; the TUI spreads scans across refreshes.
| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero. Refuse only when both snapshots name conflicting adapters (vendor, PCI address, a non-Windows key); when they cannot be compared, warn (`name-only`) instead, because a Windows LUID changes at boot and `diff` has no override flag. A PID whose two `started_at` values are both valid and more than `SAME_PROCESS_SECS` apart is a different process: no pairing rule may join it (unknown or malformed start times never conflict, so older snapshots pair as before) |
| Change system RAM/CPU | `host.py` | Machine memory and load as the OS reports them: never GPU bytes, never added to `Snapshot.attributed`, and `diff` ignores it. Unknown is `None`/`null` and a failure carries a reason, never zeros. The JSON key is `system` because `host` already means the WSL Windows host. Windows uses the NT process list (psutil per process takes about 6 s per pass for 1,000 processes), checked against psutil at startup; CPU rates are keyed by `(pid, create_time)` so a reused PID never inherits another process's history (on the psutil path, `is_running()` is what makes psutil drop its cached object for a reused PID; keep that call) |
| Change WSL matching | `wslhost.py` `ContextTracker` | Keep `tests/test_wslhost.py` replays passing; they are measured sequences |
+| Change recording | `record.py`, `cli.py` `record_main` | Every line must equal the `--json` object (`report.to_json_dict`, which `dump()` also uses), so unknown stays `null` and nothing is added or summed. Append only, one compact line per sample, flushed as it is written: Ctrl+C or a kill keeps every finished sample, and an interrupted sample writes nothing. Sample the way the TUI does (budgeted scanning, `LiveCollectors` on Windows so a GPU disable/re-enable is recorded as unavailable with a reason); never add a probe, `scan_budget=None`, or `psutil.open_files()` |
## Verify
diff --git a/README.md b/README.md
index b1f3a92..e2f1083 100644
--- a/README.md
+++ b/README.md
@@ -432,6 +432,7 @@ uv run --locked gputaskman -i 2 # sample every two seconds
uv run --locked gputaskman --once # one text snapshot
uv run --locked gputaskman --json # one JSON snapshot; implies --once
uv run --locked gputaskman diff before.json after.json # compare two --json snapshots
+uv run --locked gputaskman record -o session.jsonl --duration 300 # append one --json snapshot per second for 5 minutes
uv run --locked gputaskman --no-host # WSL: don't read Windows host counters through interop
uv run --locked gputaskman --no-system # don't sample system RAM/CPU: no panel, columns, or JSON section
uv run --locked gputaskman --help
@@ -632,6 +633,45 @@ estimated +504 MB of KV cache, but the server's measured dedicated memory grew b
Gemma 3 uses sliding-window attention, so the estimate is an upper bound. The comparison makes
that gap visible instead of hiding it.
+### Recording a session
+
+To keep what a long run did to the GPU (a training job, an overnight server, a game session),
+`record` appends one `--json` snapshot per interval to a file:
+
+```powershell
+uv run --locked gputaskman record -o session.jsonl # until Ctrl+C
+uv run --locked gputaskman record --gpu 0 --interval 5 --duration 300 -o session.jsonl
+```
+
+Each line is one complete `--json` snapshot: the same fields, the same `null` for anything that could
+not be measured, the same measured, inferred, and estimated labels. `taken` is the sample time in Unix
+seconds. The file is only ever appended to, so recording again continues it (a last line cut short by
+a crash is closed first). Every line is flushed to the operating system as it is written, so Ctrl+C
+or killing the process keeps every sample taken so far; a power loss can still lose the last few
+seconds, and a reader should skip a final line that does not parse. A full disk stops the recording
+with an error and leaves the lines already written.
+
+Recording samples the way the live TUI does, with no extra probes. Scanning is budgeted per sample, so
+model details can appear a sample or two after a process starts; on WSL the first sample comes a few
+seconds late while the Windows host counters warm up. On Windows, a GPU that is disabled and re-enabled
+stays in the recording: the samples in between carry `null` measurements and the reason, then resume
+when the adapter returns. The first version records one GPU (`--gpu`, default 0).
+
+A snapshot of 52 processes is about 27 KB, so one second between samples is roughly 100 MB an hour; use
+`--interval 5` or more for long runs. The lines carry the same paths, command-line flags, model names, and
+local ports as `--json` (see [Privacy](#privacy-and-troubleshooting)), so review a recording before you share it.
+
+Any line of the file is a normal snapshot, so two of them can go to `diff`:
+
+```powershell
+Get-Content session.jsonl -TotalCount 1 | Set-Content first.json
+Get-Content session.jsonl -Tail 1 | Set-Content last.json
+uv run --locked gputaskman diff first.json last.json
+```
+
+On Linux, WSL, and macOS use `head -n 1 session.jsonl > first.json` and `tail -n 1 session.jsonl > last.json`.
+Replaying or summarizing a whole recording is not built yet.
+
## How can it know what's inside?
It combines evidence rather than reading arbitrary GPU memory. Windows' graphics kernel supplies
@@ -779,6 +819,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:
| macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above |
| macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) |
| System RAM/CPU: process list + psutil totals | Windows (`NtQuerySystemInformation`): ~28 ms median for 995 processes (Windows 11, RTX 4080 SUPER machine, 32 threads); the contributor measured ~6 ms for ~480 processes (Windows 11, i9-13900K). Linux under WSL2 (psutil): ~3.4 ms median for 44 processes, ~9.5 ms for the first sample of 123; the per-pid reuse check (`is_running()`) adds ~0.3 ms for 44 processes, about 7 µs each | Every refresh; `--no-system` turns it off. psutil's per-process `memory_info`/`cpu_times` took ~6 s per pass for 995 processes on Windows, so it is only the non-Windows path. macOS unmeasured |
+| `record`: build and write one `--json` line | ~0.4 ms to build the object plus ~0.1 ms to serialize it, median of 30 for a 52-process, 27 KB line (Windows 11, RTX 4080 SUPER machine). A recorded sample is otherwise a normal refresh: 283 ms wall time for the first, ~90 ms from the fifth | Once per `--interval`, only while recording; no extra probes. Other platforms unmeasured |
With two llama-servers running, a steady refresh costs roughly 15-30 ms of CPU (about 340 ms before
process details were cached).
@@ -934,6 +975,7 @@ downloads are bundled in this repository.
| `collector.py` | Shared snapshot collection |
| `app.py` / `report.py` / `cli.py` | TUI, text/JSON, and command-line entry points |
| `diff.py` | Before/after comparison of two `--json` snapshots |
+| `record.py` | `gputaskman record`: appends one `--json` snapshot per interval to a JSON Lines file |
## Next
@@ -946,6 +988,7 @@ downloads are bundled in this repository.
`swapped`) as the paged column.
- WSL: per-process GPU load for Linux rows, and Ollama's systemd journal as a log source.
- Live throughput deltas, LM Studio validation, and per-model context accounting.
+- Replay and summaries of a `gputaskman record` file; today any one line works with `diff`.
- Opt-in ETW file-open tracing to name models loaded by read-and-close runtimes (any whisper or ONNX
app), correlated with the process's VRAM jump at load time.
- Data-driven app adapters (settings path plus load/unload log patterns) once a second app needs one.
diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.py
index df2b606..666d034 100644
--- a/src/gputaskman/cli.py
+++ b/src/gputaskman/cli.py
@@ -27,11 +27,68 @@ def diff_main(argv: list[str]) -> None:
print(json.dumps(result, indent=2) if args.json else to_text(result))
+def _check_platform() -> None:
+ if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):
+ sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu/Intel DRM) and macOS (Apple Silicon) only.")
+
+
+def _check_gpu(ap: argparse.ArgumentParser, index: int | None) -> None:
+ if index:
+ from .collector import enumerate_adapters, select_adapter
+
+ try:
+ select_adapter(enumerate_adapters(), index)
+ except ValueError as error:
+ ap.error(str(error))
+
+
+def record_main(argv: list[str]) -> None:
+ ap = argparse.ArgumentParser(prog="gputaskman record",
+ description="Append one --json snapshot per interval to a JSON Lines file, "
+ "for example to keep what a long run did to the GPU.")
+ ap.add_argument("-o", "--output", required=True, help="file to append to, one --json snapshot per line")
+ ap.add_argument("--gpu", type=int, default=0, help="GPU to record (default 0; gputaskman --list-gpus shows the indexes)")
+ ap.add_argument("-i", "--interval", type=float, default=1.0, help="seconds between samples (default 1)")
+ ap.add_argument("--duration", type=float, help="stop after this many seconds (default: until Ctrl+C)")
+ ap.add_argument("--no-host", action="store_true",
+ help="WSL: don't read Windows host GPU counters through interop (WSL VM total, Windows apps)")
+ ap.add_argument("--no-system", action="store_true", help="don't sample system RAM/CPU: no system section in the records")
+ args = ap.parse_args(argv)
+ if args.gpu < 0:
+ ap.error("--gpu must be a non-negative index")
+ if not math.isfinite(args.interval) or args.interval <= 0:
+ ap.error("--interval must be a finite number greater than zero")
+ if args.duration is not None and (not math.isfinite(args.duration) or args.duration <= 0):
+ ap.error("--duration must be a finite number greater than zero")
+ _check_platform()
+ _check_gpu(ap, args.gpu)
+
+ from .record import append_to, run, source
+
+ try:
+ out = append_to(args.output)
+ except OSError as error:
+ ap.error(f"cannot write {args.output}: {error}")
+ count = 0
+ try:
+ with out, source(args.gpu, host=not args.no_host, system=not args.no_system) as sample:
+ span = f" for {args.duration:g} s" if args.duration is not None else ""
+ print(f"Recording GPU {args.gpu} every {args.interval:g} s{span} to {args.output}; "
+ "press Ctrl+C to stop.", file=sys.stderr)
+ count = run(sample, out, interval=args.interval, duration=args.duration)
+ except KeyboardInterrupt:
+ pass # Ctrl+C while starting up: nothing was recorded yet
+ print(f"Recorded {count} sample(s) to {args.output}", file=sys.stderr)
+
+
def main() -> None:
if len(sys.argv) > 1 and sys.argv[1] == "diff":
return diff_main(sys.argv[2:])
+ if len(sys.argv) > 1 and sys.argv[1] == "record":
+ return record_main(sys.argv[2:])
ap = argparse.ArgumentParser(prog="gputaskman", description="What's inside your GPU: per-process VRAM, engines, and LLM weights/KV-cache breakdown.",
- epilog="gputaskman diff BEFORE.json AFTER.json compares two --json snapshots.")
+ epilog="gputaskman diff BEFORE.json AFTER.json compares two --json snapshots; "
+ "gputaskman record -o FILE appends one --json snapshot per interval to FILE.")
ap.add_argument("--once", action="store_true", help="print one snapshot and exit (no TUI)")
ap.add_argument("--json", action="store_true", help="print one JSON snapshot and exit (implies --once)")
ap.add_argument("--gpu", type=int, help="show only this GPU (TUI default: all; one-shot default: 0)")
@@ -47,8 +104,7 @@ def main() -> None:
if not math.isfinite(args.interval) or args.interval <= 0:
ap.error("--interval must be a finite number greater than zero")
- if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):
- sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu/Intel DRM) and macOS (Apple Silicon) only.")
+ _check_platform()
if args.list_gpus:
from .collector import enumerate_adapters, ordered_adapters
@@ -57,13 +113,7 @@ def main() -> None:
print(f"{i}: {adapter.name} ({adapter.vendor}, LUID 0x{adapter.luid & 0xffffffffffffffff:016x})")
return
- if args.gpu:
- from .collector import enumerate_adapters, select_adapter
-
- try:
- select_adapter(enumerate_adapters(), args.gpu)
- except ValueError as error:
- ap.error(str(error))
+ _check_gpu(ap, args.gpu)
if args.once or args.json:
from .collector import Collector
diff --git a/src/gputaskman/record.py b/src/gputaskman/record.py
new file mode 100644
index 0000000..c688a39
--- /dev/null
+++ b/src/gputaskman/record.py
@@ -0,0 +1,107 @@
+"""gputaskman record: append one --json snapshot per interval to a JSON Lines file."""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+from contextlib import ExitStack, contextmanager
+
+from .report import to_json_dict
+
+
+def make_sample(collect, system=None):
+ """A callable returning one record: exactly the object --json prints, so every saved line is a normal snapshot."""
+
+ def sample() -> dict:
+ snapshot = collect()
+ host, error = None, None
+ if system is not None:
+ try:
+ host = system()
+ except Exception as exc: # system RAM/CPU is extra: its failure must not cost the GPU record
+ error = f"{type(exc).__name__}: {exc}"
+ return to_json_dict(snapshot, host, error)
+
+ return sample
+
+
+def run(sample, out, *, interval: float, duration: float | None = None,
+ clock=time.monotonic, sleep=time.sleep) -> int:
+ """Write one compact JSON line per interval, flushed as it goes; return how many were written.
+
+ Stops after duration seconds, or on Ctrl+C. An interrupted sample writes nothing, so every line is complete.
+ A slow sample delays the next one instead of triggering a burst to catch up.
+ """
+ count = 0
+ start = due = clock()
+ try:
+ while duration is None or due - start < duration:
+ if (wait := due - clock()) > 0:
+ sleep(wait)
+ out.write(json.dumps(sample(), default=str, separators=(",", ":")) + "\n")
+ out.flush()
+ count += 1
+ due = max(due + interval, clock())
+ except KeyboardInterrupt:
+ pass
+ return count
+
+
+def append_to(path: str):
+ """Open path for appending (never overwriting); a record never glues onto a last line that was cut short."""
+ needs_newline = False
+ try:
+ with open(path, "rb") as existing:
+ existing.seek(-1, os.SEEK_END)
+ needs_newline = existing.read(1) != b"\n"
+ except OSError:
+ pass # missing, empty, or not seekable: nothing to protect
+ out = open(path, "a", encoding="utf-8", newline="\n")
+ if needs_newline:
+ out.write("\n")
+ return out
+
+
+@contextmanager
+def source(gpu: int, *, host: bool = True, system: bool = True):
+ """Sample the way the TUI does and yield a sample() callable.
+
+ Scanning is budgeted like the TUI's, and on Windows a GPU that is disabled and re-enabled is rediscovered:
+ the records in between carry null measurements and the reason, never zeros.
+ """
+ from .collector import BACKEND, Collector, enumerate_adapters
+ from .host import HostSampler
+
+ sampler = None
+
+ def read_system():
+ nonlocal sampler
+ if sampler is None:
+ sampler = HostSampler()
+ return sampler.sample()
+
+ with ExitStack() as stack:
+ if BACKEND == "wddm":
+ from .recovery import LiveCollectors
+
+ live = LiveCollectors(gpu, collector_factory=Collector, discover=enumerate_adapters)
+ stack.callback(live.close)
+
+ def collect():
+ return live.sample()[0]
+ else:
+ collector = Collector(gpu, host=host)
+ stack.callback(collector.close)
+ if collector.host is not None:
+ collector.host.wait(15.0) # WSL: the first Windows counter sample takes several seconds
+ if hasattr(collector.counters, "warm_up"):
+ collector.counters.warm_up(15.0) # macOS: per-process GPU memory comes from a background pass
+ collect = collector.sample
+ if system:
+ try:
+ read_system() # per-process CPU% is a rate too; this primes it
+ except Exception:
+ pass # each record carries the reason if it still fails
+ time.sleep(1.0) # engine utilization is a rate; needs two samples
+ yield make_sample(collect, read_system if system else None)
diff --git a/src/gputaskman/report.py b/src/gputaskman/report.py
index ac6c8c3..5bb75ce 100644
--- a/src/gputaskman/report.py
+++ b/src/gputaskman/report.py
@@ -273,17 +273,24 @@ def to_text(s: Snapshot, limit: int = 30) -> str:
return "\n".join(out)
+def to_json_dict(s: Snapshot, system=None, system_error: str | None = None) -> dict:
+ """The --json object: to_dict plus the optional OS-reported "system" section (or why it is missing)."""
+ from . import host as hostmod
+
+ d = to_dict(s)
+ if system is not None:
+ d["system"] = hostmod.to_dict(system)
+ elif system_error:
+ d["system"] = {"error": system_error}
+ return d
+
+
def dump(s: Snapshot, as_json: bool, system=None, system_error: str | None = None) -> str:
"""system: an optional host.HostSample (OS-reported system RAM/CPU); system_error: why it is missing."""
from . import host as hostmod
if as_json:
- d = to_dict(s)
- if system is not None:
- d["system"] = hostmod.to_dict(system)
- elif system_error:
- d["system"] = {"error": system_error}
- return json.dumps(d, indent=2, default=str)
+ return json.dumps(to_json_dict(s, system, system_error), indent=2, default=str)
text = to_text(s)
if system is not None:
return text + "\n\n" + hostmod.to_text(system)
diff --git a/tests/test_record.py b/tests/test_record.py
new file mode 100644
index 0000000..bf20ad6
--- /dev/null
+++ b/tests/test_record.py
@@ -0,0 +1,513 @@
+"""gputaskman record: one --json snapshot per interval, appended to a JSON Lines file.
+
+Hardware-independent: samples are fakes, time is a fake clock and the collectors are mocks. A recording is the
+same data as --json, so unknown stays null (never 0) and every line must be one complete JSON object.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import errno
+import io
+import json
+import os
+import tempfile
+import types
+import unittest
+from unittest.mock import MagicMock, patch
+
+from gputaskman import collector as collector_module
+from gputaskman import record
+from gputaskman.cli import main
+from gputaskman.collector import ProcRow, Snapshot
+from gputaskman.diff import diff, load
+from gputaskman.dxgi import Adapter
+from gputaskman.host import HostProc, HostSample
+from gputaskman.host import to_dict as host_to_dict
+from gputaskman.pdh import ProcGpu
+from gputaskman.procinfo import ProcDetail
+from gputaskman.report import dump, to_dict, to_json_dict
+
+GiB = 1 << 30
+
+
+def row(pid: int, name: str, kind: str = "other", dedicated: int = GiB) -> ProcRow:
+ return ProcRow(pid, name, kind, ProcGpu(dedicated=dedicated), ProcDetail(pid, name))
+
+
+def snapshot(taken: float = 1.0, used: int = 10 * GiB) -> Snapshot:
+ return Snapshot(taken=taken, adapter=None, adapters=[], nv=None, driver="", cuda="",
+ vram_total=16 * GiB, vram_used=used, shared_used=GiB,
+ procs=[row(10, "llama-server.exe", "LLM", 8 * GiB), row(20, "dwm.exe", "compositor", 2 * GiB)])
+
+
+def host_sample() -> HostSample:
+ return HostSample(cpu_pct=37.0, per_core=[10.0, 90.0], logical=2, physical=2, ram_total=32 * GiB,
+ ram_used=12 * GiB, ram_available=20 * GiB, swap_total=4 * GiB, swap_used=GiB,
+ procs=[HostProc(10, "llama-server.exe", 6 * GiB, 5 * GiB, 12.0)])
+
+
+class Clock:
+ """Time only moves when something sleeps or a sample takes time."""
+
+ def __init__(self):
+ self.now = 1000.0
+ self.sleeps = []
+
+ def __call__(self) -> float:
+ return self.now
+
+ def sleep(self, seconds: float) -> None:
+ self.sleeps.append(seconds)
+ self.now += seconds
+
+
+class Sink:
+ """A file that keeps what was flushed apart from what is only buffered."""
+
+ def __init__(self):
+ self.pending = self.flushed = ""
+
+ def write(self, text: str) -> None:
+ self.pending += text
+
+ def flush(self) -> None:
+ self.flushed, self.pending = self.flushed + self.pending, ""
+
+ def lines(self) -> list[str]:
+ return self.flushed.splitlines()
+
+
+class RunTests(unittest.TestCase):
+ def setUp(self):
+ self.clock = Clock()
+ self.sink = Sink()
+
+ def run_for(self, sample, **kwargs) -> int:
+ return record.run(sample, self.sink, clock=self.clock, sleep=self.clock.sleep, **kwargs)
+
+ def test_one_compact_json_line_per_sample_flushed_as_it_goes(self):
+ seen = []
+
+ def sample():
+ seen.append(len(self.sink.lines())) # lines already flushed when this sample is taken
+ return {"taken": len(seen), "vram_used": None}
+
+ self.assertEqual(self.run_for(sample, interval=1.0, duration=3.0), 3)
+ self.assertEqual(seen, [0, 1, 2])
+ self.assertEqual(self.sink.pending, "")
+ self.assertEqual(self.sink.lines(), ['{"taken":1,"vram_used":null}', '{"taken":2,"vram_used":null}',
+ '{"taken":3,"vram_used":null}'])
+
+ def test_a_duration_sets_the_sample_count_and_nothing_waits_after_the_last_one(self):
+ self.assertEqual(self.run_for(lambda: {}, interval=1.0, duration=300.0), 300)
+ self.assertEqual(len(self.clock.sleeps), 299)
+ self.assertEqual(self.clock.now - 1000.0, 299.0)
+
+ def test_a_duration_shorter_than_the_interval_records_one_sample(self):
+ self.assertEqual(self.run_for(lambda: {}, interval=5.0, duration=1.0), 1)
+ self.assertEqual(self.clock.sleeps, [])
+
+ def test_samples_keep_their_schedule_however_long_each_one_takes(self):
+ starts = []
+
+ def sample():
+ starts.append(self.clock.now - 1000.0)
+ self.clock.now += 0.25 # collecting takes time too
+ return {}
+
+ self.run_for(sample, interval=1.0, duration=4.0)
+ self.assertEqual(starts, [0.0, 1.0, 2.0, 3.0]) # not 0, 1.25, 2.5 ...
+
+ def test_a_slow_sample_delays_the_next_one_with_no_burst_to_catch_up(self):
+ starts, took = [], iter([0.0, 3.5, 0.0, 0.0])
+
+ def sample():
+ starts.append(self.clock.now - 1000.0)
+ self.clock.now += next(took)
+ return {}
+
+ self.assertEqual(self.run_for(sample, interval=1.0, duration=6.0), 4)
+ self.assertEqual(starts, [0.0, 1.0, 4.5, 5.5])
+
+ def test_ctrl_c_while_waiting_leaves_every_line_complete(self):
+ def interrupt(seconds):
+ raise KeyboardInterrupt
+
+ self.assertEqual(record.run(lambda: {"n": 1}, self.sink, interval=1.0, clock=self.clock, sleep=interrupt), 1)
+ self.assertEqual(self.sink.flushed, '{"n":1}\n')
+
+ def test_ctrl_c_during_a_sample_writes_nothing_for_it(self):
+ calls = []
+
+ def sample():
+ calls.append(1)
+ if len(calls) == 3:
+ raise KeyboardInterrupt
+ return {"n": len(calls)}
+
+ self.assertEqual(self.run_for(sample, interval=1.0), 2) # with no duration only Ctrl+C ends it
+ self.assertEqual(self.sink.flushed, '{"n":1}\n{"n":2}\n')
+ self.assertEqual(self.sink.pending, "")
+
+ def test_each_record_is_one_ascii_line_even_with_odd_names_and_newlines(self):
+ data = {"name": "mödel-日本.exe", "cmd": "a\nb\r\nc", "path": "C:\\Users\\x", "unknown": None}
+ self.run_for(lambda: data, interval=1.0, duration=0.5)
+ self.assertTrue(self.sink.flushed.isascii())
+ self.assertEqual(self.sink.flushed.count("\n"), 1)
+ self.assertEqual(json.loads(self.sink.flushed), data)
+
+ def test_a_value_json_cannot_encode_is_written_as_its_text_not_a_traceback(self):
+ class Odd:
+ def __str__(self):
+ return "odd value"
+
+ self.run_for(lambda: {"n": 1, "odd": Odd()}, interval=1.0, duration=0.5)
+ self.assertEqual(json.loads(self.sink.flushed), {"n": 1, "odd": "odd value"})
+
+ def test_a_full_disk_stops_the_recording_with_its_error_and_keeps_the_earlier_lines(self):
+ class FullDisk(Sink):
+ def write(self, text: str) -> None:
+ if len(self.lines()) == 2:
+ raise OSError(errno.ENOSPC, "No space left on device")
+ super().write(text)
+
+ sink = FullDisk()
+ with self.assertRaises(OSError) as stop:
+ record.run(lambda: {"n": 1}, sink, interval=1.0, clock=self.clock, sleep=self.clock.sleep)
+ self.assertEqual(stop.exception.errno, errno.ENOSPC)
+ self.assertEqual(sink.lines(), ['{"n":1}', '{"n":1}'])
+
+
+class MakeSampleTests(unittest.TestCase):
+ def test_a_record_is_the_object_json_prints(self):
+ sample = record.make_sample(snapshot, host_sample)
+ self.assertEqual(json.loads(json.dumps(sample(), default=str)),
+ json.loads(dump(snapshot(), True, host_sample())))
+
+ def test_system_figures_are_optional_and_their_failure_is_recorded_not_fatal(self):
+ def broken():
+ raise OSError("layout not recognised")
+
+ failed = record.make_sample(snapshot, broken)()
+ self.assertEqual(failed["system"], {"error": "OSError: layout not recognised"})
+ self.assertEqual(failed["processes"], to_dict(snapshot())["processes"]) # the GPU part is intact
+ self.assertNotIn("system", record.make_sample(snapshot)())
+
+ def test_a_gpu_that_is_gone_is_recorded_as_unavailable_not_zero(self):
+ from gputaskman.recovery import LiveCollectors
+
+ live = LiveCollectors(0, collector_factory=MagicMock(), discover=lambda strict=True: [])
+ data = record.make_sample(lambda: live.sample()[0])()
+ self.assertIn("Adapter unavailable", data["counter_status"])
+ self.assertIs(data["counter_adapter_matched"], False)
+ for key in ("vram_total", "vram_used", "shared_used", "attributed", "device"):
+ self.assertIsNone(data[key], key)
+ self.assertEqual(data["processes"], [])
+
+
+class JsonDumpTests(unittest.TestCase):
+ def test_dump_json_is_still_the_indented_object_with_its_optional_system_section(self):
+ s, h = snapshot(), host_sample()
+ self.assertEqual(dump(s, True), json.dumps(to_dict(s), indent=2, default=str))
+ self.assertEqual(dump(s, True, h), json.dumps({**to_dict(s), "system": host_to_dict(h)}, indent=2, default=str))
+ self.assertEqual(dump(s, True, None, "OSError: boom"),
+ json.dumps({**to_dict(s), "system": {"error": "OSError: boom"}}, indent=2, default=str))
+ self.assertEqual(to_json_dict(s, h), {**to_dict(s), "system": host_to_dict(h)})
+ self.assertNotIn("system", to_json_dict(s))
+
+
+class FileTestCase(unittest.TestCase):
+ def setUp(self):
+ self.dir = tempfile.TemporaryDirectory()
+ self.addCleanup(self.dir.cleanup)
+ self.path = os.path.join(self.dir.name, "session.jsonl")
+
+ def content(self) -> bytes:
+ with open(self.path, "rb") as f:
+ return f.read()
+
+
+class AppendTests(FileTestCase):
+ def append(self, text: str) -> None:
+ with record.append_to(self.path) as out:
+ out.write(text)
+
+ def test_creates_the_file_and_appends_without_touching_what_is_there(self):
+ self.append('{"n":1}\n')
+ self.append('{"n":2}\n')
+ self.assertEqual(self.content(), b'{"n":1}\n{"n":2}\n')
+
+ def test_lines_end_with_a_bare_newline_on_every_platform(self):
+ self.append('{"n":1}\n')
+ self.assertNotIn(b"\r", self.content())
+
+ def test_a_last_line_cut_short_by_a_crash_is_closed_before_the_next_record(self):
+ with open(self.path, "wb") as f:
+ f.write(b'{"n":1}\n{"n":2')
+ self.append('{"n":3}\n')
+ self.assertEqual(self.content(), b'{"n":1}\n{"n":2\n{"n":3}\n')
+
+ def test_an_empty_file_gets_no_stray_newline(self):
+ open(self.path, "wb").close()
+ self.append('{"n":1}\n')
+ self.assertEqual(self.content(), b'{"n":1}\n')
+
+ def test_a_path_that_cannot_be_written_raises_for_the_cli_to_report(self):
+ with self.assertRaises(OSError):
+ record.append_to(self.dir.name) # a directory
+
+ def test_a_recording_written_to_disk_is_valid_json_lines_that_diff_accepts(self):
+ snapshots = iter([snapshot(taken=1.0, used=4 * GiB), snapshot(taken=2.0, used=6 * GiB),
+ snapshot(taken=3.0, used=9 * GiB)])
+ clock = Clock()
+ with record.append_to(self.path) as out:
+ count = record.run(record.make_sample(lambda: next(snapshots), host_sample), out,
+ interval=1.0, duration=3.0, clock=clock, sleep=clock.sleep)
+ self.assertEqual(count, 3)
+ with open(self.path, encoding="utf-8") as f:
+ records = [json.loads(line) for line in f]
+ self.assertEqual([r["taken"] for r in records], [1.0, 2.0, 3.0])
+ self.assertTrue(all(r["system"]["ram"]["total"] == 32 * GiB for r in records))
+ first, last = (os.path.join(self.dir.name, name) for name in ("first.json", "last.json"))
+ for target, line in ((first, records[0]), (last, records[-1])):
+ with open(target, "w", encoding="utf-8") as f:
+ f.write(json.dumps(line)) # one line saved to its own file is a normal --json snapshot
+ self.assertEqual(diff(load(first), load(last))["device"]["vram_used"]["delta"], 5 * GiB)
+
+
+class SourceTests(unittest.TestCase):
+ def setUp(self):
+ self.sleep = patch("gputaskman.record.time.sleep").start()
+ self.sampler = patch("gputaskman.host.HostSampler").start()
+ self.addCleanup(patch.stopall)
+ self.sampler.return_value.sample.return_value = host_sample()
+
+ def collector(self) -> MagicMock:
+ c = MagicMock()
+ c.host = None
+ c.counters = types.SimpleNamespace() # no warm_up, like Windows and native Linux
+ c.sample.return_value = snapshot()
+ return c
+
+ def test_native_linux_samples_one_collector_with_budgeted_scanning_and_closes_it(self):
+ c = self.collector()
+ with patch("gputaskman.collector.BACKEND", "nvml"), \
+ patch("gputaskman.collector.Collector", return_value=c) as factory:
+ with record.source(2) as sample:
+ data = sample()
+ factory.assert_called_once_with(2, host=True) # the TUI's budget, not scan_budget=None
+ c.close.assert_called_once_with()
+ self.sleep.assert_called_once_with(1.0) # engine utilization is a rate
+ self.assertEqual(data["taken"], 1.0)
+ self.assertEqual(self.sampler.return_value.sample.call_count, 2) # the first reading only primes CPU%
+ self.assertEqual(data["system"]["ram"]["total"], 32 * GiB)
+
+ def test_no_host_is_passed_on(self):
+ with patch("gputaskman.collector.BACKEND", "wsl"), \
+ patch("gputaskman.collector.Collector", return_value=self.collector()) as factory:
+ with record.source(0, host=False):
+ pass
+ factory.assert_called_once_with(0, host=False)
+
+ def test_wsl_host_counters_and_macos_footprints_get_their_warm_up(self):
+ c = self.collector()
+ c.host, c.counters = MagicMock(), MagicMock()
+ with patch("gputaskman.collector.BACKEND", "wsl"), patch("gputaskman.collector.Collector", return_value=c):
+ with record.source(0):
+ pass
+ c.host.wait.assert_called_once_with(15.0)
+ c.counters.warm_up.assert_called_once_with(15.0)
+
+ def test_windows_rediscovers_a_gpu_that_was_disabled_and_records_the_selected_slot(self):
+ with patch("gputaskman.collector.BACKEND", "wddm"), patch("gputaskman.recovery.LiveCollectors") as live:
+ live.return_value.sample.return_value = [snapshot(taken=1.0), snapshot(taken=2.0)]
+ with record.source(1, system=False) as sample:
+ data = sample()
+ args, kwargs = live.call_args
+ self.assertEqual(args, (1,))
+ self.assertIs(kwargs["collector_factory"], collector_module.Collector)
+ self.assertIs(kwargs["discover"], collector_module.enumerate_adapters)
+ live.return_value.close.assert_called_once_with()
+ self.assertEqual(data["taken"], 1.0)
+ self.assertNotIn("system", data)
+ self.sampler.assert_not_called()
+
+ def test_collectors_are_closed_even_when_recording_fails(self):
+ c = self.collector()
+ with patch("gputaskman.collector.BACKEND", "nvml"), patch("gputaskman.collector.Collector", return_value=c):
+ with self.assertRaises(RuntimeError), record.source(0):
+ raise RuntimeError("disk full")
+ c.close.assert_called_once_with()
+
+ def test_a_broken_system_sampler_never_stops_the_gpu_records(self):
+ self.sampler.side_effect = OSError("layout not recognised")
+ with patch("gputaskman.collector.BACKEND", "nvml"), \
+ patch("gputaskman.collector.Collector", return_value=self.collector()):
+ with record.source(0) as sample:
+ data = sample()
+ self.assertEqual(data["system"], {"error": "OSError: layout not recognised"})
+ self.assertEqual(len(data["processes"]), 2)
+
+ def test_one_system_sampler_serves_every_record_because_cpu_percent_is_a_rate_between_readings(self):
+ with patch("gputaskman.collector.BACKEND", "nvml"), \
+ patch("gputaskman.collector.Collector", return_value=self.collector()):
+ with record.source(0) as sample:
+ sample()
+ sample()
+ self.sampler.assert_called_once_with()
+ self.assertEqual(self.sampler.return_value.sample.call_count, 3) # one priming read, then one per record
+
+ def test_no_system_never_builds_a_sampler_and_leaves_no_section(self):
+ with patch("gputaskman.collector.BACKEND", "nvml"), \
+ patch("gputaskman.collector.Collector", return_value=self.collector()):
+ with record.source(0, system=False) as sample:
+ data = sample()
+ self.sampler.assert_not_called()
+ self.assertNotIn("system", data)
+
+
+class CliTests(FileTestCase):
+ def setUp(self):
+ super().setUp()
+ self.calls = []
+ self.samples = 0
+ self.interrupt_at = None
+
+ def fake_source(self, fail=None):
+ calls = self.calls
+
+ @contextlib.contextmanager
+ def source(gpu, *, host, system):
+ calls.append((gpu, host, system))
+ if fail is not None:
+ raise fail
+
+ def sample():
+ self.samples += 1
+ if self.samples == self.interrupt_at:
+ raise KeyboardInterrupt
+ if self.samples > 5: # a recording that ignores its --duration must fail the test, not hang it
+ raise AssertionError("the recording ran on past what the test asked for")
+ return to_json_dict(snapshot(taken=float(self.samples)))
+
+ yield sample
+
+ return patch("gputaskman.record.source", source)
+
+ def refused(self):
+ """A source that fails the test at once: a recording must never start for input the command refuses."""
+ return self.fake_source(fail=AssertionError("a recording started for input that must be refused"))
+
+ def run_cli(self, *args, fail=None) -> str:
+ err = io.StringIO()
+ with self.fake_source(fail), patch("sys.argv", ["gputaskman", "record", *args]), \
+ contextlib.redirect_stderr(err):
+ main()
+ return err.getvalue()
+
+ def lines(self) -> list[str]:
+ return self.content().decode("utf-8").splitlines()
+
+ def test_records_a_snapshot_and_says_so_on_stderr(self):
+ err = self.run_cli("-o", self.path, "--duration", "0.5")
+ self.assertEqual(len(self.lines()), 1)
+ self.assertEqual(json.loads(self.lines()[0]), json.loads(dump(snapshot(taken=1.0), True)))
+ self.assertEqual(self.calls, [(0, True, True)])
+ self.assertIn("Recording GPU 0 every 1 s for 0.5 s", err)
+ self.assertIn("Recorded 1 sample(s)", err)
+
+ def test_the_flags_reach_the_sampling_side(self):
+ self.run_cli("--output", self.path, "--gpu", "0", "--interval", "2", "--duration", "1", "--no-host", "--no-system")
+ self.assertEqual(self.calls, [(0, False, False)])
+
+ def test_every_value_reaches_the_side_that_uses_it(self):
+ adapters = [Adapter(1, "GPU 0", 0x10DE, GiB, 0, False), Adapter(2, "GPU 1", 0x10DE, GiB, 0, False)]
+ for name, args, source_args, run_kwargs in (
+ ("given", ["--gpu", "1", "--interval", "2", "--duration", "1", "--no-host"],
+ (1, False, True), {"interval": 2.0, "duration": 1.0}),
+ ("defaults", [], (0, True, True), {"interval": 1.0, "duration": None})):
+ with self.subTest(name):
+ self.calls.clear()
+ with patch("gputaskman.collector.enumerate_adapters", return_value=adapters), \
+ patch("gputaskman.record.run", return_value=1) as run:
+ self.run_cli("-o", self.path, *args)
+ self.assertEqual(self.calls, [source_args])
+ self.assertEqual(run.call_args.kwargs, run_kwargs)
+
+ def test_a_second_recording_appends_and_a_cut_last_line_is_closed_first(self):
+ with open(self.path, "wb") as f:
+ f.write(b'{"taken":0.5}\n{"taken"')
+ self.run_cli("-o", self.path, "--duration", "0.5")
+ self.run_cli("-o", self.path, "--duration", "0.5")
+ lines = self.lines()
+ self.assertEqual(lines[:2], ['{"taken":0.5}', '{"taken"'])
+ self.assertEqual([json.loads(line)["taken"] for line in lines[2:]], [1.0, 2.0]) # no blank lines in between
+
+ def test_ctrl_c_ends_a_recording_cleanly_and_keeps_what_was_written(self):
+ self.interrupt_at = 3
+ err = self.run_cli("-o", self.path, "-i", "0.01")
+ self.assertEqual([json.loads(line)["taken"] for line in self.lines()], [1.0, 2.0])
+ self.assertIn("Recorded 2 sample(s)", err)
+
+ def test_ctrl_c_while_starting_up_exits_quietly(self):
+ err = self.run_cli("-o", self.path, fail=KeyboardInterrupt())
+ self.assertIn("Recorded 0 sample(s)", err)
+ self.assertEqual(self.content(), b"")
+
+ def test_bad_arguments_are_refused_before_anything_is_opened(self):
+ cases = {
+ "--interval 0": ["-o", self.path, "--interval", "0"],
+ "--interval -1": ["-o", self.path, "--interval", "-1"],
+ "--interval nan": ["-o", self.path, "--interval", "nan"],
+ "--interval inf": ["-o", self.path, "--interval", "inf"],
+ "--duration 0": ["-o", self.path, "--duration", "0"],
+ "--duration -1": ["-o", self.path, "--duration", "-1"],
+ "--duration nan": ["-o", self.path, "--duration", "nan"],
+ "--duration inf": ["-o", self.path, "--duration", "inf"],
+ "--gpu -1": ["-o", self.path, "--gpu", "-1"],
+ "no output": [],
+ }
+ for name, args in cases.items():
+ with self.subTest(name):
+ err = io.StringIO()
+ with self.refused(), patch("sys.argv", ["gputaskman", "record", *args]), \
+ patch("gputaskman.collector.enumerate_adapters",
+ side_effect=AssertionError("no GPU may be touched while refusing the arguments")), \
+ contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as stop:
+ main()
+ self.assertEqual(stop.exception.code, 2)
+ self.assertEqual(self.calls, [])
+ self.assertFalse(os.path.exists(self.path))
+
+ def test_a_gpu_that_does_not_exist_is_refused_up_front(self):
+ err = io.StringIO()
+ with self.refused(), patch("gputaskman.collector.enumerate_adapters", return_value=[]), \
+ patch("sys.argv", ["gputaskman", "record", "-o", self.path, "--gpu", "2"]), \
+ contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as stop:
+ main()
+ self.assertEqual(stop.exception.code, 2)
+ self.assertIn("--gpu 2 is out of range", err.getvalue())
+ self.assertEqual(self.calls, [])
+ self.assertFalse(os.path.exists(self.path))
+
+ def test_an_output_that_cannot_be_written_is_refused(self):
+ err = io.StringIO()
+ with self.refused(), patch("sys.argv", ["gputaskman", "record", "-o", self.dir.name]), \
+ contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as stop:
+ main()
+ self.assertEqual(stop.exception.code, 2)
+ self.assertIn("cannot write", err.getvalue())
+ self.assertEqual(self.calls, [])
+
+ def test_help_finds_record_from_the_main_command(self):
+ for argv, expected in ((["gputaskman", "--help"], "gputaskman record -o FILE"),
+ (["gputaskman", "record", "--help"], "--duration")):
+ out = io.StringIO()
+ with patch("sys.argv", argv), contextlib.redirect_stdout(out), self.assertRaises(SystemExit):
+ main()
+ self.assertIn(expected, out.getvalue())
+
+
+if __name__ == "__main__":
+ unittest.main()