A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
Add a record command that appends one JSON snapshot per interval
Scott Hanselmane2d7ddb80439
diff --git a/AGENTS.md b/AGENTS.mdindex 352e9dc..41e73ce 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -60,6 +60,7 @@ cli.py ──► Collector.sample() (collector.py)──► Snapshot ──► app.py (Textual TUI) / report.py (text and JSON)cli.py ──► HostSampler.sample() (host.py: system RAM/CPU, OS-reported; sampled beside the Collector, not in Snapshot)cli.py diff ──► diff.py: compares two saved --json snapshots (no collection)+cli.py record ──► record.py: appends one --json snapshot per interval (Collector, or LiveCollectors on Windows, sampled as the TUI does)```The backend is chosen at import time in `collector.py`: `BACKEND` is `wddm` on Windows, `nvml` on@@ -83,6 +84,7 @@ identify everything immediately; the TUI spreads scans across refreshes.| Change snapshot comparison | `diff.py` | Works only on saved `--json` files, never live state. Compare measured fields directly; keep runtime-reported and estimated values in their own sections; an absent row is `not listed`, not zero. Refuse only when both snapshots name conflicting adapters (vendor, PCI address, a non-Windows key); when they cannot be compared, warn (`name-only`) instead, because a Windows LUID changes at boot and `diff` has no override flag. A PID whose two `started_at` values are both valid and more than `SAME_PROCESS_SECS` apart is a different process: no pairing rule may join it (unknown or malformed start times never conflict, so older snapshots pair as before) || Change system RAM/CPU | `host.py` | Machine memory and load as the OS reports them: never GPU bytes, never added to `Snapshot.attributed`, and `diff` ignores it. Unknown is `None`/`null` and a failure carries a reason, never zeros. The JSON key is `system` because `host` already means the WSL Windows host. Windows uses the NT process list (psutil per process takes about 6 s per pass for 1,000 processes), checked against psutil at startup; CPU rates are keyed by `(pid, create_time)` so a reused PID never inherits another process's history (on the psutil path, `is_running()` is what makes psutil drop its cached object for a reused PID; keep that call) || Change WSL matching | `wslhost.py` `ContextTracker` | Keep `tests/test_wslhost.py` replays passing; they are measured sequences |+| Change recording | `record.py`, `cli.py` `record_main` | Every line must equal the `--json` object (`report.to_json_dict`, which `dump()` also uses), so unknown stays `null` and nothing is added or summed. Append only, one compact line per sample, flushed as it is written: Ctrl+C or a kill keeps every finished sample, and an interrupted sample writes nothing. Sample the way the TUI does (budgeted scanning, `LiveCollectors` on Windows so a GPU disable/re-enable is recorded as unavailable with a reason); never add a probe, `scan_budget=None`, or `psutil.open_files()` |## Verifydiff --git a/README.md b/README.mdindex b1f3a92..e2f1083 100644--- a/README.md+++ b/README.md@@ -432,6 +432,7 @@ uv run --locked gputaskman -i 2 # sample every two secondsuv run --locked gputaskman --once # one text snapshotuv run --locked gputaskman --json # one JSON snapshot; implies --onceuv run --locked gputaskman diff before.json after.json # compare two --json snapshots+uv run --locked gputaskman record -o session.jsonl --duration 300 # append one --json snapshot per second for 5 minutesuv run --locked gputaskman --no-host # WSL: don't read Windows host counters through interopuv run --locked gputaskman --no-system # don't sample system RAM/CPU: no panel, columns, or JSON sectionuv run --locked gputaskman --help@@ -632,6 +633,45 @@ estimated +504 MB of KV cache, but the server's measured dedicated memory grew bGemma 3 uses sliding-window attention, so the estimate is an upper bound. The comparison makesthat gap visible instead of hiding it.+### Recording a session++To keep what a long run did to the GPU (a training job, an overnight server, a game session),+`record` appends one `--json` snapshot per interval to a file:++```powershell+uv run --locked gputaskman record -o session.jsonl # until Ctrl+C+uv run --locked gputaskman record --gpu 0 --interval 5 --duration 300 -o session.jsonl+```++Each line is one complete `--json` snapshot: the same fields, the same `null` for anything that could+not be measured, the same measured, inferred, and estimated labels. `taken` is the sample time in Unix+seconds. The file is only ever appended to, so recording again continues it (a last line cut short by+a crash is closed first). Every line is flushed to the operating system as it is written, so Ctrl+C+or killing the process keeps every sample taken so far; a power loss can still lose the last few+seconds, and a reader should skip a final line that does not parse. A full disk stops the recording+with an error and leaves the lines already written.++Recording samples the way the live TUI does, with no extra probes. Scanning is budgeted per sample, so+model details can appear a sample or two after a process starts; on WSL the first sample comes a few+seconds late while the Windows host counters warm up. On Windows, a GPU that is disabled and re-enabled+stays in the recording: the samples in between carry `null` measurements and the reason, then resume+when the adapter returns. The first version records one GPU (`--gpu`, default 0).++A snapshot of 52 processes is about 27 KB, so one second between samples is roughly 100 MB an hour; use+`--interval 5` or more for long runs. The lines carry the same paths, command-line flags, model names, and+local ports as `--json` (see [Privacy](#privacy-and-troubleshooting)), so review a recording before you share it.++Any line of the file is a normal snapshot, so two of them can go to `diff`:++```powershell+Get-Content session.jsonl -TotalCount 1 | Set-Content first.json+Get-Content session.jsonl -Tail 1 | Set-Content last.json+uv run --locked gputaskman diff first.json last.json+```++On Linux, WSL, and macOS use `head -n 1 session.jsonl > first.json` and `tail -n 1 session.jsonl > last.json`.+Replaying or summarizing a whole recording is not built yet.+## How can it know what's inside?It combines evidence rather than reading arbitrary GPU memory. Windows' graphics kernel supplies@@ -779,6 +819,7 @@ Identification is designed to stay cheap. Measured on one Windows machine:| macOS: memory-map scan (`proc_pidinfo` region walk) | ~4 ms per process | Same policy as the memory-map scan above || macOS: `footprint` GPU memory | ~70 ms per process, background thread | Busy and new processes every ~2 s, idle ones every 30 s (54 runs in 40 s with 27 GPU processes, M4 Mac mini) || System RAM/CPU: process list + psutil totals | Windows (`NtQuerySystemInformation`): ~28 ms median for 995 processes (Windows 11, RTX 4080 SUPER machine, 32 threads); the contributor measured ~6 ms for ~480 processes (Windows 11, i9-13900K). Linux under WSL2 (psutil): ~3.4 ms median for 44 processes, ~9.5 ms for the first sample of 123; the per-pid reuse check (`is_running()`) adds ~0.3 ms for 44 processes, about 7 µs each | Every refresh; `--no-system` turns it off. psutil's per-process `memory_info`/`cpu_times` took ~6 s per pass for 995 processes on Windows, so it is only the non-Windows path. macOS unmeasured |+| `record`: build and write one `--json` line | ~0.4 ms to build the object plus ~0.1 ms to serialize it, median of 30 for a 52-process, 27 KB line (Windows 11, RTX 4080 SUPER machine). A recorded sample is otherwise a normal refresh: 283 ms wall time for the first, ~90 ms from the fifth | Once per `--interval`, only while recording; no extra probes. Other platforms unmeasured |With two llama-servers running, a steady refresh costs roughly 15-30 ms of CPU (about 340 ms beforeprocess details were cached).@@ -934,6 +975,7 @@ downloads are bundled in this repository.| `collector.py` | Shared snapshot collection || `app.py` / `report.py` / `cli.py` | TUI, text/JSON, and command-line entry points || `diff.py` | Before/after comparison of two `--json` snapshots |+| `record.py` | `gputaskman record`: appends one `--json` snapshot per interval to a JSON Lines file |## Next@@ -946,6 +988,7 @@ downloads are bundled in this repository.`swapped`) as the paged column.- WSL: per-process GPU load for Linux rows, and Ollama's systemd journal as a log source.- Live throughput deltas, LM Studio validation, and per-model context accounting.+- Replay and summaries of a `gputaskman record` file; today any one line works with `diff`.- Opt-in ETW file-open tracing to name models loaded by read-and-close runtimes (any whisper or ONNXapp), correlated with the process's VRAM jump at load time.- Data-driven app adapters (settings path plus load/unload log patterns) once a second app needs one.diff --git a/src/gputaskman/cli.py b/src/gputaskman/cli.pyindex df2b606..666d034 100644--- a/src/gputaskman/cli.py+++ b/src/gputaskman/cli.py@@ -27,11 +27,68 @@ def diff_main(argv: list[str]) -> None:print(json.dumps(result, indent=2) if args.json else to_text(result))+def _check_platform() -> None:+ if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):+ sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu/Intel DRM) and macOS (Apple Silicon) only.")+++def _check_gpu(ap: argparse.ArgumentParser, index: int | None) -> None:+ if index:+ from .collector import enumerate_adapters, select_adapter++ try:+ select_adapter(enumerate_adapters(), index)+ except ValueError as error:+ ap.error(str(error))+++def record_main(argv: list[str]) -> None:+ ap = argparse.ArgumentParser(prog="gputaskman record",+ description="Append one --json snapshot per interval to a JSON Lines file, "+ "for example to keep what a long run did to the GPU.")+ ap.add_argument("-o", "--output", required=True, help="file to append to, one --json snapshot per line")+ ap.add_argument("--gpu", type=int, default=0, help="GPU to record (default 0; gputaskman --list-gpus shows the indexes)")+ ap.add_argument("-i", "--interval", type=float, default=1.0, help="seconds between samples (default 1)")+ ap.add_argument("--duration", type=float, help="stop after this many seconds (default: until Ctrl+C)")+ ap.add_argument("--no-host", action="store_true",+ help="WSL: don't read Windows host GPU counters through interop (WSL VM total, Windows apps)")+ ap.add_argument("--no-system", action="store_true", help="don't sample system RAM/CPU: no system section in the records")+ args = ap.parse_args(argv)+ if args.gpu < 0:+ ap.error("--gpu must be a non-negative index")+ if not math.isfinite(args.interval) or args.interval <= 0:+ ap.error("--interval must be a finite number greater than zero")+ if args.duration is not None and (not math.isfinite(args.duration) or args.duration <= 0):+ ap.error("--duration must be a finite number greater than zero")+ _check_platform()+ _check_gpu(ap, args.gpu)++ from .record import append_to, run, source++ try:+ out = append_to(args.output)+ except OSError as error:+ ap.error(f"cannot write {args.output}: {error}")+ count = 0+ try:+ with out, source(args.gpu, host=not args.no_host, system=not args.no_system) as sample:+ span = f" for {args.duration:g} s" if args.duration is not None else ""+ print(f"Recording GPU {args.gpu} every {args.interval:g} s{span} to {args.output}; "+ "press Ctrl+C to stop.", file=sys.stderr)+ count = run(sample, out, interval=args.interval, duration=args.duration)+ except KeyboardInterrupt:+ pass # Ctrl+C while starting up: nothing was recorded yet+ print(f"Recorded {count} sample(s) to {args.output}", file=sys.stderr)++def main() -> None:if len(sys.argv) > 1 and sys.argv[1] == "diff":return diff_main(sys.argv[2:])+ if len(sys.argv) > 1 and sys.argv[1] == "record":+ return record_main(sys.argv[2:])ap = argparse.ArgumentParser(prog="gputaskman", description="What's inside your GPU: per-process VRAM, engines, and LLM weights/KV-cache breakdown.",- epilog="gputaskman diff BEFORE.json AFTER.json compares two --json snapshots.")+ epilog="gputaskman diff BEFORE.json AFTER.json compares two --json snapshots; "+ "gputaskman record -o FILE appends one --json snapshot per interval to FILE.")ap.add_argument("--once", action="store_true", help="print one snapshot and exit (no TUI)")ap.add_argument("--json", action="store_true", help="print one JSON snapshot and exit (implies --once)")ap.add_argument("--gpu", type=int, help="show only this GPU (TUI default: all; one-shot default: 0)")@@ -47,8 +104,7 @@ def main() -> None:if not math.isfinite(args.interval) or args.interval <= 0:ap.error("--interval must be a finite number greater than zero")- if sys.platform not in ("win32", "darwin") and not sys.platform.startswith("linux"):- sys.exit("gputaskman supports Windows (WDDM), Linux/WSL (NVML, or amdgpu/Intel DRM) and macOS (Apple Silicon) only.")+ _check_platform()if args.list_gpus:from .collector import enumerate_adapters, ordered_adapters@@ -57,13 +113,7 @@ def main() -> None:print(f"{i}: {adapter.name} ({adapter.vendor}, LUID 0x{adapter.luid & 0xffffffffffffffff:016x})")return- if args.gpu:- from .collector import enumerate_adapters, select_adapter-- try:- select_adapter(enumerate_adapters(), args.gpu)- except ValueError as error:- ap.error(str(error))+ _check_gpu(ap, args.gpu)if args.once or args.json:from .collector import Collectordiff --git a/src/gputaskman/record.py b/src/gputaskman/record.pynew file mode 100644index 0000000..c688a39--- /dev/null+++ b/src/gputaskman/record.py@@ -0,0 +1,107 @@+"""gputaskman record: append one --json snapshot per interval to a JSON Lines file."""++from __future__ import annotations++import json+import os+import time+from contextlib import ExitStack, contextmanager++from .report import to_json_dict+++def make_sample(collect, system=None):+ """A callable returning one record: exactly the object --json prints, so every saved line is a normal snapshot."""++ def sample() -> dict:+ snapshot = collect()+ host, error = None, None+ if system is not None:+ try:+ host = system()+ except Exception as exc: # system RAM/CPU is extra: its failure must not cost the GPU record+ error = f"{type(exc).__name__}: {exc}"+ return to_json_dict(snapshot, host, error)++ return sample+++def run(sample, out, *, interval: float, duration: float | None = None,+ clock=time.monotonic, sleep=time.sleep) -> int:+ """Write one compact JSON line per interval, flushed as it goes; return how many were written.++ Stops after duration seconds, or on Ctrl+C. An interrupted sample writes nothing, so every line is complete.+ A slow sample delays the next one instead of triggering a burst to catch up.+ """+ count = 0+ start = due = clock()+ try:+ while duration is None or due - start < duration:+ if (wait := due - clock()) > 0:+ sleep(wait)+ out.write(json.dumps(sample(), default=str, separators=(",", ":")) + "\n")+ out.flush()+ count += 1+ due = max(due + interval, clock())+ except KeyboardInterrupt:+ pass+ return count+++def append_to(path: str):+ """Open path for appending (never overwriting); a record never glues onto a last line that was cut short."""+ needs_newline = False+ try:+ with open(path, "rb") as existing:+ existing.seek(-1, os.SEEK_END)+ needs_newline = existing.read(1) != b"\n"+ except OSError:+ pass # missing, empty, or not seekable: nothing to protect+ out = open(path, "a", encoding="utf-8", newline="\n")+ if needs_newline:+ out.write("\n")+ return out+++@contextmanager+def source(gpu: int, *, host: bool = True, system: bool = True):+ """Sample the way the TUI does and yield a sample() callable.++ Scanning is budgeted like the TUI's, and on Windows a GPU that is disabled and re-enabled is rediscovered:+ the records in between carry null measurements and the reason, never zeros.+ """+ from .collector import BACKEND, Collector, enumerate_adapters+ from .host import HostSampler++ sampler = None++ def read_system():+ nonlocal sampler+ if sampler is None:+ sampler = HostSampler()+ return sampler.sample()++ with ExitStack() as stack:+ if BACKEND == "wddm":+ from .recovery import LiveCollectors++ live = LiveCollectors(gpu, collector_factory=Collector, discover=enumerate_adapters)+ stack.callback(live.close)++ def collect():+ return live.sample()[0]+ else:+ collector = Collector(gpu, host=host)+ stack.callback(collector.close)+ if collector.host is not None:+ collector.host.wait(15.0) # WSL: the first Windows counter sample takes several seconds+ if hasattr(collector.counters, "warm_up"):+ collector.counters.warm_up(15.0) # macOS: per-process GPU memory comes from a background pass+ collect = collector.sample+ if system:+ try:+ read_system() # per-process CPU% is a rate too; this primes it+ except Exception:+ pass # each record carries the reason if it still fails+ time.sleep(1.0) # engine utilization is a rate; needs two samples+ yield make_sample(collect, read_system if system else None)diff --git a/src/gputaskman/report.py b/src/gputaskman/report.pyindex ac6c8c3..5bb75ce 100644--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@ -273,17 +273,24 @@ def to_text(s: Snapshot, limit: int = 30) -> str:return "\n".join(out)+def to_json_dict(s: Snapshot, system=None, system_error: str | None = None) -> dict:+ """The --json object: to_dict plus the optional OS-reported "system" section (or why it is missing)."""+ from . import host as hostmod++ d = to_dict(s)+ if system is not None:+ d["system"] = hostmod.to_dict(system)+ elif system_error:+ d["system"] = {"error": system_error}+ return d++def dump(s: Snapshot, as_json: bool, system=None, system_error: str | None = None) -> str:"""system: an optional host.HostSample (OS-reported system RAM/CPU); system_error: why it is missing."""from . import host as hostmodif as_json:- d = to_dict(s)- if system is not None:- d["system"] = hostmod.to_dict(system)- elif system_error:- d["system"] = {"error": system_error}- return json.dumps(d, indent=2, default=str)+ return json.dumps(to_json_dict(s, system, system_error), indent=2, default=str)text = to_text(s)if system is not None:return text + "\n\n" + hostmod.to_text(system)diff --git a/tests/test_record.py b/tests/test_record.pynew file mode 100644index 0000000..bf20ad6--- /dev/null+++ b/tests/test_record.py@@ -0,0 +1,513 @@+"""gputaskman record: one --json snapshot per interval, appended to a JSON Lines file.++Hardware-independent: samples are fakes, time is a fake clock and the collectors are mocks. A recording is the+same data as --json, so unknown stays null (never 0) and every line must be one complete JSON object.+"""++from __future__ import annotations++import contextlib+import errno+import io+import json+import os+import tempfile+import types+import unittest+from unittest.mock import MagicMock, patch++from gputaskman import collector as collector_module+from gputaskman import record+from gputaskman.cli import main+from gputaskman.collector import ProcRow, Snapshot+from gputaskman.diff import diff, load+from gputaskman.dxgi import Adapter+from gputaskman.host import HostProc, HostSample+from gputaskman.host import to_dict as host_to_dict+from gputaskman.pdh import ProcGpu+from gputaskman.procinfo import ProcDetail+from gputaskman.report import dump, to_dict, to_json_dict++GiB = 1 << 30+++def row(pid: int, name: str, kind: str = "other", dedicated: int = GiB) -> ProcRow:+ return ProcRow(pid, name, kind, ProcGpu(dedicated=dedicated), ProcDetail(pid, name))+++def snapshot(taken: float = 1.0, used: int = 10 * GiB) -> Snapshot:+ return Snapshot(taken=taken, adapter=None, adapters=[], nv=None, driver="", cuda="",+ vram_total=16 * GiB, vram_used=used, shared_used=GiB,+ procs=[row(10, "llama-server.exe", "LLM", 8 * GiB), row(20, "dwm.exe", "compositor", 2 * GiB)])+++def host_sample() -> HostSample:+ return HostSample(cpu_pct=37.0, per_core=[10.0, 90.0], logical=2, physical=2, ram_total=32 * GiB,+ ram_used=12 * GiB, ram_available=20 * GiB, swap_total=4 * GiB, swap_used=GiB,+ procs=[HostProc(10, "llama-server.exe", 6 * GiB, 5 * GiB, 12.0)])+++class Clock:+ """Time only moves when something sleeps or a sample takes time."""++ def __init__(self):+ self.now = 1000.0+ self.sleeps = []++ def __call__(self) -> float:+ return self.now++ def sleep(self, seconds: float) -> None:+ self.sleeps.append(seconds)+ self.now += seconds+++class Sink:+ """A file that keeps what was flushed apart from what is only buffered."""++ def __init__(self):+ self.pending = self.flushed = ""++ def write(self, text: str) -> None:+ self.pending += text++ def flush(self) -> None:+ self.flushed, self.pending = self.flushed + self.pending, ""++ def lines(self) -> list[str]:+ return self.flushed.splitlines()+++class RunTests(unittest.TestCase):+ def setUp(self):+ self.clock = Clock()+ self.sink = Sink()++ def run_for(self, sample, **kwargs) -> int:+ return record.run(sample, self.sink, clock=self.clock, sleep=self.clock.sleep, **kwargs)++ def test_one_compact_json_line_per_sample_flushed_as_it_goes(self):+ seen = []++ def sample():+ seen.append(len(self.sink.lines())) # lines already flushed when this sample is taken+ return {"taken": len(seen), "vram_used": None}++ self.assertEqual(self.run_for(sample, interval=1.0, duration=3.0), 3)+ self.assertEqual(seen, [0, 1, 2])+ self.assertEqual(self.sink.pending, "")+ self.assertEqual(self.sink.lines(), ['{"taken":1,"vram_used":null}', '{"taken":2,"vram_used":null}',+ '{"taken":3,"vram_used":null}'])++ def test_a_duration_sets_the_sample_count_and_nothing_waits_after_the_last_one(self):+ self.assertEqual(self.run_for(lambda: {}, interval=1.0, duration=300.0), 300)+ self.assertEqual(len(self.clock.sleeps), 299)+ self.assertEqual(self.clock.now - 1000.0, 299.0)++ def test_a_duration_shorter_than_the_interval_records_one_sample(self):+ self.assertEqual(self.run_for(lambda: {}, interval=5.0, duration=1.0), 1)+ self.assertEqual(self.clock.sleeps, [])++ def test_samples_keep_their_schedule_however_long_each_one_takes(self):+ starts = []++ def sample():+ starts.append(self.clock.now - 1000.0)+ self.clock.now += 0.25 # collecting takes time too+ return {}++ self.run_for(sample, interval=1.0, duration=4.0)+ self.assertEqual(starts, [0.0, 1.0, 2.0, 3.0]) # not 0, 1.25, 2.5 ...++ def test_a_slow_sample_delays_the_next_one_with_no_burst_to_catch_up(self):+ starts, took = [], iter([0.0, 3.5, 0.0, 0.0])++ def sample():+ starts.append(self.clock.now - 1000.0)+ self.clock.now += next(took)+ return {}++ self.assertEqual(self.run_for(sample, interval=1.0, duration=6.0), 4)+ self.assertEqual(starts, [0.0, 1.0, 4.5, 5.5])++ def test_ctrl_c_while_waiting_leaves_every_line_complete(self):+ def interrupt(seconds):+ raise KeyboardInterrupt++ self.assertEqual(record.run(lambda: {"n": 1}, self.sink, interval=1.0, clock=self.clock, sleep=interrupt), 1)+ self.assertEqual(self.sink.flushed, '{"n":1}\n')++ def test_ctrl_c_during_a_sample_writes_nothing_for_it(self):+ calls = []++ def sample():+ calls.append(1)+ if len(calls) == 3:+ raise KeyboardInterrupt+ return {"n": len(calls)}++ self.assertEqual(self.run_for(sample, interval=1.0), 2) # with no duration only Ctrl+C ends it+ self.assertEqual(self.sink.flushed, '{"n":1}\n{"n":2}\n')+ self.assertEqual(self.sink.pending, "")++ def test_each_record_is_one_ascii_line_even_with_odd_names_and_newlines(self):+ data = {"name": "mödel-日本.exe", "cmd": "a\nb\r\nc", "path": "C:\\Users\\x", "unknown": None}+ self.run_for(lambda: data, interval=1.0, duration=0.5)+ self.assertTrue(self.sink.flushed.isascii())+ self.assertEqual(self.sink.flushed.count("\n"), 1)+ self.assertEqual(json.loads(self.sink.flushed), data)++ def test_a_value_json_cannot_encode_is_written_as_its_text_not_a_traceback(self):+ class Odd:+ def __str__(self):+ return "odd value"++ self.run_for(lambda: {"n": 1, "odd": Odd()}, interval=1.0, duration=0.5)+ self.assertEqual(json.loads(self.sink.flushed), {"n": 1, "odd": "odd value"})++ def test_a_full_disk_stops_the_recording_with_its_error_and_keeps_the_earlier_lines(self):+ class FullDisk(Sink):+ def write(self, text: str) -> None:+ if len(self.lines()) == 2:+ raise OSError(errno.ENOSPC, "No space left on device")+ super().write(text)++ sink = FullDisk()+ with self.assertRaises(OSError) as stop:+ record.run(lambda: {"n": 1}, sink, interval=1.0, clock=self.clock, sleep=self.clock.sleep)+ self.assertEqual(stop.exception.errno, errno.ENOSPC)+ self.assertEqual(sink.lines(), ['{"n":1}', '{"n":1}'])+++class MakeSampleTests(unittest.TestCase):+ def test_a_record_is_the_object_json_prints(self):+ sample = record.make_sample(snapshot, host_sample)+ self.assertEqual(json.loads(json.dumps(sample(), default=str)),+ json.loads(dump(snapshot(), True, host_sample())))++ def test_system_figures_are_optional_and_their_failure_is_recorded_not_fatal(self):+ def broken():+ raise OSError("layout not recognised")++ failed = record.make_sample(snapshot, broken)()+ self.assertEqual(failed["system"], {"error": "OSError: layout not recognised"})+ self.assertEqual(failed["processes"], to_dict(snapshot())["processes"]) # the GPU part is intact+ self.assertNotIn("system", record.make_sample(snapshot)())++ def test_a_gpu_that_is_gone_is_recorded_as_unavailable_not_zero(self):+ from gputaskman.recovery import LiveCollectors++ live = LiveCollectors(0, collector_factory=MagicMock(), discover=lambda strict=True: [])+ data = record.make_sample(lambda: live.sample()[0])()+ self.assertIn("Adapter unavailable", data["counter_status"])+ self.assertIs(data["counter_adapter_matched"], False)+ for key in ("vram_total", "vram_used", "shared_used", "attributed", "device"):+ self.assertIsNone(data[key], key)+ self.assertEqual(data["processes"], [])+++class JsonDumpTests(unittest.TestCase):+ def test_dump_json_is_still_the_indented_object_with_its_optional_system_section(self):+ s, h = snapshot(), host_sample()+ self.assertEqual(dump(s, True), json.dumps(to_dict(s), indent=2, default=str))+ self.assertEqual(dump(s, True, h), json.dumps({**to_dict(s), "system": host_to_dict(h)}, indent=2, default=str))+ self.assertEqual(dump(s, True, None, "OSError: boom"),+ json.dumps({**to_dict(s), "system": {"error": "OSError: boom"}}, indent=2, default=str))+ self.assertEqual(to_json_dict(s, h), {**to_dict(s), "system": host_to_dict(h)})+ self.assertNotIn("system", to_json_dict(s))+++class FileTestCase(unittest.TestCase):+ def setUp(self):+ self.dir = tempfile.TemporaryDirectory()+ self.addCleanup(self.dir.cleanup)+ self.path = os.path.join(self.dir.name, "session.jsonl")++ def content(self) -> bytes:+ with open(self.path, "rb") as f:+ return f.read()+++class AppendTests(FileTestCase):+ def append(self, text: str) -> None:+ with record.append_to(self.path) as out:+ out.write(text)++ def test_creates_the_file_and_appends_without_touching_what_is_there(self):+ self.append('{"n":1}\n')+ self.append('{"n":2}\n')+ self.assertEqual(self.content(), b'{"n":1}\n{"n":2}\n')++ def test_lines_end_with_a_bare_newline_on_every_platform(self):+ self.append('{"n":1}\n')+ self.assertNotIn(b"\r", self.content())++ def test_a_last_line_cut_short_by_a_crash_is_closed_before_the_next_record(self):+ with open(self.path, "wb") as f:+ f.write(b'{"n":1}\n{"n":2')+ self.append('{"n":3}\n')+ self.assertEqual(self.content(), b'{"n":1}\n{"n":2\n{"n":3}\n')++ def test_an_empty_file_gets_no_stray_newline(self):+ open(self.path, "wb").close()+ self.append('{"n":1}\n')+ self.assertEqual(self.content(), b'{"n":1}\n')++ def test_a_path_that_cannot_be_written_raises_for_the_cli_to_report(self):+ with self.assertRaises(OSError):+ record.append_to(self.dir.name) # a directory++ def test_a_recording_written_to_disk_is_valid_json_lines_that_diff_accepts(self):+ snapshots = iter([snapshot(taken=1.0, used=4 * GiB), snapshot(taken=2.0, used=6 * GiB),+ snapshot(taken=3.0, used=9 * GiB)])+ clock = Clock()+ with record.append_to(self.path) as out:+ count = record.run(record.make_sample(lambda: next(snapshots), host_sample), out,+ interval=1.0, duration=3.0, clock=clock, sleep=clock.sleep)+ self.assertEqual(count, 3)+ with open(self.path, encoding="utf-8") as f:+ records = [json.loads(line) for line in f]+ self.assertEqual([r["taken"] for r in records], [1.0, 2.0, 3.0])+ self.assertTrue(all(r["system"]["ram"]["total"] == 32 * GiB for r in records))+ first, last = (os.path.join(self.dir.name, name) for name in ("first.json", "last.json"))+ for target, line in ((first, records[0]), (last, records[-1])):+ with open(target, "w", encoding="utf-8") as f:+ f.write(json.dumps(line)) # one line saved to its own file is a normal --json snapshot+ self.assertEqual(diff(load(first), load(last))["device"]["vram_used"]["delta"], 5 * GiB)+++class SourceTests(unittest.TestCase):+ def setUp(self):+ self.sleep = patch("gputaskman.record.time.sleep").start()+ self.sampler = patch("gputaskman.host.HostSampler").start()+ self.addCleanup(patch.stopall)+ self.sampler.return_value.sample.return_value = host_sample()++ def collector(self) -> MagicMock:+ c = MagicMock()+ c.host = None+ c.counters = types.SimpleNamespace() # no warm_up, like Windows and native Linux+ c.sample.return_value = snapshot()+ return c++ def test_native_linux_samples_one_collector_with_budgeted_scanning_and_closes_it(self):+ c = self.collector()+ with patch("gputaskman.collector.BACKEND", "nvml"), \+ patch("gputaskman.collector.Collector", return_value=c) as factory:+ with record.source(2) as sample:+ data = sample()+ factory.assert_called_once_with(2, host=True) # the TUI's budget, not scan_budget=None+ c.close.assert_called_once_with()+ self.sleep.assert_called_once_with(1.0) # engine utilization is a rate+ self.assertEqual(data["taken"], 1.0)+ self.assertEqual(self.sampler.return_value.sample.call_count, 2) # the first reading only primes CPU%+ self.assertEqual(data["system"]["ram"]["total"], 32 * GiB)++ def test_no_host_is_passed_on(self):+ with patch("gputaskman.collector.BACKEND", "wsl"), \+ patch("gputaskman.collector.Collector", return_value=self.collector()) as factory:+ with record.source(0, host=False):+ pass+ factory.assert_called_once_with(0, host=False)++ def test_wsl_host_counters_and_macos_footprints_get_their_warm_up(self):+ c = self.collector()+ c.host, c.counters = MagicMock(), MagicMock()+ with patch("gputaskman.collector.BACKEND", "wsl"), patch("gputaskman.collector.Collector", return_value=c):+ with record.source(0):+ pass+ c.host.wait.assert_called_once_with(15.0)+ c.counters.warm_up.assert_called_once_with(15.0)++ def test_windows_rediscovers_a_gpu_that_was_disabled_and_records_the_selected_slot(self):+ with patch("gputaskman.collector.BACKEND", "wddm"), patch("gputaskman.recovery.LiveCollectors") as live:+ live.return_value.sample.return_value = [snapshot(taken=1.0), snapshot(taken=2.0)]+ with record.source(1, system=False) as sample:+ data = sample()+ args, kwargs = live.call_args+ self.assertEqual(args, (1,))+ self.assertIs(kwargs["collector_factory"], collector_module.Collector)+ self.assertIs(kwargs["discover"], collector_module.enumerate_adapters)+ live.return_value.close.assert_called_once_with()+ self.assertEqual(data["taken"], 1.0)+ self.assertNotIn("system", data)+ self.sampler.assert_not_called()++ def test_collectors_are_closed_even_when_recording_fails(self):+ c = self.collector()+ with patch("gputaskman.collector.BACKEND", "nvml"), patch("gputaskman.collector.Collector", return_value=c):+ with self.assertRaises(RuntimeError), record.source(0):+ raise RuntimeError("disk full")+ c.close.assert_called_once_with()++ def test_a_broken_system_sampler_never_stops_the_gpu_records(self):+ self.sampler.side_effect = OSError("layout not recognised")+ with patch("gputaskman.collector.BACKEND", "nvml"), \+ patch("gputaskman.collector.Collector", return_value=self.collector()):+ with record.source(0) as sample:+ data = sample()+ self.assertEqual(data["system"], {"error": "OSError: layout not recognised"})+ self.assertEqual(len(data["processes"]), 2)++ def test_one_system_sampler_serves_every_record_because_cpu_percent_is_a_rate_between_readings(self):+ with patch("gputaskman.collector.BACKEND", "nvml"), \+ patch("gputaskman.collector.Collector", return_value=self.collector()):+ with record.source(0) as sample:+ sample()+ sample()+ self.sampler.assert_called_once_with()+ self.assertEqual(self.sampler.return_value.sample.call_count, 3) # one priming read, then one per record++ def test_no_system_never_builds_a_sampler_and_leaves_no_section(self):+ with patch("gputaskman.collector.BACKEND", "nvml"), \+ patch("gputaskman.collector.Collector", return_value=self.collector()):+ with record.source(0, system=False) as sample:+ data = sample()+ self.sampler.assert_not_called()+ self.assertNotIn("system", data)+++class CliTests(FileTestCase):+ def setUp(self):+ super().setUp()+ self.calls = []+ self.samples = 0+ self.interrupt_at = None++ def fake_source(self, fail=None):+ calls = self.calls++ @contextlib.contextmanager+ def source(gpu, *, host, system):+ calls.append((gpu, host, system))+ if fail is not None:+ raise fail++ def sample():+ self.samples += 1+ if self.samples == self.interrupt_at:+ raise KeyboardInterrupt+ if self.samples > 5: # a recording that ignores its --duration must fail the test, not hang it+ raise AssertionError("the recording ran on past what the test asked for")+ return to_json_dict(snapshot(taken=float(self.samples)))++ yield sample++ return patch("gputaskman.record.source", source)++ def refused(self):+ """A source that fails the test at once: a recording must never start for input the command refuses."""+ return self.fake_source(fail=AssertionError("a recording started for input that must be refused"))++ def run_cli(self, *args, fail=None) -> str:+ err = io.StringIO()+ with self.fake_source(fail), patch("sys.argv", ["gputaskman", "record", *args]), \+ contextlib.redirect_stderr(err):+ main()+ return err.getvalue()++ def lines(self) -> list[str]:+ return self.content().decode("utf-8").splitlines()++ def test_records_a_snapshot_and_says_so_on_stderr(self):+ err = self.run_cli("-o", self.path, "--duration", "0.5")+ self.assertEqual(len(self.lines()), 1)+ self.assertEqual(json.loads(self.lines()[0]), json.loads(dump(snapshot(taken=1.0), True)))+ self.assertEqual(self.calls, [(0, True, True)])+ self.assertIn("Recording GPU 0 every 1 s for 0.5 s", err)+ self.assertIn("Recorded 1 sample(s)", err)++ def test_the_flags_reach_the_sampling_side(self):+ self.run_cli("--output", self.path, "--gpu", "0", "--interval", "2", "--duration", "1", "--no-host", "--no-system")+ self.assertEqual(self.calls, [(0, False, False)])++ def test_every_value_reaches_the_side_that_uses_it(self):+ adapters = [Adapter(1, "GPU 0", 0x10DE, GiB, 0, False), Adapter(2, "GPU 1", 0x10DE, GiB, 0, False)]+ for name, args, source_args, run_kwargs in (+ ("given", ["--gpu", "1", "--interval", "2", "--duration", "1", "--no-host"],+ (1, False, True), {"interval": 2.0, "duration": 1.0}),+ ("defaults", [], (0, True, True), {"interval": 1.0, "duration": None})):+ with self.subTest(name):+ self.calls.clear()+ with patch("gputaskman.collector.enumerate_adapters", return_value=adapters), \+ patch("gputaskman.record.run", return_value=1) as run:+ self.run_cli("-o", self.path, *args)+ self.assertEqual(self.calls, [source_args])+ self.assertEqual(run.call_args.kwargs, run_kwargs)++ def test_a_second_recording_appends_and_a_cut_last_line_is_closed_first(self):+ with open(self.path, "wb") as f:+ f.write(b'{"taken":0.5}\n{"taken"')+ self.run_cli("-o", self.path, "--duration", "0.5")+ self.run_cli("-o", self.path, "--duration", "0.5")+ lines = self.lines()+ self.assertEqual(lines[:2], ['{"taken":0.5}', '{"taken"'])+ self.assertEqual([json.loads(line)["taken"] for line in lines[2:]], [1.0, 2.0]) # no blank lines in between++ def test_ctrl_c_ends_a_recording_cleanly_and_keeps_what_was_written(self):+ self.interrupt_at = 3+ err = self.run_cli("-o", self.path, "-i", "0.01")+ self.assertEqual([json.loads(line)["taken"] for line in self.lines()], [1.0, 2.0])+ self.assertIn("Recorded 2 sample(s)", err)++ def test_ctrl_c_while_starting_up_exits_quietly(self):+ err = self.run_cli("-o", self.path, fail=KeyboardInterrupt())+ self.assertIn("Recorded 0 sample(s)", err)+ self.assertEqual(self.content(), b"")++ def test_bad_arguments_are_refused_before_anything_is_opened(self):+ cases = {+ "--interval 0": ["-o", self.path, "--interval", "0"],+ "--interval -1": ["-o", self.path, "--interval", "-1"],+ "--interval nan": ["-o", self.path, "--interval", "nan"],+ "--interval inf": ["-o", self.path, "--interval", "inf"],+ "--duration 0": ["-o", self.path, "--duration", "0"],+ "--duration -1": ["-o", self.path, "--duration", "-1"],+ "--duration nan": ["-o", self.path, "--duration", "nan"],+ "--duration inf": ["-o", self.path, "--duration", "inf"],+ "--gpu -1": ["-o", self.path, "--gpu", "-1"],+ "no output": [],+ }+ for name, args in cases.items():+ with self.subTest(name):+ err = io.StringIO()+ with self.refused(), patch("sys.argv", ["gputaskman", "record", *args]), \+ patch("gputaskman.collector.enumerate_adapters",+ side_effect=AssertionError("no GPU may be touched while refusing the arguments")), \+ contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as stop:+ main()+ self.assertEqual(stop.exception.code, 2)+ self.assertEqual(self.calls, [])+ self.assertFalse(os.path.exists(self.path))++ def test_a_gpu_that_does_not_exist_is_refused_up_front(self):+ err = io.StringIO()+ with self.refused(), patch("gputaskman.collector.enumerate_adapters", return_value=[]), \+ patch("sys.argv", ["gputaskman", "record", "-o", self.path, "--gpu", "2"]), \+ contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as stop:+ main()+ self.assertEqual(stop.exception.code, 2)+ self.assertIn("--gpu 2 is out of range", err.getvalue())+ self.assertEqual(self.calls, [])+ self.assertFalse(os.path.exists(self.path))++ def test_an_output_that_cannot_be_written_is_refused(self):+ err = io.StringIO()+ with self.refused(), patch("sys.argv", ["gputaskman", "record", "-o", self.dir.name]), \+ contextlib.redirect_stderr(err), self.assertRaises(SystemExit) as stop:+ main()+ self.assertEqual(stop.exception.code, 2)+ self.assertIn("cannot write", err.getvalue())+ self.assertEqual(self.calls, [])++ def test_help_finds_record_from_the_main_command(self):+ for argv, expected in ((["gputaskman", "--help"], "gputaskman record -o FILE"),+ (["gputaskman", "record", "--help"], "--duration")):+ out = io.StringIO()+ with patch("sys.argv", argv), contextlib.redirect_stdout(out), self.assertRaises(SystemExit):+ main()+ self.assertIn(expected, out.getvalue())+++if __name__ == "__main__":+ unittest.main()