A GPU task manager for Windows, Linux, and WSL: per-process VRAM, GPU activity, and LLM model/KV-cache insights (Windows AMD and NVIDIA).
Python
Merge main into Intel temperature PR
Scott Hanselman8136447f58df
diff --cc README.mdindex b07dd9a,256f5c4..fcff4f3--- a/README.md+++ b/README.md@@@ -245,11 -245,11 +245,11 @@@ covered by unit tests only and **not ye| | Source ||---|---|- | Device temperature, power, GT clock | `hwmon` and `gt/gt0` under `/sys/class/drm/card*`. Power is the rate of the cumulative energy counter between two refreshes, so it is `n/a` on the first. Without a GPU temperature sensor (an integrated GPU has no `hwmon` of its own), the temperature falls back to the CPU package sensor (`coretemp`), labelled as shared with the CPU |-| Device temperature, power, GT clock, throttle | `hwmon` and `gt/gt0` under `/sys/class/drm/card*`. Power is the rate of the cumulative energy counter between two refreshes, so it is `n/a` on the first. The GT clock is the actual frequency (`rps_act_freq_mhz`, the requested one as fallback); throttle reasons come from `throttle_reason_*` |++| Device temperature, power, GT clock, throttle | `hwmon` and `gt/gt0` under `/sys/class/drm/card*`. Power is the rate of the cumulative energy counter between two refreshes, so it is `n/a` on the first. The GT clock is the actual frequency (`rps_act_freq_mhz`, the requested one as fallback); throttle reasons come from `throttle_reason_*`. Without a GPU temperature sensor, an integrated GPU falls back to the CPU package sensor (`coretemp`), labelled as shared with the CPU || Device VRAM total | The size of the local-memory PCI window (`resource2`), an **estimate**: kernel-owned memory is missing from the sysfs view, resizable BAR rounds to a power of two, and a partial window can be smaller than the card, so the total can differ from the card's memory. A window under 1 GiB (an integrated GPU, or a discrete card without resizable BAR) shows `N/A` (`null` in JSON). Free memory is never shown for Intel |- | Device VRAM used | The sum of its clients' resident local memory: i915 has no device-wide counter, so buffers shared between processes can be counted twice, and kernel-owned memory and processes gputaskman can't inspect are missing. The UI, `--once`, `--json` (`memory_note`) and `gputaskman diff` all say so. `n/a` when any client's local memory is unknown |+ | Device VRAM used | The sum of its clients' resident memory: i915 has no device-wide counter, so buffers shared between processes can be counted twice, and kernel-owned memory and processes gputaskman can't inspect are missing. Local memory on a discrete card, system memory on an integrated GPU. The UI, `--once`, `--json` (`memory_note`) and `gputaskman diff` all say so. `n/a` when any client's memory is unknown || Integrated or discrete | An iGPU has no `local0`, so a card with a local-memory window of 1 GiB or more is discrete, and so is any card whose fdinfo reports local memory, even when its window is smaller |- | Per-process VRAM and SPILL | `drm-resident-local0` and `drm-resident-system0`, plus their `stolen-` regions (the firmware-reserved part of each), under the same private-client rules as AMD. `system0` is memory the GPU's buffers currently sit in, including buffers placed there on purpose, so SPILL here (like AMD's GTT) is where they are, not proof they were pushed out of VRAM. Both `drm-shared-local0` and `drm-shared-system0` must be present to charge a process, so an integrated GPU (no `local0`) shows `n/a` (`legacy`). A region value that can't be parsed makes that figure `n/a`, not a smaller number |+ | Per-process VRAM and SPILL | `drm-resident-local0` and `drm-resident-system0`, plus their `stolen-` regions (the firmware-reserved part of each), under the same private-client rules as AMD. On an integrated GPU there is no local memory: a provably private client's `system0` bytes are its GPU memory in the VRAM column, and SPILL stays `N/A` (system memory is its memory, not spill). On a discrete card `system0` is memory the GPU's buffers currently sit in, including buffers placed there on purpose, so SPILL there (like AMD's GTT) is where they are, not proof they were pushed out of VRAM. The shared gate needs only the regions the device has, so a discrete card with a missing `drm-shared-local0` still shows `n/a` (`legacy`), while an integrated client with shared buffers shows `n/a` (`shared_buffers`). A region value that can't be parsed makes that figure `n/a`, not a smaller number || Per-process engine load (3d, compute, copy, video) | `drm-engine-render`, `-compute`, `-copy`, `-video`, `-video-enhance` busy time, sampled twice. `n/a` where the driver reports none (before Gen 8). A class with several identical engines (`drm-engine-capacity-*`, such as video on an Arc) is not divided by its capacity, so one busy engine reads 100% |Device GPU load, memory bandwidth, and the PCIe link are `n/a`: i915 has no device-wide busy counter, and itsdiff --cc src/gputaskman/app.pyindex a394ef1,86157f9..5bf00d8--- a/src/gputaskman/app.py+++ b/src/gputaskman/app.py@@@ -581,14 -585,14 +585,15 @@@ class GpuTaskManager(App)if nv.enc or nv.dec:m.append(f" enc {format_metric(nv.enc, unit='%')} dec {format_metric(nv.dec, unit='%')}")lines.append(m)- clock_label = (f"gfx {format_metric(nv.gfx_clock)} MHz" if nv.source.startswith("AMD ") else+ clock_label = (f"gfx {format_metric(nv.gfx_clock)} MHz" if is_amd else+ f"GT {format_metric(nv.sm_clock)}/{format_metric(nv.sm_clock_max)} MHz" if is_intel elsef"SM {format_metric(nv.sm_clock)}/{format_metric(nv.sm_clock_max)} MHz")power_kind = f" {nv.power_kind}" if nv.power_kind else ""+ temp_kind = f" ({nv.temp_kind})" if nv.temp_kind else ""if self.monitoring:- clock = nv.gfx_clock if nv.source.startswith("AMD ") else nv.sm_clock- label = "gfx" if nv.source.startswith("AMD ") else "SM"+ clock = nv.gfx_clock if is_amd else nv.sm_clock+ label = "gfx" if is_amd else ("GT" if is_intel else "SM")- st = Text(f"{format_metric(nv.temp, unit='°C')} "+ st = Text(f"{format_metric(nv.temp, unit='°C')}{temp_kind} "f"{format_metric(nv.power_w, '.0f')} W{power_kind} "f"{label} {format_metric(clock)} MHz", style="grey70")else:diff --cc src/gputaskman/report.pyindex ef87afa,b2f67c1..a123e9f--- a/src/gputaskman/report.py+++ b/src/gputaskman/report.py@@@ -198,13 -199,14 +199,15 @@@ def to_text(s: Snapshot, limit: int = 3rx = nv.pcie_rx_kbs / 1024 if nv.pcie_rx_kbs is not None else Nonetx = nv.pcie_tx_kbs / 1024 if nv.pcie_tx_kbs is not None else Nonepstate = f"P{nv.pstate}" if nv.pstate is not None else "P N/A"- label = "AMD gfx" if nv.source.startswith("AMD ") else "GPU"+ is_intel = nv.source == "Intel sysfs"+ label = "AMD gfx" if nv.source.startswith("AMD ") else ("Intel GT" if is_intel else "GPU")clock = (f"gfx {format_metric(nv.gfx_clock)}" if nv.source.startswith("AMD ") else+ f"GT {format_metric(nv.sm_clock)}" if is_intel elsef"SM {format_metric(nv.sm_clock)}")power_kind = f" ({nv.power_kind})" if nv.power_kind else ""+ temp_label = f" ({nv.temp_kind})" if nv.temp_kind else ""out.append(f"{label} {format_metric(nv.util_gpu, unit='%')} mem-bandwidth {format_metric(nv.util_mem, unit='%')} "- f"{format_metric(nv.temp, unit='C')} {format_metric(nv.power_w, '.0f')}/"+ f"{format_metric(nv.temp, unit='C')}{temp_label} {format_metric(nv.power_w, '.0f')}/"f"{format_metric(nv.power_limit_w, '.0f')} W{power_kind} {pstate} "f"{clock} MHz "f"video {format_metric(nv.video_clock)} MHz "diff --cc tests/test_drm.pyindex 793e50e,b5e1a00..e9d72a9--- a/tests/test_drm.py+++ b/tests/test_drm.py@@@ -422,86 -447,72 +447,152 @@@ class ReadDeviceTests(unittest.TestCaseself.assertIsNone(d.enc)self.assertIsNone(d.dec)+ def test_cpu_package_temp_prefers_package_id_0(self):+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ nvme = root / "hwmon0"+ nvme.mkdir()+ (nvme / "name").write_text("nvme\n")+ cpu = root / "hwmon1"+ cpu.mkdir()+ (cpu / "name").write_text("coretemp\n")+ (cpu / "temp1_label").write_text("Package id 0\n")+ (cpu / "temp1_input").write_text("55000\n")+ (cpu / "temp2_label").write_text("Core 0\n")+ (cpu / "temp2_input").write_text("99000\n") # a core is not the package+ self.assertEqual(drm._cpu_package_temp(str(root)), 55)++ def test_cpu_package_temp_missing_is_unknown(self):+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ cpu = root / "hwmon0"+ cpu.mkdir()+ (cpu / "name").write_text("coretemp\n")+ (cpu / "temp1_label").write_text("Core 0\n") # no package sensor: no guessing from cores+ (cpu / "temp1_input").write_text("99000\n")+ self.assertIsNone(drm._cpu_package_temp(str(root)))+ self.assertIsNone(drm._cpu_package_temp(str(root / "no-such-dir")))++ def test_cpu_package_temp_skips_torn_readings(self):+ with tempfile.TemporaryDirectory() as tmp:+ root = Path(tmp)+ cpu = root / "hwmon0"+ cpu.mkdir()+ (cpu / "name").write_text("coretemp\n")+ (cpu / "temp1_label").write_text("Package id 0\n")+ (cpu / "temp1_input").write_text("lots\n") # torn: try the other sensors+ (cpu / "temp2_label").write_text("Package id 1\n")+ (cpu / "temp2_input").write_text("48000\n")+ self.assertEqual(drm._cpu_package_temp(str(root)), 48)++ def test_read_device_intel_falls_back_to_cpu_package_temp(self):+ with tempfile.TemporaryDirectory() as tmp:+ card_sysfs = Path(tmp) / "card1"+ dev = card_sysfs / "device"+ dev.mkdir(parents=True)+ card = drm.DrmCard(key=32, pdev="0000:00:02.0", sysfs=str(dev), driver="i915",+ vendor_id=0x8086, device_id=0xA7AD, name="Intel Intel Graphics",+ vram_total=0, integrated=True, intel=True)+ with patch.object(drm, "_cpu_package_temp", return_value=55):+ d = drm.read_device(card, 0)+ self.assertEqual((d.temp, d.temp_kind), (55, "CPU package"))++ def test_read_device_intel_clock_and_throttle(self):+ with tempfile.TemporaryDirectory() as tmp:+ card_sysfs = Path(tmp) / "card1"+ dev = card_sysfs / "device"+ dev.mkdir(parents=True)+ gt = card_sysfs / "gt" / "gt0"+ gt.mkdir(parents=True)+ (gt / "rps_act_freq_mhz").write_text("300\n")+ (gt / "rps_cur_freq_mhz").write_text("350\n")+ (gt / "rps_RP0_freq_mhz").write_text("1250\n")+ (gt / "throttle_reason_pl1").write_text("1\n")+ (gt / "throttle_reason_pl2").write_text("0\n")+ (gt / "throttle_reason_thermal").write_text("0\n")++ card = drm.DrmCard(key=32, pdev="0000:00:02.0", sysfs=str(dev), driver="i915",+ vendor_id=0x8086, device_id=0xA7AD, name="Intel Intel Graphics",+ vram_total=0, integrated=True, intel=True)+ d = drm.read_device(card, 0)++ self.assertEqual(d.sm_clock, 300) # the actual frequency, not the requested one+ self.assertEqual(d.gfx_clock, 350)+ self.assertEqual(d.sm_clock_max, 1250)+ self.assertEqual(d.throttle, ["PL1"])+ self.assertEqual(d.source, "Intel sysfs")++ def test_read_device_intel_clock_falls_back_to_requested(self):+ with tempfile.TemporaryDirectory() as tmp:+ card_sysfs = Path(tmp) / "card1"+ dev = card_sysfs / "device"+ dev.mkdir(parents=True)+ gt = card_sysfs / "gt" / "gt0"+ gt.mkdir(parents=True)+ (gt / "rps_cur_freq_mhz").write_text("300\n")+ (gt / "rps_RP0_freq_mhz").write_text("1250\n")++ card = drm.DrmCard(key=32, pdev="0000:00:02.0", sysfs=str(dev), driver="i915",+ vendor_id=0x8086, device_id=0xA7AD, name="Intel Intel Graphics",+ vram_total=0, integrated=True, intel=True)+ d = drm.read_device(card, 0)++ self.assertEqual(d.sm_clock, 300)+ # No throttle node is readable here: unknown, not "no throttling".+ self.assertIsNone(d.throttle)++ def test_read_device_intel_throttle_all_idle_is_empty_not_unknown(self):+ with tempfile.TemporaryDirectory() as tmp:+ card_sysfs = Path(tmp) / "card1"+ dev = card_sysfs / "device"+ dev.mkdir(parents=True)+ gt = card_sysfs / "gt" / "gt0"+ gt.mkdir(parents=True)+ (gt / "throttle_reason_pl1").write_text("0\n")+ (gt / "throttle_reason_thermal").write_text("0\n")++ card = drm.DrmCard(key=32, pdev="0000:00:02.0", sysfs=str(dev), driver="i915",+ vendor_id=0x8086, device_id=0xA7AD, name="Intel Intel Graphics",+ vram_total=0, integrated=True, intel=True)+ self.assertEqual(drm.read_device(card, 0).throttle, [])++ def test_intel_throttle_skips_unreadable_nodes(self):+ with tempfile.TemporaryDirectory() as tmp:+ gt = Path(tmp)+ (gt / "throttle_reason_pl2").write_text("1\n")+ (gt / "throttle_reason_thermal").write_text("lots\n") # unparseable: unknown, not active+ self.assertEqual(drm._intel_throttle(str(gt)), ["PL2"])++ def test_read_device_intel_gpu_sensor_wins_over_cpu_package(self):+ with tempfile.TemporaryDirectory() as tmp:+ card_sysfs = Path(tmp) / "card1"+ dev = card_sysfs / "device"+ dev.mkdir(parents=True)+ hwmon = dev / "hwmon" / "hwmon0"+ hwmon.mkdir(parents=True)+ (hwmon / "temp1_label").write_text("edge\n")+ (hwmon / "temp1_input").write_text("61000\n")+ card = drm.DrmCard(key=32, pdev="0000:00:02.0", sysfs=str(dev), driver="i915",+ vendor_id=0x8086, device_id=0xA7AD, name="Intel Intel Graphics",+ vram_total=0, integrated=True, intel=True)+ with patch.object(drm, "_cpu_package_temp", return_value=55):+ d = drm.read_device(card, 0)+ self.assertEqual((d.temp, d.temp_kind), (61, ""))++ def test_read_device_discrete_intel_without_sensor_stays_unknown(self):+ # The CPU package shares a die with an integrated GPU only; on a discrete card it is+ # not the GPU's temperature, so there is no fallback.+ with tempfile.TemporaryDirectory() as tmp:+ card_sysfs = Path(tmp) / "card1"+ dev = card_sysfs / "device"+ dev.mkdir(parents=True)+ card = drm.DrmCard(key=12288, pdev="0000:03:00.0", sysfs=str(dev), driver="i915",+ vendor_id=0x8086, device_id=0x56A0, name="Intel Arc A770",+ vram_total=16 * GiB, integrated=False, intel=True)+ with patch.object(drm, "_cpu_package_temp", return_value=55):+ d = drm.read_device(card, 0)+ self.assertEqual((d.temp, d.temp_kind), (None, ""))+class DrmCountersTests(unittest.TestCase):def test_drm_counters(self):