Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
bench(giant-moe): seq-scaling probe on the synthetic 1T step (NO VERDICT, gate V5) (#2163)
Alpamys Makazhan3ad7ec34ceca
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex d65a1612..9b808e6d 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -60,6 +60,8 @@ They are the evidence behind the preprint:| [`gate-stream-recompute.md`](gate-stream-recompute.md) | `training.stream_recompute`: streamed layers that keep their activations instead of the checkpoint recompute, on a streamed Mistral-7B NF4 run (RTX 5070 Laptop 8 GB, a new baseline) | **Measured on one laptop:** -22 % / -17 % / -8.5 % step time on the RAM tier at 512 / 1024 / 2048 tokens per step (32 / 24 / 12 of 32 layers fit), loss and 256 of 256 LoRA gradients bit-identical under deterministic algorithms; no gain on a read-bound disk tier. `soup train` with `auto` chose 32/32 layers at max_length 512 and 10/32 at 2048, train_loss equal to `on`. || [`gate-fp8-shard.md`](gate-fp8-shard.md) | Sharding a DeepSeek-V3 FP8 block-quantised checkpoint by dequantising it at shard time (CPU only): dequant exactness, streamed-from-FP8 vs resident parity on a tiny `DeepseekV3ForCausalLM`, and Moonlight-16B-A3B converted to the format and sharded | **G1, G2, G3 pass.** Bit-exact against an independent per-block reference (and transformers' own dequant where its block rule applies); logits, loss and LoRA gradients bit-exact in both expert layouts, caches tensor-for-tensor equal incl. NF4; the 27-layer NF4 cache of the FP8 copy came within 0.0002% (0.000115%) of the pre-flight estimate, and every checked FP8-derived tensor equals the reference. No GPU run || [`probe-rtx5070-moonlight-fp8-source.md`](probe-rtx5070-moonlight-fp8-source.md) | The GPU loss check `gate-fp8-shard.md` leaves to the maintainer: does a `soup train` LoRA run on the FP8-block copy of Moonlight-16B-A3B train like the same run on its bf16 source? Two arms, the same code and config apart from `base` and `output`, fused NF4 caches; limits on the step-1, per-step, epoch-3 and evaluation loss differences, GPU peak and seconds per step, set from a matched-size reference pair measured before the run | **FAIL (T2_max_step, T6_step_time)**, 2026-10-20. Both arms complete on the 8 GB card at the same 3.1 GiB GPU peak. Step-1, final-evaluation and epoch-3-mean losses agree to 0.081, 0.072 and 0.101, within their limits, but the 5-token row diverges by 3.27 nats at step 7 (limit 1.6), and B's median step took 1.91x A's (limit 1.10). A diagnosis after the verdict finds the cache layouts identical and the reader on direct I/O in both arms, with B's files reading at A's rate or faster on a rested drive; the most likely cause of T6 is the D: drive's sustained-read plateau (section 6), with B's timed steps starting 17 s after its own shard I/O. The rule's cool-down covered the GPU, not the drive. Speed parity and per-step parity are not shown |+| [`probe-rtx5070-giant-moe-step.md`](probe-rtx5070-giant-moe-step.md) | Does one LoRA training step of a synthetic Kimi-K2-shaped MoE (1,026,408,232,448 parameters, random weights, 533 GB NF4 per-expert store on the disk tier) complete on the RTX 5070 Laptop 8 GB? Measurement harness `benchmarks/harness/moe_stream_step.py`, not `soup train`; seq 512, batch 1, two steps | **COMPLETES.** With routing forced uniform (every expert read), step 0 took 867.95 s (0.590 tok/s) and step 1 1,108.77 s (0.462 tok/s), reading 69,120 experts and 1,570.34 GB per step at a 4,342 / 4,723 MiB peak allocation; loss and all 610 gradients finite. With the model's own routing only about 9.7 of 384 experts per layer were read, so that run is not a K2 step time. Shape only: no quality, convergence or real checkpoint |+| [`probe-rtx5070-giant-moe-seq-scaling.md`](probe-rtx5070-giant-moe-seq-scaling.md) | Does a longer sequence raise the tokens/s of that synthetic 1T step, since the per-step expert reads do not grow with the tokens? ABBA series at seq 512 and 2048 (loss over the last 512 positions), one step per process, under a rule committed before the run; random weights, shape only, measurement harness, not `soup train` | **NO VERDICT (2026-10-23).** Gate V5 (at most 2 GiB read from C: inside the step) failed in three of four runs, so the rule's claim is not made. Measured without standing: 1,036.4 s at seq 512 (0.494 tok/s) against about 1,119 s at seq 2048 (1.83 tok/s), ratios 3.70-3.71 against a 3.0 SHIP floor; the step is fetch-bound, not disk-bound (1.42-1.53 GB/s from D:) |## Harnessesdiff --git a/benchmarks/harness/moe_stream_step.py b/benchmarks/harness/moe_stream_step.pyindex c650bab9..9a07adce 100644--- a/benchmarks/harness/moe_stream_step.py+++ b/benchmarks/harness/moe_stream_step.py@@ -13,6 +13,14 @@ synthetic store every number is "random weights, shape only".With ``--weights`` a real checkpoint is sharded into ``--store`` first (NF4 onCUDA, the way ``soup train`` does it: native transformers class, the config'slayer count, ``--expert-layout``), unless ``--store`` already holds an index.++``--loss-tokens N`` takes the loss over the last N positions only (transformers'+``logits_to_keep``). Every position still runs forward and backward through every+decoder layer; only the vocabulary projection and the cross entropy shrink. At a+163,840-entry vocabulary they hold about 1.9 MB of VRAM per loss position, which+is what stops a long step from fitting (``probe-rtx5070-giant-moe-seq-scaling.md``).+Each step also records ``os_io``: the bytes the process read and the bytes each+physical drive read during the step, from the OS counters."""from __future__ import annotations@@ -26,6 +34,16 @@ import time_ATTENTION = ["q_proj", "q_a_proj", "q_b_proj", "kv_a_proj_with_mqa", "kv_b_proj", "o_proj"]+def _loss_window(text):+ try:+ value = int(text)+ except ValueError:+ value = -1+ if value < 0:+ raise argparse.ArgumentTypeError(f"--loss-tokens must be a whole number >= 0, got {text!r}")+ return value++def _args(argv=None):parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])parser.add_argument("--store", required=True, help="shard cache dir")@@ -38,6 +56,10 @@ def _args(argv=None):parser.add_argument("--seq", type=int, default=512)parser.add_argument("--batch", type=int, default=1)parser.add_argument("--steps", type=int, default=1)+ parser.add_argument(+ "--loss-tokens", type=_loss_window, default=0,+ help="loss over the last N positions of each row only (0, or N >= --seq: all)",+ )parser.add_argument("--lora-r", type=int, default=8)parser.add_argument("--buffers", type=int, default=2)parser.add_argument("--tier", choices=("ram", "disk"), default="ram")@@ -108,6 +130,65 @@ def _uniform_routing(model):return replaced+def _loss_inputs(ids, loss_tokens):+ """``(labels, extra forward kwargs)`` for a loss over the last ``loss_tokens`` positions.++ 0, or a window at least as long as the row, is the full loss: ``labels=ids`` and no+ extra kwargs, the harness's behaviour before ``--loss-tokens``. Otherwise+ ``logits_to_keep=N`` projects only the last N hidden states and the labels are the+ last N ids, so the loss's own shift keeps next-token targets: position T-N+j+ predicts id T-N+j+1, and the window's last position is ignored as the row's is.+ """+ if loss_tokens <= 0 or loss_tokens >= int(ids.shape[-1]):+ return ids, {}+ return ids[:, -loss_tokens:], {"logits_to_keep": int(loss_tokens)}+++def _forward_loss(model, ids, loss_tokens):+ """The step's loss and the number of loss positions per row."""+ labels, extra = _loss_inputs(ids, loss_tokens)+ return model(input_ids=ids, labels=labels, **extra).loss, int(labels.shape[-1])+++def _io_snapshot():+ """Bytes read so far by this process and by each physical drive (psutil).++ ``None`` for a counter the platform does not give. Drive read TIMES are not+ recorded: psutil's Windows ``read_time`` was not credible on this box (seconds+ for terabytes).+ """+ try:+ import psutil+ except ImportError:+ return {"process_read_bytes": None, "disk_read_bytes": None}+ refused = (AttributeError, OSError, RuntimeError, psutil.Error)+ try:+ process = int(psutil.Process().io_counters().read_bytes)+ except refused:+ process = None+ try:+ counters = psutil.disk_io_counters(perdisk=True) or {}+ disks = {name: int(value.read_bytes) for name, value in counters.items()}+ except refused:+ disks = None+ return {"process_read_bytes": process, "disk_read_bytes": disks}+++def _io_delta(before, after):+ """What was read between two snapshots: the process's bytes and each drive's."""+ process = None+ if before["process_read_bytes"] is not None and after["process_read_bytes"] is not None:+ process = after["process_read_bytes"] - before["process_read_bytes"]+ disks = None+ if before["disk_read_bytes"] is not None and after["disk_read_bytes"] is not None:+ disks = {+ name: value - before["disk_read_bytes"][name]+ for name, value in after["disk_read_bytes"].items()+ if name in before["disk_read_bytes"]+ }+ return {"process_read_bytes": process, "disk_read_bytes": disks}++def _peak_host_bytes():try:import psutil@@ -167,8 +248,9 @@ def main(argv=None):before = runtime.stats()torch.cuda.synchronize()torch.cuda.reset_peak_memory_stats()+ io_before = _io_snapshot()tick = time.perf_counter()- loss = model(input_ids=ids, labels=ids).loss+ loss, loss_positions = _forward_loss(model, ids, args.loss_tokens)loss.backward()torch.cuda.synchronize()seconds_to_grads = time.perf_counter() - tick@@ -183,6 +265,7 @@ def main(argv=None):optimizer.zero_grad(set_to_none=True)torch.cuda.synchronize()seconds = seconds_to_grads + (time.perf_counter() - tick_opt)+ io_after = _io_snapshot()after = runtime.stats()report["steps"].append({@@ -190,12 +273,17 @@ def main(argv=None):"seconds": round(seconds, 2),"tokens_per_second": round(args.batch * args.seq / seconds, 3),"loss": float(loss.detach()),+ "loss_positions": loss_positions,"loss_finite": bool(torch.isfinite(loss.detach())),"grads_finite": grads_finite,"grads_present": grads_present,"expert_fetches": after["expert_fetches"] - before["expert_fetches"],"expert_bytes_read": after["expert_bytes_read"] - before["expert_bytes_read"],"layer_loads": after["layer_loads"] - before["layer_loads"],+ # OS counters over the timed region: every byte this process read+ # (experts, dense layer files, vocabulary matrices) and what each+ # physical drive read, whoever asked for it.+ "os_io": _io_delta(io_before, io_after),# This step's peaks (reset at its start); the host figure is the# process's peak working set so far (Windows ``peak_wset``)."peak_vram_mib": torch.cuda.max_memory_allocated() // 2**20,diff --git a/benchmarks/probe-rtx5070-giant-moe-seq-scaling.md b/benchmarks/probe-rtx5070-giant-moe-seq-scaling.mdnew file mode 100644index 00000000..33ec469c--- /dev/null+++ b/benchmarks/probe-rtx5070-giant-moe-seq-scaling.md@@ -0,0 +1,394 @@+# Probe: do more tokens per step raise the tokens/s of the synthetic 1T MoE step? RTX 5070 Laptop 8 GB++Status: **PRE-REGISTERED (d83e57721); RUN 2026-10-23: NO VERDICT, gate V5 failed in 3 of 4 runs+(section 12).** This rule, its thresholds and the harness change it needs (section 3) were+committed together, before any run. Results are in section 12. Nothing in sections 1-11 may be edited after the first run except to+correct a typo, and any such correction is listed in section 12.++**Every number this probe produces is "random weights, shape only; routing forced uniform".**+It comes from a measurement harness (`benchmarks/harness/moe_stream_step.py`), not from+`soup train`. It runs one LoRA step per process on the synthetic store of+`probe-rtx5070-giant-moe-step.md` (1,026,408,232,448 parameters, K2 shape). This probe claims+nothing about "1T on 8 GB" beyond what that record shows: one full step completes, on random+weights, in a measurement harness. Prior art for streaming layers through a small device+memory, and for amortising the per-layer transfer over more samples per step, is L2L+(Pudipeddi et al. 2020, "Training Large Neural Networks with Constant Memory using a New+Execution Algorithm"). This probe tests the simplest form of that amortisation, a longer+sequence in one step. It does not implement L2L's layer-major micro-batching.++## 1. Question++The base record (`probe-rtx5070-giant-moe-step.md`, run 2) measured one step at seq 512 at+867.95 s (step 0) and 1,108.77 s (step 1). Each step read 1,570,336,174,080 bytes of NF4+experts, about 0.5 tokens/s. If the step's time is the expert fetch, and the fetch does not+grow with the tokens, then 4x the tokens per step should cost little extra time, and tokens/s+should rise nearly 4x. Does it, on this laptop?++## 2. What the code says (read, not run)++### 2.1 Knobs and what is timed++`moe_stream_step.py` takes `--seq`, `--batch` (already present, default 1), `--steps`,+`--route model|uniform`, `--tier ram|disk`, `--no-pin`, `--buffers`, `--lora-r` and, since+this commit, `--loss-tokens`. The timed region runs from one `cuda.synchronize()` to the next:+forward, backward and `AdamW.step()`. The `tokens_per_second` it reports is+`batch * seq / seconds`.++### 2.2 What a step reads, and why it does not grow with seq++`StreamedExperts` (`src/soup_cli/utils/expert_stream.py`) fetches every routed-to expert+three times: in the forward, in the decoder layer's checkpoint recompute, and in its own+backward, where it recomputes the expert MLP locally to get the input gradient. Each fetch is+synchronous: an unbuffered read, a host-to-device copy, an NF4 dequantisation, then the+compute. The dense part of each layer (attention, norms, router, shared expert) and the two+vocabulary matrices stream through the disk tier, which also reads with direct I/O (#974, PR+#991). So no byte of the step goes through the page cache.++`--route uniform` sends flattened row t to experts 8t .. 8t+7 (mod 384). Every expert of+every MoE layer is hit once `batch * seq >= 48`. A step then makes 60 x 384 x 3 = 69,120 expert+fetches, 1,570,336,174,080 bytes, **at every seq in this probe**.++| seq (batch 1) | token-expert pairs per expert per pass | experts hit per layer, uniform | untouched fraction, uniformly random top-8 |+|---|---|---|---|+| 512 | 10.7 | 384 / 384 | 2.1e-5 |+| 1024 | 21.3 | 384 / 384 | 4.3e-10 |+| 2048 | 42.7 | 384 / 384 | 1.9e-19 |+| 4096 | 85.3 | 384 / 384 | 3.5e-38 |++The last column is `(376/384)^T`. Even with random routing, coverage does not grow past 512.+With random weights, the model's own routing collapsed to about 9.7 experts per layer and pass+(base record section 4.1), so it is not an arm here.++### 2.3 What grows with the tokens++| work per step | seq 512 | seq 2048 |+|---|---|---|+| expert fetches / bytes | 69,120 / 1.570 TB | 69,120 / 1.570 TB |+| token-proportional compute (TFLOP) | ~116 | ~483 |++The compute row counts four kinds of work:+- MLA projections, the shared expert and layer 0's dense MLP: forward, recompute and input-gradient.+- The routed-expert GEMMs: forward, recompute, local recompute and input-gradient.+- Causal attention scores.+- The vocabulary projection and loss on 512 positions in both arms (identical, so it cancels).++The extra 367 TFLOP at an achieved 10-30 TFLOPS is +12 to +37 s. The routed GEMMs have 10.7+and 42.7 rows per expert, so they stay bound by reading the dense weight from VRAM, which is+fixed per fetch. That should keep the real increment near the low end. Allowing for elementwise,+gather and scatter work, **the prediction is t(2048) = t(512) + 10 to 60 s**. This is an+estimate, not a threshold.++### 2.4 VRAM per seq: the vocabulary is what stops a long step++Calibration: the base record's step-0 peak (allocated) was 4,342 MiB at seq 512. Three terms+add to a fixed base:+- **Checkpoint boundaries.** The 61 decoder inputs plus the final norm and lm_head inputs stay on the card from the forward to their backward: 63 x 7168 x 2 B = 0.861 MiB per token.+- **The loss at the vocabulary of 163,840.** transformers upcasts the logits to fp32, `cross_entropy` saves its log-softmax, and its backward holds two more fp32 tensors: 12 x 163,840 B = 1.875 MiB per loss position.+- **One decoder layer's recompute plus backward working set**, estimated from the saved tensors of MLA attention and the expert loop: 0.32-0.52 MiB per token. Its peak falls at the top layer, while all boundaries are still live.++Without the per-token terms, the base comes out at 2,941 MiB, which closes against the+bottom-up 2,900 MiB (2,783 MiB stream buffers from the run's JSON plus 116 MiB of fp32+adapters).++Attention: transformers' default `sdpa` path gets no mask (`is_causal=True`). The query/key+head dim is 192 and the value head dim 128. Torch's SDPA dispatch is expected to refuse its+flash kernel, which needs equal head dims, and to take the memory-efficient one, which+accepts a different value dim. That keeps attention memory linear in T. This was not checked+on the GPU at registration. If the math kernel were chosen instead, the fp32 scores alone+would add 1 GiB at seq 2048. Gate V3 would catch that, and the run would be reported as not+fitting.++| seq (batch 1) | peak alloc, full loss (harness as on main) | peak alloc, `--loss-tokens 512` | under the 7,500 MiB gate? |+|---|---|---|---|+| 512 | 4,342 (measured) | 4,342 (same computation) | yes |+| 1024 | 5,743 | 4,783 | yes either way (not an arm) |+| 2048 | **8,545**, over the 8,151 MiB card | **5,665-5,770** | only with `--loss-tokens` |+| 4096 | 14,149 | 7,780-8,599 | **no** |++- Seq 4096 does not fit even with a short loss window. The boundaries alone are 3,528 MiB on top of the 2,941 MiB base. Keeping them off the card (host-resident boundary activations) is a larger change than this probe.+- `--batch` is not a way round this. The uniform router flattens rows, and every term above scales with `batch * seq` (the loss term with `batch * loss_tokens`).+- So the arms are **seq 512 and seq 2048, batch 1, `--loss-tokens 512` in both**.+- Peak reserved ran 447-710 MiB above allocated in the base record, so the 2048 arm is predicted at **6,120-6,480 MiB reserved**.++### 2.5 Host RAM per seq++Nothing on the host scales with the tokens: activations stay on the card, nothing is+offloaded, and the ids are 8 bytes a token. The base record's peak working set was 7.75 GB+(step 0) and 7.89 GB (step 1). **The prediction is 7.6-8.2 GB at both seq values.** Each run+starts with at least 10 GB free, so at least ~2 GB stays free. The watchdog floor (section 8)+is 1.5 GB.++## 3. The harness change (same commit as this rule)++- **`--loss-tokens N`** (default 0 = all positions, unchanged behaviour). For 0 < N < seq, the step passes `logits_to_keep=N` and `labels=ids[:, -N:]`. The vocabulary projection and the cross entropy then cover the last N positions. The loss's own shift keeps next-token targets: position T-N+j predicts id T-N+j+1.+ - Every position still runs forward and backward through all 61 layers, and the gradient reaches positions outside the window: `tests/test_moe_stream_step.py` checks both on a tiny CPU DeepSeek-V3.+ - With N = seq, the step is the base record's computation exactly (`labels=ids`, no `logits_to_keep`).+- **`loss_positions` and `os_io`** in every step entry. `os_io` comes from psutil over the timed region: the bytes this process read, and the bytes each physical drive read. Drive read times are not recorded: psutil's Windows `read_time` was not credible on this box (7.4 s for 7.1 TB).+- Tests: `tests/test_moe_stream_step.py`.+ - 15 CPU tests. Eight mutations of the change were each killed (label slicing, the full-window guard, an off-by-one window caught only by the real-model test, the labels passed to the model, the negative-value refusal, the drive delta, the platform-refusal catch, the process delta).+ - One `gpu` test runs the windowed step end to end on the tiny store. It was not run at registration (CPU-only session), so it is pre-flight P2.++## 4. Arms, order, cache state++| run | arm | command differs only in |+|---|---|---|+| a1 | A | `--seq 512` |+| b1 | B | `--seq 2048` |+| b2 | B | `--seq 2048` |+| a2 | A | `--seq 512` |++- **Order ABBA**, one fresh process per run, `--steps 1`. Every measured step is the first step of its process, as the base record's 867.95 s was.+- The pairs are (a1, b1) and (b2, a2). A drift that slows later runs works against the claim in the first pair and for it in the second, and cancels in the pooled ratio.+- The base record's two consecutive steps differed by 1.277x for no known reason, so drift is a real risk on this box.+- **Page cache:** not a variable. Every expert read and every dense-layer read is direct I/O (section 2.2), so each run reads its 1.57 TB from the drive whatever ran before. Nothing is evicted or warmed. Gates V7 and V8 check that the bytes came from the drive.+- The next run starts as soon as the previous run's block returns. There is no cool-down, and the gaps are recorded in `box.jsonl`.++## 5. Primary metric and decision rule (thresholds fixed now)++`t(run)` is `steps[0].seconds`. Tokens/s is `seq / t`, so the tokens/s ratio of a pair is+`4 x t(A) / t(B)`.++- R1 = 4 x t(a1) / t(b1)+- R2 = 4 x t(a2) / t(b2)+- Rpool = 4 x (t(a1) + t(a2)) / (t(b1) + t(b2))++Outcomes, valid only if every gate in section 6 passes in all four runs:++- **SHIP** iff min(R1, R2, Rpool) >= 3.0. Then the record may state the claim below, with the measured Rpool. The words "nearly in proportion" are allowed only if also min(R1, R2) >= 3.4 and Rpool >= 3.6.+- **REFUTED** iff Rpool < 2.0: 4x the tokens cost at least 2x the time. At seq 2048 the step is no longer dominated by a token-independent cost.+- **NOT SHIPPED** otherwise: the record states the measured R1, R2 and Rpool, and makes no scaling claim.+- **NO VERDICT** if any gate fails in any run, or if the series is cut by the time cap (section 9).++Why 3.0: the base record's 1.277x step-to-step change, landing wholly against the claim,+gives 4 / 1.277 = 3.13. So 3.0 tolerates drift of that size plus a compute increment of up+to ~4%. Anything worse fails, which is the conservative direction.++The claim, fixed now: *"On an RTX 5070 Laptop (8 GB), one LoRA step of a synthetic+1.026 T-parameter Kimi K2-shaped store (random weights, routing forced uniform; a+measurement harness, not `soup train`) ran at >= 3x the tokens per second at seq 2048 as at+seq 512 (loss over the last 512 positions in both), while the step's expert reads stayed at+1.57 TB."*++## 6. Validity gates (every run)++- **V1 completes.** Exit code 0. `step.json` has exactly one step with `loss_finite`, `grads_finite`, `grads_present == 610` and `loss_positions == 512`.+- **V2 same work.** `store_bytes == 533,242,299,230`. `expert_fetches == 69,120` and `expert_bytes_read == 1,570,336,174,080` in every run (the base record's per-step values). `layer_loads` is equal in all four runs (120 expected, the base record's step 0).+- **V3 VRAM.** `peak_vram_mib <= 7,500` (the base record's gate; the card has 8,151 MiB) and `peak_reserved_mib <= 7,800`, so Windows never spills to shared memory, which would slow the step silently.+- **V4 host RAM.** `AvailableBytes >= 10e9` in the run's `before` line of `box.jsonl` (the owner's floor). `peak_host_bytes_so_far < 20e9`. The watchdog did not fire.+- **V5 no swap in the step.** `os_io.disk_read_bytes["PhysicalDrive0"] <= 2,147,483,648`. C: is disk 0 and holds `C:\pagefile.sys`, so paging in shows up there. The allowance covers DLL pages a first CUDA step maps in; P2 runs just before a1, so those pages should already be cached.+- **V6 awake, on AC.** In `watchdog.json`, `gaps == []` and `off_ac_samples == 0`. `ac == 2` in both box lines. No Kernel-Power event 41 or 42 in the System log between a1's `before` line and a2's `after` line.+- **V7 the bytes came from the drive, and only for this process.** With D = `os_io.disk_read_bytes["PhysicalDrive1"]` (D: is disk 1; P3 confirms) and P = `os_io.process_read_bytes`: D >= `expert_bytes_read` and |D - P| <= 0.02 x P. `watchdog.json` has `foreign_compute_pids == []`. A measurement hold stands on `.claude/SESSIONS.md` for the whole window.+- **V8 direct I/O.** `stats.direct_io_drives >= 1`.+- **V9 code.** P0-P2 pass (section 8).++A run that fails V4-V7 is a disturbed box, not a result. The whole ABBA series may then be+repeated once, in a new window, and runs from two series are never mixed. A failure of V1-V3+or V8-V9 is itself the finding, and the verdict is NO VERDICT.++## 7. Showing where the time goes (reported; it fixes the wording, not the verdict)++Per run: E = `expert_bytes_read`, P, D (section 6), and the drive rate D / t. From the+pooled arms, fit t(T) = F + c x T:+- c = (mean t(B) - mean t(A)) / 1536 seconds per token;+- F = mean t(A) - 512 c;+- F / mean t(A) is the token-independent share of the seq-512 step.++Wording rule:+- The record may call the step **disk-bound** only if D / t >= 3.27 GB/s in every run. That is 80% of the 4.09 GB/s (spread 4.09-4.14) that the async direct-I/O disk tier sustained over a step in `probe-rtx5070-what-bounds-streaming.md` section 21. That was measured on C:, the same Samsung MZAL81T0HFLB model as D:.+- Otherwise, if F / mean t(A) >= 0.9, it calls the step **fetch-bound**: the time is the per-expert fetch chain (synchronous read, host-to-device copy, dequantisation), which scales with bytes, not tokens, while the drive ran below its ceiling.+- Otherwise it calls it neither.++Prediction: the base record's expert-read rate was 1.81 and 1.42 GB/s, so expect+"fetch-bound, not disk-bound". This probe cannot split the read from the copy and the+dequantisation inside that chain.++## 8. Procedure (PowerShell, in this order)++All commands run from the worktree at this commit, with the base interpreter (not a venv).++**P0, import path.** The output must be a path under the worktree's `src` and `True`:++ $Tree = "C:\Users\user\projects\soup-wt\moe-seq-scaling"; $env:PYTHONPATH = "$Tree\src"; Set-Location $Tree+ python -c "import sys, soup_cli; print(soup_cli.__file__, sys.prefix == sys.base_prefix)"++**P1, code.** `git log -1 --format=%H` is this commit, or a later one that touches only+`benchmarks/results/` and section 12.++**P2, wiring on the GPU** (seconds, tiny store). One test must pass:++ python -m pytest tests/test_moe_stream_step.py -m gpu --no-cov -q++**P3, box.** Before anything else, check all of these and write a measurement hold on the board:+- `Get-Partition -DriveLetter D | Select-Object DiskNumber` returns 1. If not, use the number shown in V7.+- `D:/soup-bench/moe-k2-synth/index.json` exists. The store is on the owner's post-demo cleanup list, so check it first.+- `nvidia-smi --query-compute-apps=pid --format=csv,noheader` prints nothing.+- At least 10 GB of RAM is free, and the box is on AC with the lid open.++**Arms.** Run this block four times, in order, with `$Name, $Seq` = `a1, 512`, `b1, 2048`,+`b2, 2048`, `a2, 512`. Each block takes 15-20 minutes, which is longer than a 10-minute+foreground tool timeout. Launch each one as its own background call and wait for it to+return before starting the next.++ $Name = "a1"; $Seq = 512+ $Tree = "C:\Users\user\projects\soup-wt\moe-seq-scaling"; $env:PYTHONPATH = "$Tree\src"; Set-Location $Tree+ $Dir = "$Tree\benchmarks\results\probe-rtx5070\giant-moe\seq-scaling\$Name"+ New-Item -ItemType Directory -Force $Dir | Out-Null+ $Box = { param($At) [ordered]@{ at = $At; utc = (Get-Date).ToUniversalTime().ToString("o")+ mem = Get-CimInstance Win32_PerfRawData_PerfOS_Memory | Select-Object AvailableBytes, CommittedBytes, PagesInputPersec+ ac = (Get-CimInstance Win32_Battery).BatteryStatus+ gpu = (nvidia-smi --query-gpu=memory.used,temperature.gpu,clocks.sm --format=csv,noheader) } |+ ConvertTo-Json -Compress | Add-Content "$Dir\box.jsonl" }+ & $Box "before"+ $Step = Start-Process -FilePath python -PassThru -NoNewWindow `+ -RedirectStandardOutput "$Dir\stdout.log" -RedirectStandardError "$Dir\stderr.log" -ArgumentList @(+ "benchmarks/harness/moe_stream_step.py", "--store", "D:/soup-bench/moe-k2-synth",+ "--tier", "disk", "--no-pin", "--route", "uniform", "--batch", "1", "--steps", "1",+ "--loss-tokens", "512", "--seq", "$Seq", "--out", "$Dir\step.json")+ $null = $Step.Handle+ python benchmarks/harness/fp8_loss_check_watchdog.py --pid $Step.Id --out $Dir --min-free-gb 1.5 --disk D:/+ $Step.WaitForExit(); "exit $($Step.ExitCode)" | Set-Content "$Dir\exit.txt"+ & $Box "after"++The watchdog follows the step's process to its exit, sampling every 5 s: available RAM, the+process tree's RSS, `nvidia-smi`, AC, and gaps (sleep). It stops the step below 1.5 GB free,+which makes V4 fail. Its `--match` mode needs a standalone `train` argument, so the step is+named by `--pid`.++**After a2:**++ Get-WinEvent -FilterHashtable @{ LogName = "System"; ProviderName = "Microsoft-Windows-Kernel-Power"; Id = 41, 42; StartTime = (Get-Item "$Tree\benchmarks\results\probe-rtx5070\giant-moe\seq-scaling\a1\box.jsonl").CreationTime } -ErrorAction SilentlyContinue | Select-Object TimeCreated, Id++It must print nothing (V6). Then lift the hold, compute section 5 and section 7 from the four+`step.json` files, and write section 12.++## 9. Wall-clock budget++| part | time |+|---|---|+| P0-P3 | ~5 min |+| each run | 15-20.5 min (step 14-19.5 min, predicted t(512) 850-1,150 s and t(2048) 860-1,210 s, plus ~1 min of start, build and exit) |+| four runs | 60-82 min |+| **total** | **65-87 min, ~80 min expected** |++**Cap:** no run starts later than 95 minutes after a1's `before` line. A series the cap cuts+short is NO VERDICT.++## 10. Predictions (written now; not thresholds)++| | seq 512 | seq 2048 |+|---|---|---|+| step seconds | 850-1,150 | t(512) + 10 to 60 |+| tokens/s | 0.45-0.60 | 1.7-2.4 |+| R1, R2, Rpool | | 3.74-3.96 |+| peak alloc VRAM | 4,342 MiB | 5,665-5,770 MiB |+| peak reserved | about 5,050 MiB (base: 5,052) | 6,120-6,480 MiB |+| peak working set | 7.6-8.2 GB | 7.6-8.2 GB |+| expert bytes per step | 1,570,336,174,080 | 1,570,336,174,080 |+| drive rate D / t | 1.4-1.9 GB/s | 1.3-1.9 GB/s |+| wording (section 7) | fetch-bound, not disk-bound | |++## 11. What this probe cannot show++- Quality or convergence: the weights are random.+- A full-position loss at seq 2048: predicted at 8,545 MiB, it does not fit. The 2048 arm trains the loss on 512 of its 2,048 positions. Every position runs forward and backward, but a real fine-tune at that length would need a chunked loss.+- Seq 4096 or beyond: the boundary activations do not fit on the card (section 2.4).+- Model routing: random weights collapse it (section 2.2).+- How the fetch chain splits between the drive, the copy and the dequantisation.+- More than one step per process, or whether the base record's step-1 slowdown recurs.++## 12. Results++Series run on 2026-10-23, from a1's `before` line (14:05:38Z) to a2's `after` line (15:20:45Z), about 75 minutes. Commands exactly as section 8, at commit `d83e57721`, with the base interpreter, one fresh process per run, `--steps 1`. Raw files are in `benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/{a1,b1,b2,a2}/` (`box.jsonl`, `watchdog.jsonl`, `watchdog.json`, `step.json`, `exit.txt`, `stdout.log`, `stderr.log`).++Every number below is "random weights, shape only; routing forced uniform", from a measurement harness (`benchmarks/harness/moe_stream_step.py`), not `soup train`, one step per process. Nothing here shows more than that.++A first attempt at a1 started at 13:54:46Z the same day and died at about 14:01Z when the session that launched it closed (no `exit.txt`, process gone). Its files were moved out of the repository and are not used here. The series below is a full run from scratch.++### 12.1 Verdict++**NO VERDICT.** Gate V5 (no swap in the step) failed in three of the four runs (a1, b1, b2). Section 5 makes an outcome valid only if every gate passes in all four runs, and section 6 calls a run that fails V4-V7 "a disturbed box, not a result". So the section-5 claim is not made, and none of SHIP, REFUTED or NOT SHIPPED is declared. Section 6 allows the whole ABBA series to be repeated once, in a new window, with the two series never mixed. That was not done in this window.++The measured ratios are listed in 12.3 as numbers, not as a result. For the record, they sit above the 3.0 SHIP floor and above the 3.4 / 3.6 "nearly in proportion" floors, and the rule gives them no standing because V5 failed.++### 12.2 Arms++| run | seq | step seconds | tokens/s | peak alloc (MiB) | peak reserved (MiB) | peak working set (GB) | RAM available before (GB) | bytes read from D: (GB) | drive rate D / t (GB/s) |+|---|---|---|---|---|---|---|---|---|---|+| a1 | 512 | 1036.40 | 0.494 | 4,342 | 5,052 | 7.620 | 13.72 | 1,586.79 | 1.531 |+| b1 | 2048 | 1117.61 | 1.832 | 6,091 | 7,176 | 7.608 | 10.47 | 1,586.79 | 1.420 |+| b2 | 2048 | 1120.52 | 1.828 | 6,091 | 7,176 | 7.603 | 15.14 | 1,586.79 | 1.416 |+| a2 | 512 | 1036.30 | 0.494 | 4,342 | 5,052 | 7.729 | 15.61 | 1,586.79 | 1.531 |++- Peak working set is the watchdog's process-tree peak. The harness's own `peak_host_bytes_so_far` is 7.77, 7.76, 7.76, 7.76 GB.+- The loss repeats exactly inside each pair: 13.4771 at seq 512 (a1, a2) and 13.4916 at seq 2048 (b1, b2).+- Gaps between runs, `after` to the next `before`: 37 s, 27 s, 28 s. The latest start was a2, 57 min 19 s after a1's `before` line (cap: 95 min).++### 12.3 Ratios (section 5)++- R1 = 4 x 1036.40 / 1117.61 = **3.709**+- R2 = 4 x 1036.30 / 1120.52 = **3.699**+- Rpool = 4 x (1036.40 + 1036.30) / (1117.61 + 1120.52) = **3.704**+- Seq 2048 cost 81.21 s more than seq 512 in the first pair, 84.22 s in the second, 82.72 s pooled. The two runs of the same arm differed by 0.10 s (a1 vs a2) and 2.91 s (b1 vs b2).++### 12.4 Validity gates (section 6)++| gate | a1 | b1 | b2 | a2 |+|---|---|---|---|---|+| V1 completes | pass | pass | pass | pass |+| V2 same work | pass | pass | pass | pass |+| V3 VRAM | pass | pass | pass | pass |+| V4 host RAM | pass | pass | pass | pass |+| V5 no swap in the step | **FAIL** | **FAIL** | **FAIL** | pass |+| V6 awake, on AC | pass | pass | pass | pass |+| V7 bytes from the drive | pass | pass | pass | pass |+| V8 direct I/O | pass | pass | pass | pass |+| V9 code | pass | pass | pass | pass |++- **V1:** exit 0 in all four, one step each, `loss_finite` and `grads_finite` true, `grads_present` 610, `loss_positions` 512.+- **V2:** `store_bytes` 533,242,299,230, `expert_fetches` 69,120, `expert_bytes_read` 1,570,336,174,080 and `layer_loads` 120 in all four runs.+- **V3:** peak alloc 4,342 / 6,091 / 6,091 / 4,342 MiB against 7,500; peak reserved 5,052 / 7,176 / 7,176 / 5,052 MiB against 7,800. The seq-2048 margins are 1,409 MiB (alloc) and 624 MiB (reserved).+- **V4:** available RAM in the `before` lines was 13.72 / 10.47 / 15.14 / 15.61 GB (floor 10). `peak_host_bytes_so_far` was under 7.8 GB in every run (limit 20 GB). The watchdog did not fire. The lowest free RAM it sampled inside a run was 2.87 / 3.47 / 7.31 / 7.72 GB (its floor is 1.5 GB).+- **V5:** `os_io.disk_read_bytes["PhysicalDrive0"]` was 2,312,677,888 / 2,457,071,616 / 8,110,742,528 / 1,722,067,456 bytes against an allowance of 2,147,483,648. That is 1.08x, 1.14x and 3.78x the allowance in a1, b1 and b2, and 0.80x in a2.+- **V6:** `gaps == []`, `off_ac_samples == 0` and `ac == 2` in both box lines of all four runs. The Kernel-Power query of section 8 (events 41 and 42, from a1's `box.jsonl` creation time) printed nothing. Lid state cannot be read by software. No sleep gap was seen in the 5 s samples.+- **V7:** D (`PhysicalDrive1`) was 1,586,785,970,176 bytes in a1, a2 and b1 and 1,586,785,978,368 in b2, which is 1.0105x `expert_bytes_read`. |D - P| / P was 0.045%, 0.054%, 0.055%, 0.055% (limit 2%). `foreign_compute_pids == []` in all four. A measurement hold covering the window was declared on `.claude/SESSIONS.md` before the series (by the session that launched this one) and was read there, read-only, after it.+- **V8:** `direct_io_drives == 1` in all four.+- **V9:** P0: the import path was under the worktree's `src` and the prefix check printed `True`. P1: `git log -1` gave `d83e5772187ff1b218cfe11797964e0e22ee12ee`, with a clean tree. P2: `1 passed, 15 deselected in 25.73s`.++What the V5 failure looked like. The `PagesInputPersec` counter in the box lines rose by 597,063 / 639,377 / 1,898,033 / 488,689 pages between each run's `before` and `after` line. At 4,096 B per page that is 2.45 / 2.62 / 7.77 / 2.00 GB, against PhysicalDrive0 reads of 2.31 / 2.46 / 8.11 / 1.72 GB, in the same rank order. That is consistent with page-ins (hard faults) being the source of the C: reads. Whether they came from the pagefile or from mapped library files was not separated, and nothing here tests it. Step time did not follow the C: reads: b1 and b2 differ by 0.26% in step time with 2.46 GB and 8.11 GB read from C:.++### 12.5 Where the time goes (section 7)++- mean t(A) = 1036.35 s, mean t(B) = 1119.065 s.+- c = (1119.065 - 1036.35) / 1536 = 0.05385 s per token.+- F = 1036.35 - 512 x 0.05385 = 1008.78 s. F / mean t(A) = **0.973**.+- Drive rate D / t: 1.531, 1.420, 1.416, 1.531 GB/s. None reaches the 3.27 GB/s line, so the step is not called disk-bound.+- F / mean t(A) is at least 0.9, so by the wording rule the step is called **fetch-bound**: the time is the per-expert fetch chain, which scales with bytes, not tokens, while the drive ran below its ceiling. This probe cannot split that chain into read, copy and dequantisation. The section-7 wording does not depend on the verdict, but it comes from a series that failed V5.++### 12.6 Predictions (section 10) against measured++| | predicted | measured | |+|---|---|---|---|+| step seconds, seq 512 | 850-1,150 | 1036.4, 1036.3 | inside |+| step seconds, seq 2048 | t(512) + 10 to 60 | 1117.61, 1120.52 (t(512) + 81.2, + 84.2) | outside, 21-24 s above the top of the range |+| tokens/s, seq 512 | 0.45-0.60 | 0.494 | inside |+| tokens/s, seq 2048 | 1.7-2.4 | 1.832, 1.828 | inside |+| R1, R2, Rpool | 3.74-3.96 | 3.709, 3.699, 3.704 | below the range by 0.03-0.04 |+| peak alloc, seq 512 | 4,342 MiB | 4,342 | equal |+| peak alloc, seq 2048 | 5,665-5,770 MiB | 6,091 | 321 MiB above the top of the range |+| peak reserved, seq 512 | about 5,050 MiB | 5,052 | inside |+| peak reserved, seq 2048 | 6,120-6,480 MiB | 7,176 | 696 MiB above the top of the range |+| peak working set | 7.6-8.2 GB | 7.60-7.73 GB | inside |+| expert bytes per step | 1,570,336,174,080 | 1,570,336,174,080 | equal |+| drive rate, seq 512 | 1.4-1.9 GB/s | 1.531 | inside |+| drive rate, seq 2048 | 1.3-1.9 GB/s | 1.420, 1.416 | inside |+| wording | fetch-bound, not disk-bound | fetch-bound, not disk-bound | as predicted |++The VRAM estimate of section 2.4 was low at seq 2048 (both allocated and reserved), and the compute increment of section 2.3 was larger than estimated. The gates were not affected by the VRAM miss.++### 12.7 Procedure notes++- The P3 box checks ran before P2 (the rule puts P3 "before anything else"). P2 ran directly as a plain pytest call, not through the lock script. P3 values: `Get-Partition` gave disk 1 for D:, `index.json` existed, `nvidia-smi` listed no compute apps, 13.19 GB RAM was available, `BatteryStatus` was 2.+- A sleep guard ran for the whole series: a wrapper process that sets Windows' execution-state request (`SetThreadExecutionState`) while it lives, around a 7,500 s sleep, stopped after a2. No power setting was changed.+- Available RAM was at least 10 GB at every launch, so no arm waited for it.+- The hold on the shared board was not edited or lifted by the session that ran the arms.+- After the series, the Status line at the top of this file (above section 1) was changed from "PRE-REGISTERED, NOT RUN." to name the run and its outcome. No text of sections 1-11 was edited, and no typo correction was made.diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/box.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/box.jsonlnew file mode 100644index 00000000..4aa2aca6--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/box.jsonl@@ -0,0 +1,2 @@+{"at":"before","utc":"2026-10-23T14:05:38.3068581Z","mem":{"AvailableBytes":13718814720,"CommittedBytes":26984484864,"PagesInputPersec":198203300},"ac":2,"gpu":"0 MiB, 44, 210 MHz"}+{"at":"after","utc":"2026-10-23T14:23:12.1512629Z","mem":{"AvailableBytes":10765676544,"CommittedBytes":32647303168,"PagesInputPersec":198800363},"ac":2,"gpu":"0 MiB, 51, 5137 MHz"}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/exit.txt b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/exit.txtnew file mode 100644index 00000000..ca916d09--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/exit.txt@@ -0,0 +1 @@+exit 0diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/stderr.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/stderr.lognew file mode 100644index 00000000..635542ff--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/stderr.log@@ -0,0 +1,2 @@+W1023 19:05:47.319000 12344 site-packages\torch\utils\flop_counter.py:113] triton not found; flop counting will not work for triton kernels+[transformers] `torch_dtype` is deprecated! Use `dtype` instead!diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/stdout.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/stdout.lognew file mode 100644index 00000000..da657164--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/stdout.log@@ -0,0 +1 @@+{"step": 0, "seconds": 1036.4, "tokens_per_second": 0.494, "loss": 13.47708511352539, "loss_positions": 512, "loss_finite": true, "grads_finite": true, "grads_present": 610, "expert_fetches": 69120, "expert_bytes_read": 1570336174080, "layer_loads": 120, "os_io": {"process_read_bytes": 1587506334220, "disk_read_bytes": {"PhysicalDrive0": 2312677888, "PhysicalDrive1": 1586785970176}}, "peak_vram_mib": 4342, "peak_reserved_mib": 5052, "peak_host_bytes_so_far": 7773634560}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/step.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/step.jsonnew file mode 100644index 00000000..23319b89--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/step.json@@ -0,0 +1,77 @@+{+ "label": "random weights, shape only; routing forced uniform (token t -> experts kt..kt+k-1 mod E)",+ "args": {+ "store": "D:/soup-bench/moe-k2-synth",+ "weights": null,+ "model": null,+ "expert_layout": "per_expert",+ "out": "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\a1\\step.json",+ "seq": 512,+ "batch": 1,+ "steps": 1,+ "loss_tokens": 512,+ "lora_r": 8,+ "buffers": 2,+ "tier": "disk",+ "no_pin": true,+ "seed": 3,+ "label": "random weights, shape only",+ "route": "uniform"+ },+ "steps": [+ {+ "step": 0,+ "seconds": 1036.4,+ "tokens_per_second": 0.494,+ "loss": 13.47708511352539,+ "loss_positions": 512,+ "loss_finite": true,+ "grads_finite": true,+ "grads_present": 610,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080,+ "layer_loads": 120,+ "os_io": {+ "process_read_bytes": 1587506334220,+ "disk_read_bytes": {+ "PhysicalDrive0": 2312677888,+ "PhysicalDrive1": 1586785970176+ }+ },+ "peak_vram_mib": 4342,+ "peak_reserved_mib": 5052,+ "peak_host_bytes_so_far": 7773634560+ }+ ],+ "store_bytes": 533242299230,+ "store_expert_layout": "per_expert",+ "store_quant": "nf4",+ "store_double_quant": true,+ "build_seconds": 4.47,+ "routers_overridden": 60,+ "trainable_params": 30513664,+ "stats": {+ "n_layers": 61,+ "buffers": 2,+ "buffer_bytes": 2918596920,+ "large_buffer_bytes": 2348810240,+ "store_bytes": 5115134896,+ "pinned": false,+ "pinned_bytes": 0,+ "tier": "disk",+ "read_ahead": 2,+ "disk_bytes": 9779642448,+ "drive_count": 1,+ "direct_io_drives": 1,+ "direct_io_roots": {+ "D:/soup-bench/moe-k2-synth": true+ },+ "layer_loads": 120,+ "large_loads": 3,+ "device": "cuda",+ "total_params": 1026408232448,+ "nf4_scale_cache_bytes": 0,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080+ }+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/watchdog.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/watchdog.jsonnew file mode 100644index 00000000..f451e804--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/watchdog.json@@ -0,0 +1,39 @@+{+ "started": "2026-10-23T14:05:39.423+00:00",+ "match": null,+ "min_free_gb": 1.5,+ "period_s": 5.0,+ "fired": false,+ "gaps": [],+ "off_ac_samples": 0,+ "foreign_compute_pids": [],+ "pid": 12344,+ "cmdline": [+ "C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\python.exe",+ "benchmarks/harness/moe_stream_step.py",+ "--store",+ "D:/soup-bench/moe-k2-synth",+ "--tier",+ "disk",+ "--no-pin",+ "--route",+ "uniform",+ "--batch",+ "1",+ "--steps",+ "1",+ "--loss-tokens",+ "512",+ "--seq",+ "512",+ "--out",+ "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\a1\\step.json"+ ],+ "ended": "2026-10-23T14:23:12.113+00:00",+ "samples": 210,+ "min_free_ram_gb": 2.868,+ "max_tree_rss_gb": 7.62,+ "peak_wset_gb": 7.62,+ "max_gpu_memory_used_mib": 4996.0,+ "max_gpu_temperature_c": 56.0+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/watchdog.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/watchdog.jsonlnew file mode 100644index 00000000..bb5bea1e--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a1/watchdog.jsonl@@ -0,0 +1,210 @@+{"t": "2026-10-23T14:05:39.652+00:00", "free_ram_gb": 13.756, "tree_rss_gb": 0.022, "peak_wset_gb": 0.022, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 44.0, "clocks.sm": 210.0, "power.draw": 11.33, "compute_pids": []}}+{"t": "2026-10-23T14:05:45.263+00:00", "free_ram_gb": 13.049, "tree_rss_gb": 0.642, "peak_wset_gb": 0.642, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 45.0, "clocks.sm": 2947.0, "power.draw": 11.3, "compute_pids": []}}+{"t": "2026-10-23T14:05:49.543+00:00", "free_ram_gb": 12.792, "tree_rss_gb": 0.893, "peak_wset_gb": 0.893, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 12.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 45.0, "clocks.sm": 20772.0, "power.draw": 11.26, "compute_pids": [12344]}}+{"t": "2026-10-23T14:05:54.538+00:00", "free_ram_gb": 9.87, "tree_rss_gb": 3.935, "peak_wset_gb": 3.954, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3096.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 1417.0, "power.draw": 11.65, "compute_pids": [12344]}}+{"t": "2026-10-23T14:05:59.541+00:00", "free_ram_gb": 9.877, "tree_rss_gb": 3.998, "peak_wset_gb": 4.0, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3096.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 1417.0, "power.draw": 11.73, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:04.569+00:00", "free_ram_gb": 6.933, "tree_rss_gb": 6.881, "peak_wset_gb": 6.883, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3190.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 225.0, "power.draw": 11.69, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:09.584+00:00", "free_ram_gb": 6.91, "tree_rss_gb": 7.04, "peak_wset_gb": 7.044, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3680.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 45.0, "clocks.sm": 180.0, "power.draw": 3.89, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:14.615+00:00", "free_ram_gb": 6.814, "tree_rss_gb": 7.143, "peak_wset_gb": 7.157, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 46.0, "clocks.sm": 1087.0, "power.draw": 15.32, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:19.630+00:00", "free_ram_gb": 6.796, "tree_rss_gb": 7.147, "peak_wset_gb": 7.157, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 47.0, "clocks.sm": 1297.0, "power.draw": 18.58, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:24.645+00:00", "free_ram_gb": 6.763, "tree_rss_gb": 7.151, "peak_wset_gb": 7.157, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1267.0, "power.draw": 17.15, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:29.640+00:00", "free_ram_gb": 6.735, "tree_rss_gb": 7.156, "peak_wset_gb": 7.157, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 48.0, "clocks.sm": 1215.0, "power.draw": 23.54, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:34.644+00:00", "free_ram_gb": 6.652, "tree_rss_gb": 7.16, "peak_wset_gb": 7.16, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 49.0, "clocks.sm": 1215.0, "power.draw": 18.0, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:39.669+00:00", "free_ram_gb": 6.506, "tree_rss_gb": 7.164, "peak_wset_gb": 7.164, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 48.0, "clocks.sm": 1110.0, "power.draw": 17.33, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:44.667+00:00", "free_ram_gb": 6.788, "tree_rss_gb": 7.169, "peak_wset_gb": 7.169, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 49.0, "clocks.sm": 1110.0, "power.draw": 17.31, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:49.653+00:00", "free_ram_gb": 6.791, "tree_rss_gb": 7.174, "peak_wset_gb": 7.174, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1200.0, "power.draw": 18.16, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:54.657+00:00", "free_ram_gb": 6.937, "tree_rss_gb": 7.178, "peak_wset_gb": 7.178, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1110.0, "power.draw": 17.3, "compute_pids": [12344]}}+{"t": "2026-10-23T14:06:59.653+00:00", "free_ram_gb": 6.827, "tree_rss_gb": 7.183, "peak_wset_gb": 7.183, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1155.0, "power.draw": 17.16, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:04.657+00:00", "free_ram_gb": 6.844, "tree_rss_gb": 7.187, "peak_wset_gb": 7.187, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1140.0, "power.draw": 16.6, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:09.669+00:00", "free_ram_gb": 6.826, "tree_rss_gb": 7.191, "peak_wset_gb": 7.191, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1140.0, "power.draw": 17.41, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:14.658+00:00", "free_ram_gb": 6.797, "tree_rss_gb": 7.195, "peak_wset_gb": 7.195, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1140.0, "power.draw": 18.58, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:19.663+00:00", "free_ram_gb": 6.795, "tree_rss_gb": 7.2, "peak_wset_gb": 7.2, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1185.0, "power.draw": 17.99, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:24.666+00:00", "free_ram_gb": 6.87, "tree_rss_gb": 7.204, "peak_wset_gb": 7.204, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 17.08, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:29.707+00:00", "free_ram_gb": 6.882, "tree_rss_gb": 7.209, "peak_wset_gb": 7.209, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 16.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:34.687+00:00", "free_ram_gb": 6.822, "tree_rss_gb": 7.213, "peak_wset_gb": 7.213, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 17.33, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:39.690+00:00", "free_ram_gb": 6.759, "tree_rss_gb": 7.217, "peak_wset_gb": 7.217, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1005.0, "power.draw": 15.86, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:44.717+00:00", "free_ram_gb": 6.751, "tree_rss_gb": 7.216, "peak_wset_gb": 7.221, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 23.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 22.32, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:49.727+00:00", "free_ram_gb": 6.937, "tree_rss_gb": 7.22, "peak_wset_gb": 7.221, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1020.0, "power.draw": 17.0, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:54.742+00:00", "free_ram_gb": 6.659, "tree_rss_gb": 7.222, "peak_wset_gb": 7.222, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 5.0, "temperature.gpu": 50.0, "clocks.sm": 712.0, "power.draw": 10.0, "compute_pids": [12344]}}+{"t": "2026-10-23T14:07:59.800+00:00", "free_ram_gb": 6.789, "tree_rss_gb": 7.224, "peak_wset_gb": 7.224, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 2.0, "temperature.gpu": 49.0, "clocks.sm": 352.0, "power.draw": 9.14, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:04.763+00:00", "free_ram_gb": 6.49, "tree_rss_gb": 7.226, "peak_wset_gb": 7.226, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 1027.0, "power.draw": 14.84, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:09.797+00:00", "free_ram_gb": 6.789, "tree_rss_gb": 7.229, "peak_wset_gb": 7.229, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1140.0, "power.draw": 15.69, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:14.754+00:00", "free_ram_gb": 6.754, "tree_rss_gb": 7.231, "peak_wset_gb": 7.231, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 1065.0, "power.draw": 15.5, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:19.792+00:00", "free_ram_gb": 6.783, "tree_rss_gb": 7.235, "peak_wset_gb": 7.235, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1087.0, "power.draw": 16.94, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:24.780+00:00", "free_ram_gb": 6.792, "tree_rss_gb": 7.239, "peak_wset_gb": 7.239, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1117.0, "power.draw": 15.6, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:29.799+00:00", "free_ram_gb": 6.777, "tree_rss_gb": 7.242, "peak_wset_gb": 7.242, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1012.0, "power.draw": 15.69, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:34.795+00:00", "free_ram_gb": 6.684, "tree_rss_gb": 7.245, "peak_wset_gb": 7.245, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 937.0, "power.draw": 16.12, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:39.821+00:00", "free_ram_gb": 6.844, "tree_rss_gb": 7.248, "peak_wset_gb": 7.248, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 915.0, "power.draw": 14.73, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:44.858+00:00", "free_ram_gb": 6.813, "tree_rss_gb": 7.251, "peak_wset_gb": 7.251, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1042.0, "power.draw": 15.47, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:49.988+00:00", "free_ram_gb": 6.741, "tree_rss_gb": 7.254, "peak_wset_gb": 7.254, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 960.0, "power.draw": 16.71, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:54.879+00:00", "free_ram_gb": 6.79, "tree_rss_gb": 7.257, "peak_wset_gb": 7.257, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 15.01, "compute_pids": [12344]}}+{"t": "2026-10-23T14:08:59.923+00:00", "free_ram_gb": 6.865, "tree_rss_gb": 7.26, "peak_wset_gb": 7.26, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 15.08, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:05.379+00:00", "free_ram_gb": 6.835, "tree_rss_gb": 7.263, "peak_wset_gb": 7.263, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 997.0, "power.draw": 13.68, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:10.365+00:00", "free_ram_gb": 7.021, "tree_rss_gb": 7.266, "peak_wset_gb": 7.266, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1027.0, "power.draw": 16.02, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:14.887+00:00", "free_ram_gb": 6.892, "tree_rss_gb": 7.269, "peak_wset_gb": 7.269, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1207.0, "power.draw": 16.48, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:19.913+00:00", "free_ram_gb": 6.914, "tree_rss_gb": 7.273, "peak_wset_gb": 7.273, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 922.0, "power.draw": 15.94, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:24.934+00:00", "free_ram_gb": 6.853, "tree_rss_gb": 7.275, "peak_wset_gb": 7.275, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 1020.0, "power.draw": 13.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:29.954+00:00", "free_ram_gb": 6.873, "tree_rss_gb": 7.278, "peak_wset_gb": 7.278, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 975.0, "power.draw": 14.86, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:34.947+00:00", "free_ram_gb": 6.873, "tree_rss_gb": 7.281, "peak_wset_gb": 7.281, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 49.0, "clocks.sm": 1042.0, "power.draw": 14.58, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:39.969+00:00", "free_ram_gb": 6.921, "tree_rss_gb": 7.284, "peak_wset_gb": 7.284, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 49.0, "clocks.sm": 1035.0, "power.draw": 15.24, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:45.013+00:00", "free_ram_gb": 6.857, "tree_rss_gb": 7.287, "peak_wset_gb": 7.287, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1042.0, "power.draw": 15.19, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:49.976+00:00", "free_ram_gb": 6.845, "tree_rss_gb": 7.291, "peak_wset_gb": 7.291, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 48.0, "clocks.sm": 1027.0, "power.draw": 14.76, "compute_pids": [12344]}}+{"t": "2026-10-23T14:09:55.002+00:00", "free_ram_gb": 6.768, "tree_rss_gb": 7.293, "peak_wset_gb": 7.293, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1057.0, "power.draw": 15.25, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:00.022+00:00", "free_ram_gb": 6.757, "tree_rss_gb": 7.296, "peak_wset_gb": 7.296, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1080.0, "power.draw": 16.03, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:05.017+00:00", "free_ram_gb": 6.804, "tree_rss_gb": 7.299, "peak_wset_gb": 7.299, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 48.0, "clocks.sm": 982.0, "power.draw": 23.0, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:10.046+00:00", "free_ram_gb": 6.806, "tree_rss_gb": 7.302, "peak_wset_gb": 7.302, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 997.0, "power.draw": 14.69, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:15.056+00:00", "free_ram_gb": 6.86, "tree_rss_gb": 7.305, "peak_wset_gb": 7.305, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 1110.0, "power.draw": 15.07, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:20.094+00:00", "free_ram_gb": 6.678, "tree_rss_gb": 7.308, "peak_wset_gb": 7.308, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 907.0, "power.draw": 14.48, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:25.088+00:00", "free_ram_gb": 6.846, "tree_rss_gb": 7.311, "peak_wset_gb": 7.311, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 1027.0, "power.draw": 14.46, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:30.125+00:00", "free_ram_gb": 6.86, "tree_rss_gb": 7.314, "peak_wset_gb": 7.314, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 1005.0, "power.draw": 15.69, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:35.160+00:00", "free_ram_gb": 6.935, "tree_rss_gb": 7.317, "peak_wset_gb": 7.317, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 48.0, "clocks.sm": 1087.0, "power.draw": 13.43, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:40.181+00:00", "free_ram_gb": 6.771, "tree_rss_gb": 7.32, "peak_wset_gb": 7.32, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 990.0, "power.draw": 15.31, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:45.179+00:00", "free_ram_gb": 6.703, "tree_rss_gb": 7.323, "peak_wset_gb": 7.323, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 997.0, "power.draw": 14.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:50.222+00:00", "free_ram_gb": 6.867, "tree_rss_gb": 7.326, "peak_wset_gb": 7.326, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1140.0, "power.draw": 15.76, "compute_pids": [12344]}}+{"t": "2026-10-23T14:10:55.247+00:00", "free_ram_gb": 6.866, "tree_rss_gb": 7.329, "peak_wset_gb": 7.329, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1042.0, "power.draw": 15.06, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:00.221+00:00", "free_ram_gb": 6.884, "tree_rss_gb": 7.332, "peak_wset_gb": 7.332, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1140.0, "power.draw": 15.62, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:05.228+00:00", "free_ram_gb": 6.871, "tree_rss_gb": 7.334, "peak_wset_gb": 7.334, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 48.0, "clocks.sm": 300.0, "power.draw": 9.91, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:10.261+00:00", "free_ram_gb": 6.858, "tree_rss_gb": 7.337, "peak_wset_gb": 7.337, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 960.0, "power.draw": 14.81, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:15.269+00:00", "free_ram_gb": 6.818, "tree_rss_gb": 7.34, "peak_wset_gb": 7.34, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 48.0, "clocks.sm": 900.0, "power.draw": 13.44, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:20.329+00:00", "free_ram_gb": 6.52, "tree_rss_gb": 7.343, "peak_wset_gb": 7.343, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 1027.0, "power.draw": 15.08, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:25.334+00:00", "free_ram_gb": 6.95, "tree_rss_gb": 7.345, "peak_wset_gb": 7.345, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 1050.0, "power.draw": 14.4, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:30.356+00:00", "free_ram_gb": 6.875, "tree_rss_gb": 7.428, "peak_wset_gb": 7.428, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4814.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 1065.0, "power.draw": 14.57, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:35.352+00:00", "free_ram_gb": 6.682, "tree_rss_gb": 7.591, "peak_wset_gb": 7.592, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4878.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 48.0, "clocks.sm": 1020.0, "power.draw": 12.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:40.379+00:00", "free_ram_gb": 6.647, "tree_rss_gb": 7.591, "peak_wset_gb": 7.592, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4878.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 48.0, "clocks.sm": 1072.0, "power.draw": 15.66, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:45.359+00:00", "free_ram_gb": 6.668, "tree_rss_gb": 7.619, "peak_wset_gb": 7.619, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4878.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1065.0, "power.draw": 15.86, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:50.371+00:00", "free_ram_gb": 6.666, "tree_rss_gb": 7.619, "peak_wset_gb": 7.619, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4880.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1087.0, "power.draw": 16.89, "compute_pids": [12344]}}+{"t": "2026-10-23T14:11:55.362+00:00", "free_ram_gb": 6.662, "tree_rss_gb": 7.619, "peak_wset_gb": 7.619, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4880.0, "memory.total": 8151.0, "utilization.gpu": 6.0, "temperature.gpu": 48.0, "clocks.sm": 922.0, "power.draw": 14.37, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:00.401+00:00", "free_ram_gb": 6.685, "tree_rss_gb": 7.619, "peak_wset_gb": 7.619, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4882.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1080.0, "power.draw": 16.21, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:05.413+00:00", "free_ram_gb": 6.641, "tree_rss_gb": 7.62, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4882.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 922.0, "power.draw": 13.97, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:10.475+00:00", "free_ram_gb": 6.628, "tree_rss_gb": 7.62, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4884.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 1215.0, "power.draw": 15.07, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:15.496+00:00", "free_ram_gb": 6.687, "tree_rss_gb": 7.62, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4884.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.94, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:20.477+00:00", "free_ram_gb": 6.478, "tree_rss_gb": 7.62, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4884.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 967.0, "power.draw": 14.28, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:25.473+00:00", "free_ram_gb": 6.445, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4886.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 967.0, "power.draw": 15.28, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:30.502+00:00", "free_ram_gb": 6.327, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4886.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 1027.0, "power.draw": 14.33, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:35.503+00:00", "free_ram_gb": 6.413, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4886.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 915.0, "power.draw": 13.89, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:40.539+00:00", "free_ram_gb": 6.367, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4888.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 1057.0, "power.draw": 15.57, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:45.539+00:00", "free_ram_gb": 6.233, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4888.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 900.0, "power.draw": 13.31, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:50.572+00:00", "free_ram_gb": 6.364, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4890.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 997.0, "power.draw": 14.05, "compute_pids": [12344]}}+{"t": "2026-10-23T14:12:55.578+00:00", "free_ram_gb": 6.231, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4890.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 1080.0, "power.draw": 15.53, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:00.580+00:00", "free_ram_gb": 6.209, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4890.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 952.0, "power.draw": 13.8, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:05.614+00:00", "free_ram_gb": 6.08, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4892.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 967.0, "power.draw": 15.08, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:10.651+00:00", "free_ram_gb": 6.06, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4892.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 1005.0, "power.draw": 16.03, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:15.646+00:00", "free_ram_gb": 6.025, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4892.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 922.0, "power.draw": 14.03, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:20.683+00:00", "free_ram_gb": 5.822, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4894.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 48.0, "clocks.sm": 1132.0, "power.draw": 16.15, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:25.675+00:00", "free_ram_gb": 5.813, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4894.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 49.0, "clocks.sm": 1275.0, "power.draw": 18.07, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:30.722+00:00", "free_ram_gb": 5.744, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4894.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 48.0, "clocks.sm": 945.0, "power.draw": 14.07, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:35.704+00:00", "free_ram_gb": 5.683, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4896.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1080.0, "power.draw": 15.38, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:40.724+00:00", "free_ram_gb": 5.39, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4896.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 907.0, "power.draw": 13.01, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:45.722+00:00", "free_ram_gb": 4.916, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4898.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 16.47, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:50.793+00:00", "free_ram_gb": 4.254, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4898.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1140.0, "power.draw": 17.07, "compute_pids": [12344]}}+{"t": "2026-10-23T14:13:55.761+00:00", "free_ram_gb": 4.309, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4898.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1087.0, "power.draw": 15.12, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:00.775+00:00", "free_ram_gb": 4.975, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4900.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 53.0, "clocks.sm": 1020.0, "power.draw": 16.81, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:05.795+00:00", "free_ram_gb": 5.998, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4900.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1027.0, "power.draw": 15.74, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:10.793+00:00", "free_ram_gb": 6.187, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4902.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 52.0, "clocks.sm": 1147.0, "power.draw": 18.4, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:15.791+00:00", "free_ram_gb": 6.241, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4902.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 15.71, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:20.782+00:00", "free_ram_gb": 6.248, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4904.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 17.99, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:25.792+00:00", "free_ram_gb": 6.265, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4904.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1117.0, "power.draw": 16.13, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:30.785+00:00", "free_ram_gb": 6.277, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4906.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 17.52, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:35.793+00:00", "free_ram_gb": 6.254, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4908.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1065.0, "power.draw": 16.86, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:40.795+00:00", "free_ram_gb": 6.216, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4908.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1012.0, "power.draw": 15.5, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:45.798+00:00", "free_ram_gb": 6.227, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4910.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 17.05, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:50.806+00:00", "free_ram_gb": 6.174, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4910.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1042.0, "power.draw": 15.84, "compute_pids": [12344]}}+{"t": "2026-10-23T14:14:55.813+00:00", "free_ram_gb": 6.191, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4912.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1050.0, "power.draw": 17.14, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:00.812+00:00", "free_ram_gb": 5.87, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4912.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 982.0, "power.draw": 15.93, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:05.821+00:00", "free_ram_gb": 5.826, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4914.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1027.0, "power.draw": 16.43, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:10.826+00:00", "free_ram_gb": 5.903, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4914.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 975.0, "power.draw": 16.09, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:15.849+00:00", "free_ram_gb": 6.163, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4916.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1035.0, "power.draw": 17.04, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:20.839+00:00", "free_ram_gb": 5.902, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4918.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 16.17, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:25.846+00:00", "free_ram_gb": 5.807, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4918.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 990.0, "power.draw": 18.26, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:30.864+00:00", "free_ram_gb": 5.771, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4920.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 18.12, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:35.862+00:00", "free_ram_gb": 5.795, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4920.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 18.31, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:40.921+00:00", "free_ram_gb": 5.5, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4922.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1072.0, "power.draw": 16.6, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:45.881+00:00", "free_ram_gb": 5.305, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4922.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1747.0, "power.draw": 18.15, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:50.893+00:00", "free_ram_gb": 5.68, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4924.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 52.0, "clocks.sm": 1080.0, "power.draw": 17.54, "compute_pids": [12344]}}+{"t": "2026-10-23T14:15:55.890+00:00", "free_ram_gb": 5.689, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4924.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 967.0, "power.draw": 15.09, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:00.917+00:00", "free_ram_gb": 5.219, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4926.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 54.0, "clocks.sm": 960.0, "power.draw": 15.23, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:05.936+00:00", "free_ram_gb": 5.293, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4926.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 54.0, "clocks.sm": 1005.0, "power.draw": 15.65, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:10.947+00:00", "free_ram_gb": 4.994, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4926.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 53.0, "clocks.sm": 937.0, "power.draw": 14.6, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:15.956+00:00", "free_ram_gb": 5.227, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4928.0, "memory.total": 8151.0, "utilization.gpu": 7.0, "temperature.gpu": 52.0, "clocks.sm": 870.0, "power.draw": 13.63, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:20.997+00:00", "free_ram_gb": 5.19, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4928.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 14.35, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:25.989+00:00", "free_ram_gb": 5.225, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4928.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 952.0, "power.draw": 25.34, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:30.999+00:00", "free_ram_gb": 5.226, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4930.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1102.0, "power.draw": 15.51, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:36.048+00:00", "free_ram_gb": 5.03, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4930.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 14.94, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:41.044+00:00", "free_ram_gb": 5.1, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4930.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 14.71, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:46.069+00:00", "free_ram_gb": 4.895, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4932.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 945.0, "power.draw": 14.27, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:51.072+00:00", "free_ram_gb": 4.599, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4932.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1035.0, "power.draw": 14.67, "compute_pids": [12344]}}+{"t": "2026-10-23T14:16:56.107+00:00", "free_ram_gb": 4.632, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4932.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 967.0, "power.draw": 13.81, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:01.084+00:00", "free_ram_gb": 4.596, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4934.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 960.0, "power.draw": 13.66, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:06.102+00:00", "free_ram_gb": 5.006, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4934.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 9.1, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:11.096+00:00", "free_ram_gb": 4.826, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4934.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 52.0, "clocks.sm": 945.0, "power.draw": 12.79, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:16.108+00:00", "free_ram_gb": 4.223, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4936.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 55.0, "clocks.sm": 1087.0, "power.draw": 16.17, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:21.117+00:00", "free_ram_gb": 4.535, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4936.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 56.0, "clocks.sm": 1125.0, "power.draw": 16.6, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:26.114+00:00", "free_ram_gb": 4.529, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4938.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 56.0, "clocks.sm": 1147.0, "power.draw": 18.91, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:31.140+00:00", "free_ram_gb": 4.65, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4938.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 55.0, "clocks.sm": 1230.0, "power.draw": 18.69, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:36.140+00:00", "free_ram_gb": 4.55, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4940.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 55.0, "clocks.sm": 1297.0, "power.draw": 17.48, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:41.160+00:00", "free_ram_gb": 4.606, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4942.0, "memory.total": 8151.0, "utilization.gpu": 21.0, "temperature.gpu": 55.0, "clocks.sm": 1170.0, "power.draw": 19.39, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:46.154+00:00", "free_ram_gb": 4.702, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4942.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 54.0, "clocks.sm": 1140.0, "power.draw": 17.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:51.181+00:00", "free_ram_gb": 4.571, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4944.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 54.0, "clocks.sm": 1050.0, "power.draw": 15.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:17:56.208+00:00", "free_ram_gb": 4.546, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4944.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 53.0, "clocks.sm": 1012.0, "power.draw": 14.21, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:01.235+00:00", "free_ram_gb": 4.553, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4944.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 53.0, "clocks.sm": 1042.0, "power.draw": 13.46, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:06.247+00:00", "free_ram_gb": 4.64, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4946.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 53.0, "clocks.sm": 1080.0, "power.draw": 17.78, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:11.250+00:00", "free_ram_gb": 4.285, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4946.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 53.0, "clocks.sm": 997.0, "power.draw": 15.04, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:16.307+00:00", "free_ram_gb": 4.527, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4948.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1072.0, "power.draw": 16.0, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:21.300+00:00", "free_ram_gb": 4.424, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4948.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1155.0, "power.draw": 16.08, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:26.320+00:00", "free_ram_gb": 4.475, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4948.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1020.0, "power.draw": 14.29, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:31.334+00:00", "free_ram_gb": 4.704, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4950.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1042.0, "power.draw": 15.23, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:36.367+00:00", "free_ram_gb": 4.925, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4950.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 982.0, "power.draw": 14.68, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:41.414+00:00", "free_ram_gb": 5.023, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4950.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 14.4, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:46.389+00:00", "free_ram_gb": 5.06, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4952.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 16.5, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:51.426+00:00", "free_ram_gb": 5.057, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4952.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 14.56, "compute_pids": [12344]}}+{"t": "2026-10-23T14:18:56.408+00:00", "free_ram_gb": 4.889, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4954.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 15.2, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:01.421+00:00", "free_ram_gb": 4.527, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4954.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 997.0, "power.draw": 14.15, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:06.450+00:00", "free_ram_gb": 4.361, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4954.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 975.0, "power.draw": 15.93, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:11.445+00:00", "free_ram_gb": 4.257, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4956.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 15.48, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:16.464+00:00", "free_ram_gb": 4.237, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4956.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 997.0, "power.draw": 15.31, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:21.499+00:00", "free_ram_gb": 4.089, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4958.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 14.59, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:26.522+00:00", "free_ram_gb": 4.081, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4958.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1102.0, "power.draw": 16.29, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:31.550+00:00", "free_ram_gb": 4.062, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4958.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1102.0, "power.draw": 14.14, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:36.610+00:00", "free_ram_gb": 4.31, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4960.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 16.37, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:41.623+00:00", "free_ram_gb": 4.483, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4960.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 52.0, "clocks.sm": 1132.0, "power.draw": 14.82, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:46.651+00:00", "free_ram_gb": 4.424, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4960.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 13.71, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:51.669+00:00", "free_ram_gb": 4.525, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4962.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 52.0, "clocks.sm": 1080.0, "power.draw": 17.19, "compute_pids": [12344]}}+{"t": "2026-10-23T14:19:56.647+00:00", "free_ram_gb": 4.627, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4962.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1102.0, "power.draw": 15.74, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:01.646+00:00", "free_ram_gb": 4.609, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4964.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 52.0, "clocks.sm": 1095.0, "power.draw": 18.24, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:06.691+00:00", "free_ram_gb": 4.169, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4964.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 975.0, "power.draw": 14.49, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:11.731+00:00", "free_ram_gb": 4.415, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4966.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 51.0, "clocks.sm": 982.0, "power.draw": 15.63, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:16.681+00:00", "free_ram_gb": 4.222, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4966.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 20.5, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:21.702+00:00", "free_ram_gb": 3.931, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4968.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 54.0, "clocks.sm": 1035.0, "power.draw": 17.25, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:26.720+00:00", "free_ram_gb": 3.995, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4968.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 55.0, "clocks.sm": 1222.0, "power.draw": 14.67, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:31.762+00:00", "free_ram_gb": 3.99, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4968.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 54.0, "clocks.sm": 1012.0, "power.draw": 13.95, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:36.775+00:00", "free_ram_gb": 4.119, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4970.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 53.0, "clocks.sm": 1087.0, "power.draw": 16.4, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:41.804+00:00", "free_ram_gb": 4.043, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4970.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 915.0, "power.draw": 34.22, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:46.828+00:00", "free_ram_gb": 4.034, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4972.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 53.0, "clocks.sm": 937.0, "power.draw": 16.87, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:51.834+00:00", "free_ram_gb": 4.19, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4972.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 53.0, "clocks.sm": 1012.0, "power.draw": 16.53, "compute_pids": [12344]}}+{"t": "2026-10-23T14:20:56.836+00:00", "free_ram_gb": 3.589, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4972.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 53.0, "clocks.sm": 952.0, "power.draw": 13.97, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:01.884+00:00", "free_ram_gb": 3.402, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4974.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 53.0, "clocks.sm": 1050.0, "power.draw": 16.43, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:06.860+00:00", "free_ram_gb": 2.959, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4974.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 54.0, "clocks.sm": 1020.0, "power.draw": 16.07, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:11.886+00:00", "free_ram_gb": 2.868, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4976.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 56.0, "clocks.sm": 1080.0, "power.draw": 14.16, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:16.918+00:00", "free_ram_gb": 3.547, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4976.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 55.0, "clocks.sm": 1005.0, "power.draw": 15.24, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:21.908+00:00", "free_ram_gb": 2.983, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4978.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 54.0, "clocks.sm": 1065.0, "power.draw": 17.49, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:26.910+00:00", "free_ram_gb": 3.326, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4978.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 53.0, "clocks.sm": 1005.0, "power.draw": 16.22, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:31.913+00:00", "free_ram_gb": 3.533, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4978.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 15.93, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:36.973+00:00", "free_ram_gb": 3.543, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4980.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1087.0, "power.draw": 15.18, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:41.993+00:00", "free_ram_gb": 3.515, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4980.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 52.0, "clocks.sm": 990.0, "power.draw": 15.3, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:46.987+00:00", "free_ram_gb": 3.684, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4982.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 17.43, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:52.033+00:00", "free_ram_gb": 3.7, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4982.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 16.34, "compute_pids": [12344]}}+{"t": "2026-10-23T14:21:57.037+00:00", "free_ram_gb": 3.832, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4982.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 13.89, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:02.074+00:00", "free_ram_gb": 3.693, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 15.71, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:07.058+00:00", "free_ram_gb": 3.486, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 14.83, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:12.070+00:00", "free_ram_gb": 3.572, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4986.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 51.0, "clocks.sm": 1747.0, "power.draw": 18.65, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:17.091+00:00", "free_ram_gb": 3.762, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4986.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 15.54, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:22.095+00:00", "free_ram_gb": 3.777, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4988.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 19.16, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:27.157+00:00", "free_ram_gb": 3.667, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4988.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 15.33, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:32.185+00:00", "free_ram_gb": 3.505, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4988.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1012.0, "power.draw": 14.43, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:37.152+00:00", "free_ram_gb": 3.586, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4990.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 52.0, "clocks.sm": 1027.0, "power.draw": 18.2, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:42.181+00:00", "free_ram_gb": 3.641, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4990.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 53.0, "clocks.sm": 1065.0, "power.draw": 15.42, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:47.194+00:00", "free_ram_gb": 3.916, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4992.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 53.0, "clocks.sm": 1140.0, "power.draw": 17.11, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:52.223+00:00", "free_ram_gb": 3.934, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4992.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1020.0, "power.draw": 15.94, "compute_pids": [12344]}}+{"t": "2026-10-23T14:22:57.231+00:00", "free_ram_gb": 4.005, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4994.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 52.0, "clocks.sm": 1125.0, "power.draw": 17.31, "compute_pids": [12344]}}+{"t": "2026-10-23T14:23:02.231+00:00", "free_ram_gb": 3.896, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4994.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 52.0, "clocks.sm": 1035.0, "power.draw": 15.55, "compute_pids": [12344]}}+{"t": "2026-10-23T14:23:07.222+00:00", "free_ram_gb": 4.041, "tree_rss_gb": 7.619, "peak_wset_gb": 7.62, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4996.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 52.0, "clocks.sm": 1087.0, "power.draw": 17.69, "compute_pids": [12344]}}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/box.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/box.jsonlnew file mode 100644index 00000000..1084c72f--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/box.jsonl@@ -0,0 +1,2 @@+{"at":"before","utc":"2026-10-23T15:02:57.7293933Z","mem":{"AvailableBytes":15609724928,"CommittedBytes":28849553408,"PagesInputPersec":201357154},"ac":2,"gpu":"0 MiB, 45, 7 MHz"}+{"at":"after","utc":"2026-10-23T15:20:45.8633218Z","mem":{"AvailableBytes":15271714816,"CommittedBytes":29247221760,"PagesInputPersec":201845843},"ac":2,"gpu":"0 MiB, 46, 0 MHz"}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/exit.txt b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/exit.txtnew file mode 100644index 00000000..ca916d09--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/exit.txt@@ -0,0 +1 @@+exit 0diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/stderr.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/stderr.lognew file mode 100644index 00000000..97c72f96--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/stderr.log@@ -0,0 +1,2 @@+W1023 20:03:16.280000 21972 site-packages\torch\utils\flop_counter.py:113] triton not found; flop counting will not work for triton kernels+[transformers] `torch_dtype` is deprecated! Use `dtype` instead!diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/stdout.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/stdout.lognew file mode 100644index 00000000..ebfc9e50--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/stdout.log@@ -0,0 +1 @@+{"step": 0, "seconds": 1036.3, "tokens_per_second": 0.494, "loss": 13.47708511352539, "loss_positions": 512, "loss_finite": true, "grads_finite": true, "grads_present": 610, "expert_fetches": 69120, "expert_bytes_read": 1570336174080, "layer_loads": 120, "os_io": {"process_read_bytes": 1587653703774, "disk_read_bytes": {"PhysicalDrive0": 1722067456, "PhysicalDrive1": 1586785970176}}, "peak_vram_mib": 4342, "peak_reserved_mib": 5052, "peak_host_bytes_so_far": 7759900672}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/step.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/step.jsonnew file mode 100644index 00000000..9401441e--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/step.json@@ -0,0 +1,77 @@+{+ "label": "random weights, shape only; routing forced uniform (token t -> experts kt..kt+k-1 mod E)",+ "args": {+ "store": "D:/soup-bench/moe-k2-synth",+ "weights": null,+ "model": null,+ "expert_layout": "per_expert",+ "out": "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\a2\\step.json",+ "seq": 512,+ "batch": 1,+ "steps": 1,+ "loss_tokens": 512,+ "lora_r": 8,+ "buffers": 2,+ "tier": "disk",+ "no_pin": true,+ "seed": 3,+ "label": "random weights, shape only",+ "route": "uniform"+ },+ "steps": [+ {+ "step": 0,+ "seconds": 1036.3,+ "tokens_per_second": 0.494,+ "loss": 13.47708511352539,+ "loss_positions": 512,+ "loss_finite": true,+ "grads_finite": true,+ "grads_present": 610,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080,+ "layer_loads": 120,+ "os_io": {+ "process_read_bytes": 1587653703774,+ "disk_read_bytes": {+ "PhysicalDrive0": 1722067456,+ "PhysicalDrive1": 1586785970176+ }+ },+ "peak_vram_mib": 4342,+ "peak_reserved_mib": 5052,+ "peak_host_bytes_so_far": 7759900672+ }+ ],+ "store_bytes": 533242299230,+ "store_expert_layout": "per_expert",+ "store_quant": "nf4",+ "store_double_quant": true,+ "build_seconds": 6.37,+ "routers_overridden": 60,+ "trainable_params": 30513664,+ "stats": {+ "n_layers": 61,+ "buffers": 2,+ "buffer_bytes": 2918596920,+ "large_buffer_bytes": 2348810240,+ "store_bytes": 5115134896,+ "pinned": false,+ "pinned_bytes": 0,+ "tier": "disk",+ "read_ahead": 2,+ "disk_bytes": 9779642448,+ "drive_count": 1,+ "direct_io_drives": 1,+ "direct_io_roots": {+ "D:/soup-bench/moe-k2-synth": true+ },+ "layer_loads": 120,+ "large_loads": 3,+ "device": "cuda",+ "total_params": 1026408232448,+ "nf4_scale_cache_bytes": 0,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080+ }+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/watchdog.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/watchdog.jsonnew file mode 100644index 00000000..6a5f4b3b--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/watchdog.json@@ -0,0 +1,39 @@+{+ "started": "2026-10-23T15:02:58.738+00:00",+ "match": null,+ "min_free_gb": 1.5,+ "period_s": 5.0,+ "fired": false,+ "gaps": [],+ "off_ac_samples": 0,+ "foreign_compute_pids": [],+ "pid": 21972,+ "cmdline": [+ "C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\python.exe",+ "benchmarks/harness/moe_stream_step.py",+ "--store",+ "D:/soup-bench/moe-k2-synth",+ "--tier",+ "disk",+ "--no-pin",+ "--route",+ "uniform",+ "--batch",+ "1",+ "--steps",+ "1",+ "--loss-tokens",+ "512",+ "--seq",+ "512",+ "--out",+ "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\a2\\step.json"+ ],+ "ended": "2026-10-23T15:20:45.835+00:00",+ "samples": 213,+ "min_free_ram_gb": 7.723,+ "max_tree_rss_gb": 7.705,+ "peak_wset_gb": 7.729,+ "max_gpu_memory_used_mib": 5226.0,+ "max_gpu_temperature_c": 48.0+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/watchdog.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/watchdog.jsonlnew file mode 100644index 00000000..8e751166--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/a2/watchdog.jsonl@@ -0,0 +1,213 @@+{"t": "2026-10-23T15:02:58.963+00:00", "free_ram_gb": 15.543, "tree_rss_gb": 0.022, "peak_wset_gb": 0.022, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 45.0, "clocks.sm": 1417.0, "power.draw": 12.45, "compute_pids": []}}+{"t": "2026-10-23T15:03:04.647+00:00", "free_ram_gb": 14.934, "tree_rss_gb": 0.564, "peak_wset_gb": 0.566, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 1815.0, "power.draw": 10.96, "compute_pids": []}}+{"t": "2026-10-23T15:03:09.606+00:00", "free_ram_gb": 14.736, "tree_rss_gb": 0.624, "peak_wset_gb": 0.624, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 0.0, "power.draw": 11.02, "compute_pids": []}}+{"t": "2026-10-23T15:03:14.643+00:00", "free_ram_gb": 14.709, "tree_rss_gb": 0.781, "peak_wset_gb": 0.781, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 0.0, "power.draw": 10.96, "compute_pids": []}}+{"t": "2026-10-23T15:03:19.665+00:00", "free_ram_gb": 14.56, "tree_rss_gb": 0.891, "peak_wset_gb": 0.891, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 12.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 0.0, "power.draw": 11.08, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:23.938+00:00", "free_ram_gb": 14.439, "tree_rss_gb": 1.092, "peak_wset_gb": 1.092, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 232.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 1417.0, "power.draw": 11.13, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:28.932+00:00", "free_ram_gb": 8.369, "tree_rss_gb": 6.935, "peak_wset_gb": 6.948, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3680.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 48.0, "clocks.sm": 1432.0, "power.draw": 26.0, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:33.957+00:00", "free_ram_gb": 8.489, "tree_rss_gb": 7.129, "peak_wset_gb": 7.143, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 15.5, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:38.952+00:00", "free_ram_gb": 8.479, "tree_rss_gb": 7.132, "peak_wset_gb": 7.143, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 922.0, "power.draw": 15.22, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:44.022+00:00", "free_ram_gb": 8.483, "tree_rss_gb": 7.136, "peak_wset_gb": 7.143, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.69, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:49.006+00:00", "free_ram_gb": 8.445, "tree_rss_gb": 7.139, "peak_wset_gb": 7.143, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 14.95, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:54.047+00:00", "free_ram_gb": 8.43, "tree_rss_gb": 7.142, "peak_wset_gb": 7.143, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1125.0, "power.draw": 15.43, "compute_pids": [21972]}}+{"t": "2026-10-23T15:03:59.061+00:00", "free_ram_gb": 8.426, "tree_rss_gb": 7.145, "peak_wset_gb": 7.145, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 952.0, "power.draw": 15.62, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:04.101+00:00", "free_ram_gb": 8.466, "tree_rss_gb": 7.148, "peak_wset_gb": 7.148, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1080.0, "power.draw": 15.4, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:09.061+00:00", "free_ram_gb": 8.462, "tree_rss_gb": 7.151, "peak_wset_gb": 7.151, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1132.0, "power.draw": 15.64, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:14.047+00:00", "free_ram_gb": 8.456, "tree_rss_gb": 7.155, "peak_wset_gb": 7.155, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1132.0, "power.draw": 15.39, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:19.112+00:00", "free_ram_gb": 8.437, "tree_rss_gb": 7.158, "peak_wset_gb": 7.158, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 15.99, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:24.077+00:00", "free_ram_gb": 8.42, "tree_rss_gb": 7.161, "peak_wset_gb": 7.161, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1125.0, "power.draw": 15.43, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:29.100+00:00", "free_ram_gb": 8.41, "tree_rss_gb": 7.164, "peak_wset_gb": 7.164, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.18, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:34.095+00:00", "free_ram_gb": 8.407, "tree_rss_gb": 7.167, "peak_wset_gb": 7.167, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 15.07, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:39.132+00:00", "free_ram_gb": 8.399, "tree_rss_gb": 7.17, "peak_wset_gb": 7.17, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.52, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:44.113+00:00", "free_ram_gb": 8.403, "tree_rss_gb": 7.173, "peak_wset_gb": 7.173, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.29, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:49.152+00:00", "free_ram_gb": 8.391, "tree_rss_gb": 7.177, "peak_wset_gb": 7.177, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 885.0, "power.draw": 15.29, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:54.128+00:00", "free_ram_gb": 8.413, "tree_rss_gb": 7.18, "peak_wset_gb": 7.18, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1012.0, "power.draw": 15.71, "compute_pids": [21972]}}+{"t": "2026-10-23T15:04:59.186+00:00", "free_ram_gb": 8.413, "tree_rss_gb": 7.183, "peak_wset_gb": 7.183, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1132.0, "power.draw": 15.13, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:04.191+00:00", "free_ram_gb": 8.392, "tree_rss_gb": 7.186, "peak_wset_gb": 7.186, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 48.0, "clocks.sm": 1102.0, "power.draw": 15.65, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:09.148+00:00", "free_ram_gb": 8.372, "tree_rss_gb": 7.189, "peak_wset_gb": 7.189, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 982.0, "power.draw": 15.48, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:14.176+00:00", "free_ram_gb": 8.385, "tree_rss_gb": 7.192, "peak_wset_gb": 7.192, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 48.0, "clocks.sm": 1140.0, "power.draw": 16.26, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:19.171+00:00", "free_ram_gb": 8.372, "tree_rss_gb": 7.196, "peak_wset_gb": 7.196, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.4, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:24.226+00:00", "free_ram_gb": 8.371, "tree_rss_gb": 7.198, "peak_wset_gb": 7.198, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 945.0, "power.draw": 14.34, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:29.207+00:00", "free_ram_gb": 8.359, "tree_rss_gb": 7.202, "peak_wset_gb": 7.202, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 14.91, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:34.254+00:00", "free_ram_gb": 8.435, "tree_rss_gb": 7.205, "peak_wset_gb": 7.205, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 13.12, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:39.245+00:00", "free_ram_gb": 8.394, "tree_rss_gb": 7.203, "peak_wset_gb": 7.206, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 1057.0, "power.draw": 14.91, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:44.236+00:00", "free_ram_gb": 8.421, "tree_rss_gb": 7.207, "peak_wset_gb": 7.207, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1005.0, "power.draw": 15.23, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:49.208+00:00", "free_ram_gb": 8.414, "tree_rss_gb": 7.211, "peak_wset_gb": 7.211, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 997.0, "power.draw": 15.12, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:54.259+00:00", "free_ram_gb": 8.406, "tree_rss_gb": 7.214, "peak_wset_gb": 7.214, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.42, "compute_pids": [21972]}}+{"t": "2026-10-23T15:05:59.302+00:00", "free_ram_gb": 8.361, "tree_rss_gb": 7.217, "peak_wset_gb": 7.217, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1027.0, "power.draw": 14.9, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:04.291+00:00", "free_ram_gb": 8.389, "tree_rss_gb": 7.22, "peak_wset_gb": 7.22, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 48.0, "clocks.sm": 1162.0, "power.draw": 15.33, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:09.270+00:00", "free_ram_gb": 8.381, "tree_rss_gb": 7.223, "peak_wset_gb": 7.223, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 1140.0, "power.draw": 15.61, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:14.313+00:00", "free_ram_gb": 8.377, "tree_rss_gb": 7.227, "peak_wset_gb": 7.227, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1132.0, "power.draw": 16.26, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:19.325+00:00", "free_ram_gb": 8.382, "tree_rss_gb": 7.23, "peak_wset_gb": 7.23, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1117.0, "power.draw": 15.18, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:24.810+00:00", "free_ram_gb": 8.364, "tree_rss_gb": 7.233, "peak_wset_gb": 7.233, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1170.0, "power.draw": 15.47, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:29.304+00:00", "free_ram_gb": 8.377, "tree_rss_gb": 7.236, "peak_wset_gb": 7.236, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.43, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:34.296+00:00", "free_ram_gb": 8.269, "tree_rss_gb": 7.239, "peak_wset_gb": 7.239, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.25, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:39.330+00:00", "free_ram_gb": 8.392, "tree_rss_gb": 7.242, "peak_wset_gb": 7.242, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.12, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:44.311+00:00", "free_ram_gb": 8.487, "tree_rss_gb": 7.245, "peak_wset_gb": 7.245, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 15.42, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:49.324+00:00", "free_ram_gb": 8.45, "tree_rss_gb": 7.249, "peak_wset_gb": 7.249, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1162.0, "power.draw": 15.24, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:54.347+00:00", "free_ram_gb": 8.435, "tree_rss_gb": 7.252, "peak_wset_gb": 7.252, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 15.49, "compute_pids": [21972]}}+{"t": "2026-10-23T15:06:59.340+00:00", "free_ram_gb": 8.439, "tree_rss_gb": 7.255, "peak_wset_gb": 7.255, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1102.0, "power.draw": 15.99, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:04.348+00:00", "free_ram_gb": 8.447, "tree_rss_gb": 7.259, "peak_wset_gb": 7.259, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1132.0, "power.draw": 16.37, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:09.385+00:00", "free_ram_gb": 8.433, "tree_rss_gb": 7.261, "peak_wset_gb": 7.261, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 997.0, "power.draw": 14.84, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:14.392+00:00", "free_ram_gb": 8.422, "tree_rss_gb": 7.264, "peak_wset_gb": 7.264, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 14.89, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:19.379+00:00", "free_ram_gb": 8.038, "tree_rss_gb": 7.268, "peak_wset_gb": 7.268, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 945.0, "power.draw": 13.94, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:25.321+00:00", "free_ram_gb": 8.175, "tree_rss_gb": 7.27, "peak_wset_gb": 7.27, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1125.0, "power.draw": 15.62, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:29.553+00:00", "free_ram_gb": 8.239, "tree_rss_gb": 7.273, "peak_wset_gb": 7.273, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1102.0, "power.draw": 15.6, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:34.440+00:00", "free_ram_gb": 8.229, "tree_rss_gb": 7.277, "peak_wset_gb": 7.277, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.38, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:39.429+00:00", "free_ram_gb": 8.115, "tree_rss_gb": 7.279, "peak_wset_gb": 7.279, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 14.23, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:44.529+00:00", "free_ram_gb": 8.169, "tree_rss_gb": 7.282, "peak_wset_gb": 7.282, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3682.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 47.0, "clocks.sm": 907.0, "power.draw": 13.79, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:49.987+00:00", "free_ram_gb": 8.205, "tree_rss_gb": 7.284, "peak_wset_gb": 7.284, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 48.0, "clocks.sm": 892.0, "power.draw": 13.18, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:54.498+00:00", "free_ram_gb": 8.219, "tree_rss_gb": 7.285, "peak_wset_gb": 7.285, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 47.0, "clocks.sm": 802.0, "power.draw": 12.71, "compute_pids": [21972]}}+{"t": "2026-10-23T15:07:59.544+00:00", "free_ram_gb": 8.186, "tree_rss_gb": 7.288, "peak_wset_gb": 7.288, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 47.0, "clocks.sm": 802.0, "power.draw": 12.73, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:04.546+00:00", "free_ram_gb": 8.196, "tree_rss_gb": 7.29, "peak_wset_gb": 7.29, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 47.0, "clocks.sm": 960.0, "power.draw": 13.19, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:10.050+00:00", "free_ram_gb": 8.191, "tree_rss_gb": 7.292, "peak_wset_gb": 7.292, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 47.0, "clocks.sm": 892.0, "power.draw": 12.68, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:14.603+00:00", "free_ram_gb": 8.242, "tree_rss_gb": 7.294, "peak_wset_gb": 7.294, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 15.52, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:19.596+00:00", "free_ram_gb": 8.24, "tree_rss_gb": 7.297, "peak_wset_gb": 7.297, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.57, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:24.884+00:00", "free_ram_gb": 8.241, "tree_rss_gb": 7.3, "peak_wset_gb": 7.3, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1102.0, "power.draw": 15.52, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:29.709+00:00", "free_ram_gb": 8.24, "tree_rss_gb": 7.303, "peak_wset_gb": 7.303, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 15.13, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:35.194+00:00", "free_ram_gb": 8.239, "tree_rss_gb": 7.306, "peak_wset_gb": 7.306, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 15.02, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:39.610+00:00", "free_ram_gb": 8.23, "tree_rss_gb": 7.31, "peak_wset_gb": 7.31, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.59, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:45.222+00:00", "free_ram_gb": 8.251, "tree_rss_gb": 7.313, "peak_wset_gb": 7.313, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.03, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:49.795+00:00", "free_ram_gb": 8.242, "tree_rss_gb": 7.316, "peak_wset_gb": 7.316, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 15.24, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:54.658+00:00", "free_ram_gb": 8.205, "tree_rss_gb": 7.319, "peak_wset_gb": 7.319, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 15.54, "compute_pids": [21972]}}+{"t": "2026-10-23T15:08:59.813+00:00", "free_ram_gb": 7.986, "tree_rss_gb": 7.322, "peak_wset_gb": 7.322, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 945.0, "power.draw": 15.16, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:05.132+00:00", "free_ram_gb": 7.985, "tree_rss_gb": 7.325, "peak_wset_gb": 7.325, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1102.0, "power.draw": 13.63, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:09.897+00:00", "free_ram_gb": 7.98, "tree_rss_gb": 7.328, "peak_wset_gb": 7.328, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 14.92, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:14.720+00:00", "free_ram_gb": 7.966, "tree_rss_gb": 7.331, "peak_wset_gb": 7.331, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3794.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 15.05, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:19.736+00:00", "free_ram_gb": 7.9, "tree_rss_gb": 7.412, "peak_wset_gb": 7.412, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4824.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 975.0, "power.draw": 15.34, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:25.448+00:00", "free_ram_gb": 7.74, "tree_rss_gb": 7.576, "peak_wset_gb": 7.577, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4878.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 16.57, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:29.948+00:00", "free_ram_gb": 7.723, "tree_rss_gb": 7.604, "peak_wset_gb": 7.604, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4878.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1012.0, "power.draw": 15.52, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:34.752+00:00", "free_ram_gb": 7.943, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4878.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 1027.0, "power.draw": 13.3, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:40.445+00:00", "free_ram_gb": 7.953, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4880.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1087.0, "power.draw": 16.82, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:45.001+00:00", "free_ram_gb": 7.987, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4880.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 997.0, "power.draw": 14.8, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:49.756+00:00", "free_ram_gb": 7.967, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4882.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 48.0, "clocks.sm": 1057.0, "power.draw": 16.3, "compute_pids": [21972]}}+{"t": "2026-10-23T15:09:54.791+00:00", "free_ram_gb": 7.982, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4882.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1050.0, "power.draw": 15.11, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:00.045+00:00", "free_ram_gb": 7.887, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4884.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.15, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:05.564+00:00", "free_ram_gb": 7.886, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4884.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.25, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:09.800+00:00", "free_ram_gb": 7.887, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4886.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 16.76, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:14.843+00:00", "free_ram_gb": 7.878, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4886.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1252.0, "power.draw": 17.75, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:20.125+00:00", "free_ram_gb": 7.861, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4888.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1027.0, "power.draw": 15.33, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:25.157+00:00", "free_ram_gb": 7.869, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4888.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.96, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:29.868+00:00", "free_ram_gb": 7.942, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4888.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 15.36, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:34.927+00:00", "free_ram_gb": 7.957, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4890.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 16.81, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:39.910+00:00", "free_ram_gb": 7.949, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4890.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 967.0, "power.draw": 15.13, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:44.882+00:00", "free_ram_gb": 7.953, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4892.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 16.2, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:49.950+00:00", "free_ram_gb": 7.952, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4892.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 997.0, "power.draw": 15.27, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:54.953+00:00", "free_ram_gb": 7.952, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4894.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1102.0, "power.draw": 16.39, "compute_pids": [21972]}}+{"t": "2026-10-23T15:10:59.966+00:00", "free_ram_gb": 7.951, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4894.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 16.37, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:04.957+00:00", "free_ram_gb": 7.907, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4896.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.72, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:09.986+00:00", "free_ram_gb": 7.906, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4896.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1170.0, "power.draw": 17.13, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:14.999+00:00", "free_ram_gb": 7.902, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4896.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.23, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:20.014+00:00", "free_ram_gb": 7.905, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4898.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 16.98, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:25.002+00:00", "free_ram_gb": 7.879, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4898.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 997.0, "power.draw": 15.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:29.983+00:00", "free_ram_gb": 7.883, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4900.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1027.0, "power.draw": 17.03, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:35.040+00:00", "free_ram_gb": 7.879, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4900.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 1012.0, "power.draw": 14.86, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:40.016+00:00", "free_ram_gb": 7.876, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4902.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.62, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:45.018+00:00", "free_ram_gb": 7.876, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4902.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.04, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:50.068+00:00", "free_ram_gb": 7.876, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4904.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 16.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:11:55.999+00:00", "free_ram_gb": 7.917, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4904.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 915.0, "power.draw": 14.61, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:01.005+00:00", "free_ram_gb": 7.917, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4906.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1095.0, "power.draw": 15.98, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:05.099+00:00", "free_ram_gb": 7.932, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4906.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 48.0, "clocks.sm": 1065.0, "power.draw": 16.65, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:10.063+00:00", "free_ram_gb": 7.915, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4906.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 15.38, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:15.573+00:00", "free_ram_gb": 7.868, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4908.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 48.0, "clocks.sm": 1102.0, "power.draw": 16.95, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:20.108+00:00", "free_ram_gb": 7.89, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4908.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.27, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:25.100+00:00", "free_ram_gb": 7.885, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4910.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 48.0, "clocks.sm": 1147.0, "power.draw": 16.48, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:30.082+00:00", "free_ram_gb": 7.885, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4910.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 16.35, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:35.620+00:00", "free_ram_gb": 7.885, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4912.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 15.3, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:40.170+00:00", "free_ram_gb": 7.815, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4912.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 48.0, "clocks.sm": 1080.0, "power.draw": 16.45, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:45.135+00:00", "free_ram_gb": 7.879, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4912.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 15.06, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:50.127+00:00", "free_ram_gb": 7.882, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4914.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 48.0, "clocks.sm": 1035.0, "power.draw": 17.01, "compute_pids": [21972]}}+{"t": "2026-10-23T15:12:55.170+00:00", "free_ram_gb": 7.874, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4914.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 990.0, "power.draw": 14.8, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:00.158+00:00", "free_ram_gb": 7.875, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4916.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1155.0, "power.draw": 16.7, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:05.158+00:00", "free_ram_gb": 7.872, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4916.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.21, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:10.142+00:00", "free_ram_gb": 7.874, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4918.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1050.0, "power.draw": 15.6, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:15.139+00:00", "free_ram_gb": 7.873, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4918.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 16.74, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:20.156+00:00", "free_ram_gb": 7.84, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4918.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 47.0, "clocks.sm": 877.0, "power.draw": 14.42, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:25.206+00:00", "free_ram_gb": 7.826, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4920.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 967.0, "power.draw": 15.82, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:30.208+00:00", "free_ram_gb": 7.868, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4920.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 952.0, "power.draw": 14.27, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:35.230+00:00", "free_ram_gb": 7.847, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4922.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1102.0, "power.draw": 15.69, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:40.213+00:00", "free_ram_gb": 7.84, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4922.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 15.85, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:45.487+00:00", "free_ram_gb": 7.82, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4922.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 997.0, "power.draw": 14.71, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:50.203+00:00", "free_ram_gb": 7.769, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4924.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.22, "compute_pids": [21972]}}+{"t": "2026-10-23T15:13:55.218+00:00", "free_ram_gb": 7.831, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4924.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1050.0, "power.draw": 15.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:00.219+00:00", "free_ram_gb": 7.841, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4926.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 16.68, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:05.186+00:00", "free_ram_gb": 7.874, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4926.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 17.08, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:10.221+00:00", "free_ram_gb": 7.893, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4928.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 15.83, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:15.252+00:00", "free_ram_gb": 7.914, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4928.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1140.0, "power.draw": 16.03, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:20.268+00:00", "free_ram_gb": 7.914, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4928.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 15.48, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:25.732+00:00", "free_ram_gb": 7.87, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4930.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1050.0, "power.draw": 16.21, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:30.210+00:00", "free_ram_gb": 7.909, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4930.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1012.0, "power.draw": 15.07, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:35.711+00:00", "free_ram_gb": 7.873, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4932.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1170.0, "power.draw": 16.33, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:40.236+00:00", "free_ram_gb": 7.894, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4932.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 47.0, "clocks.sm": 982.0, "power.draw": 17.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:45.682+00:00", "free_ram_gb": 7.872, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4934.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 48.0, "clocks.sm": 1035.0, "power.draw": 14.89, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:50.257+00:00", "free_ram_gb": 7.855, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4934.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.85, "compute_pids": [21972]}}+{"t": "2026-10-23T15:14:55.533+00:00", "free_ram_gb": 7.829, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4934.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.12, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:00.720+00:00", "free_ram_gb": 7.845, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4936.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.94, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:05.510+00:00", "free_ram_gb": 7.841, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4936.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 47.0, "clocks.sm": 975.0, "power.draw": 14.07, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:10.686+00:00", "free_ram_gb": 7.859, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4938.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 14.81, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:15.346+00:00", "free_ram_gb": 7.883, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4938.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 16.11, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:20.349+00:00", "free_ram_gb": 7.835, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4938.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 14.74, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:25.354+00:00", "free_ram_gb": 7.897, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4940.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 47.0, "clocks.sm": 1177.0, "power.draw": 16.9, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:30.388+00:00", "free_ram_gb": 7.904, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4940.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:35.380+00:00", "free_ram_gb": 7.888, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4942.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 48.0, "clocks.sm": 1080.0, "power.draw": 17.31, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:40.379+00:00", "free_ram_gb": 7.916, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4942.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.39, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:45.431+00:00", "free_ram_gb": 7.918, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4944.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 17.25, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:50.455+00:00", "free_ram_gb": 7.914, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4944.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 16.68, "compute_pids": [21972]}}+{"t": "2026-10-23T15:15:55.472+00:00", "free_ram_gb": 7.916, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4946.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1005.0, "power.draw": 16.25, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:00.448+00:00", "free_ram_gb": 7.913, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4946.0, "memory.total": 8151.0, "utilization.gpu": 27.0, "temperature.gpu": 47.0, "clocks.sm": 1200.0, "power.draw": 18.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:05.910+00:00", "free_ram_gb": 7.917, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4946.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.2, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:10.907+00:00", "free_ram_gb": 7.955, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4948.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 16.56, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:15.505+00:00", "free_ram_gb": 7.959, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4948.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 15.41, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:20.495+00:00", "free_ram_gb": 7.96, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4950.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1102.0, "power.draw": 16.45, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:25.995+00:00", "free_ram_gb": 7.952, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4950.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.2, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:30.526+00:00", "free_ram_gb": 7.95, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4952.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.6, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:35.623+00:00", "free_ram_gb": 7.831, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4952.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 982.0, "power.draw": 15.35, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:40.475+00:00", "free_ram_gb": 7.907, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4954.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.46, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:45.571+00:00", "free_ram_gb": 7.995, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4954.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1237.0, "power.draw": 17.49, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:51.073+00:00", "free_ram_gb": 7.992, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4954.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.64, "compute_pids": [21972]}}+{"t": "2026-10-23T15:16:55.517+00:00", "free_ram_gb": 7.958, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4956.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.65, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:00.537+00:00", "free_ram_gb": 7.951, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4956.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 15.25, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:05.503+00:00", "free_ram_gb": 7.954, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4958.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1132.0, "power.draw": 16.55, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:11.157+00:00", "free_ram_gb": 7.955, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4958.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 15.23, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:15.608+00:00", "free_ram_gb": 7.951, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4960.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.61, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:20.587+00:00", "free_ram_gb": 7.929, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4960.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.14, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:25.742+00:00", "free_ram_gb": 7.941, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4962.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1147.0, "power.draw": 16.76, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:31.228+00:00", "free_ram_gb": 7.943, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4962.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1035.0, "power.draw": 17.38, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:35.634+00:00", "free_ram_gb": 7.94, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4964.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.55, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:40.629+00:00", "free_ram_gb": 7.932, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4964.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1185.0, "power.draw": 17.75, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:45.799+00:00", "free_ram_gb": 7.936, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4966.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1005.0, "power.draw": 15.1, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:50.678+00:00", "free_ram_gb": 7.936, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4966.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.92, "compute_pids": [21972]}}+{"t": "2026-10-23T15:17:55.676+00:00", "free_ram_gb": 7.933, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4966.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1005.0, "power.draw": 15.43, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:00.705+00:00", "free_ram_gb": 7.934, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4968.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 16.3, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:06.129+00:00", "free_ram_gb": 7.921, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4968.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 15.56, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:10.688+00:00", "free_ram_gb": 7.933, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4970.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1147.0, "power.draw": 17.0, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:15.778+00:00", "free_ram_gb": 7.958, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4970.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 14.94, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:20.792+00:00", "free_ram_gb": 7.96, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4972.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1050.0, "power.draw": 16.25, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:25.750+00:00", "free_ram_gb": 7.965, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4972.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 17.17, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:30.783+00:00", "free_ram_gb": 7.966, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4974.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 990.0, "power.draw": 15.29, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:35.780+00:00", "free_ram_gb": 7.996, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4974.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 16.83, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:40.769+00:00", "free_ram_gb": 7.996, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4974.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 46.0, "clocks.sm": 1072.0, "power.draw": 14.73, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:45.749+00:00", "free_ram_gb": 8.0, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4976.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 16.84, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:50.771+00:00", "free_ram_gb": 7.999, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4976.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 47.0, "clocks.sm": 967.0, "power.draw": 15.15, "compute_pids": [21972]}}+{"t": "2026-10-23T15:18:55.810+00:00", "free_ram_gb": 8.01, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4978.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 16.65, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:01.122+00:00", "free_ram_gb": 8.007, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4978.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 15.11, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:05.799+00:00", "free_ram_gb": 8.007, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4980.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 16.21, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:10.808+00:00", "free_ram_gb": 8.005, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4980.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1147.0, "power.draw": 15.39, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:15.820+00:00", "free_ram_gb": 8.006, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4982.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1110.0, "power.draw": 15.69, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:21.649+00:00", "free_ram_gb": 8.004, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4982.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1080.0, "power.draw": 16.07, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:25.841+00:00", "free_ram_gb": 8.002, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4982.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 47.0, "clocks.sm": 1087.0, "power.draw": 15.61, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:30.858+00:00", "free_ram_gb": 8.003, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1095.0, "power.draw": 16.53, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:36.212+00:00", "free_ram_gb": 7.871, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 967.0, "power.draw": 14.65, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:40.877+00:00", "free_ram_gb": 7.775, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4986.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 16.26, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:45.834+00:00", "free_ram_gb": 7.78, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4986.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 15.07, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:50.905+00:00", "free_ram_gb": 7.785, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4988.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1072.0, "power.draw": 16.73, "compute_pids": [21972]}}+{"t": "2026-10-23T15:19:56.311+00:00", "free_ram_gb": 7.787, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4988.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 47.0, "clocks.sm": 1020.0, "power.draw": 16.8, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:01.791+00:00", "free_ram_gb": 7.781, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4990.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1177.0, "power.draw": 16.23, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:06.441+00:00", "free_ram_gb": 7.79, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4990.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1125.0, "power.draw": 16.3, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:10.937+00:00", "free_ram_gb": 8.035, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4990.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 47.0, "clocks.sm": 1117.0, "power.draw": 15.5, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:16.391+00:00", "free_ram_gb": 8.037, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4992.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 16.73, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:21.432+00:00", "free_ram_gb": 8.024, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4992.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 47.0, "clocks.sm": 1042.0, "power.draw": 15.04, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:25.959+00:00", "free_ram_gb": 7.921, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4994.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 47.0, "clocks.sm": 1170.0, "power.draw": 17.22, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:31.413+00:00", "free_ram_gb": 7.917, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4994.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 47.0, "clocks.sm": 1057.0, "power.draw": 15.13, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:36.421+00:00", "free_ram_gb": 7.909, "tree_rss_gb": 7.606, "peak_wset_gb": 7.606, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4996.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 47.0, "clocks.sm": 1065.0, "power.draw": 16.2, "compute_pids": [21972]}}+{"t": "2026-10-23T15:20:41.442+00:00", "free_ram_gb": 7.81, "tree_rss_gb": 7.705, "peak_wset_gb": 7.729, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5226.0, "memory.total": 8151.0, "utilization.gpu": 28.0, "temperature.gpu": 48.0, "clocks.sm": 622.0, "power.draw": 18.93, "compute_pids": [21972]}}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/box.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/box.jsonlnew file mode 100644index 00000000..e7565b52--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/box.jsonl@@ -0,0 +1,2 @@+{"at":"before","utc":"2026-10-23T14:23:49.1934040Z","mem":{"AvailableBytes":10469019648,"CommittedBytes":32815849472,"PagesInputPersec":198811138},"ac":2,"gpu":"0 MiB, 50, 742 MHz"}+{"at":"after","utc":"2026-10-23T14:42:58.4769925Z","mem":{"AvailableBytes":14853009408,"CommittedBytes":28918206464,"PagesInputPersec":199450515},"ac":2,"gpu":"0 MiB, 49, 0 MHz"}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/exit.txt b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/exit.txtnew file mode 100644index 00000000..ca916d09--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/exit.txt@@ -0,0 +1 @@+exit 0diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/stderr.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/stderr.lognew file mode 100644index 00000000..4d72ac2a--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/stderr.log@@ -0,0 +1,2 @@+W1023 19:24:08.577000 22644 site-packages\torch\utils\flop_counter.py:113] triton not found; flop counting will not work for triton kernels+[transformers] `torch_dtype` is deprecated! Use `dtype` instead!diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/stdout.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/stdout.lognew file mode 100644index 00000000..9ae4b68f--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/stdout.log@@ -0,0 +1 @@+{"step": 0, "seconds": 1117.61, "tokens_per_second": 1.832, "loss": 13.491609573364258, "loss_positions": 512, "loss_finite": true, "grads_finite": true, "grads_present": 610, "expert_fetches": 69120, "expert_bytes_read": 1570336174080, "layer_loads": 120, "os_io": {"process_read_bytes": 1587638315120, "disk_read_bytes": {"PhysicalDrive0": 2457071616, "PhysicalDrive1": 1586785970176}}, "peak_vram_mib": 6091, "peak_reserved_mib": 7176, "peak_host_bytes_so_far": 7762006016}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/step.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/step.jsonnew file mode 100644index 00000000..0c5cb457--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/step.json@@ -0,0 +1,77 @@+{+ "label": "random weights, shape only; routing forced uniform (token t -> experts kt..kt+k-1 mod E)",+ "args": {+ "store": "D:/soup-bench/moe-k2-synth",+ "weights": null,+ "model": null,+ "expert_layout": "per_expert",+ "out": "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\b1\\step.json",+ "seq": 2048,+ "batch": 1,+ "steps": 1,+ "loss_tokens": 512,+ "lora_r": 8,+ "buffers": 2,+ "tier": "disk",+ "no_pin": true,+ "seed": 3,+ "label": "random weights, shape only",+ "route": "uniform"+ },+ "steps": [+ {+ "step": 0,+ "seconds": 1117.61,+ "tokens_per_second": 1.832,+ "loss": 13.491609573364258,+ "loss_positions": 512,+ "loss_finite": true,+ "grads_finite": true,+ "grads_present": 610,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080,+ "layer_loads": 120,+ "os_io": {+ "process_read_bytes": 1587638315120,+ "disk_read_bytes": {+ "PhysicalDrive0": 2457071616,+ "PhysicalDrive1": 1586785970176+ }+ },+ "peak_vram_mib": 6091,+ "peak_reserved_mib": 7176,+ "peak_host_bytes_so_far": 7762006016+ }+ ],+ "store_bytes": 533242299230,+ "store_expert_layout": "per_expert",+ "store_quant": "nf4",+ "store_double_quant": true,+ "build_seconds": 6.78,+ "routers_overridden": 60,+ "trainable_params": 30513664,+ "stats": {+ "n_layers": 61,+ "buffers": 2,+ "buffer_bytes": 2918596920,+ "large_buffer_bytes": 2348810240,+ "store_bytes": 5115134896,+ "pinned": false,+ "pinned_bytes": 0,+ "tier": "disk",+ "read_ahead": 2,+ "disk_bytes": 9779642448,+ "drive_count": 1,+ "direct_io_drives": 1,+ "direct_io_roots": {+ "D:/soup-bench/moe-k2-synth": true+ },+ "layer_loads": 120,+ "large_loads": 3,+ "device": "cuda",+ "total_params": 1026408232448,+ "nf4_scale_cache_bytes": 0,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080+ }+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/watchdog.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/watchdog.jsonnew file mode 100644index 00000000..fcbb1199--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/watchdog.json@@ -0,0 +1,39 @@+{+ "started": "2026-10-23T14:23:50.323+00:00",+ "match": null,+ "min_free_gb": 1.5,+ "period_s": 5.0,+ "fired": false,+ "gaps": [],+ "off_ac_samples": 0,+ "foreign_compute_pids": [],+ "pid": 22644,+ "cmdline": [+ "C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\python.exe",+ "benchmarks/harness/moe_stream_step.py",+ "--store",+ "D:/soup-bench/moe-k2-synth",+ "--tier",+ "disk",+ "--no-pin",+ "--route",+ "uniform",+ "--batch",+ "1",+ "--steps",+ "1",+ "--loss-tokens",+ "512",+ "--seq",+ "2048",+ "--out",+ "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\b1\\step.json"+ ],+ "ended": "2026-10-23T14:42:58.445+00:00",+ "samples": 229,+ "min_free_ram_gb": 3.472,+ "max_tree_rss_gb": 7.607,+ "peak_wset_gb": 7.608,+ "max_gpu_memory_used_mib": 7350.0,+ "max_gpu_temperature_c": 56.0+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/watchdog.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/watchdog.jsonlnew file mode 100644index 00000000..3a176e84--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b1/watchdog.jsonl@@ -0,0 +1,229 @@+{"t": "2026-10-23T14:23:50.566+00:00", "free_ram_gb": 10.606, "tree_rss_gb": 0.022, "peak_wset_gb": 0.022, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 50.0, "clocks.sm": 1417.0, "power.draw": 12.97, "compute_pids": []}}+{"t": "2026-10-23T14:23:56.191+00:00", "free_ram_gb": 9.976, "tree_rss_gb": 0.562, "peak_wset_gb": 0.562, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 50.0, "clocks.sm": 5782.0, "power.draw": 11.36, "compute_pids": []}}+{"t": "2026-10-23T14:24:01.205+00:00", "free_ram_gb": 9.987, "tree_rss_gb": 0.6, "peak_wset_gb": 0.6, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 50.0, "clocks.sm": 757.0, "power.draw": 11.3, "compute_pids": []}}+{"t": "2026-10-23T14:24:06.246+00:00", "free_ram_gb": 9.983, "tree_rss_gb": 0.759, "peak_wset_gb": 0.759, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 50.0, "clocks.sm": 0.0, "power.draw": 11.31, "compute_pids": []}}+{"t": "2026-10-23T14:24:11.251+00:00", "free_ram_gb": 9.644, "tree_rss_gb": 0.862, "peak_wset_gb": 0.862, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 51.0, "clocks.sm": 36470.0, "power.draw": 11.39, "compute_pids": []}}+{"t": "2026-10-23T14:24:15.497+00:00", "free_ram_gb": 9.428, "tree_rss_gb": 1.047, "peak_wset_gb": 1.047, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 232.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 51.0, "clocks.sm": 1417.0, "power.draw": 11.79, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:20.521+00:00", "free_ram_gb": 4.247, "tree_rss_gb": 6.52, "peak_wset_gb": 6.52, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3148.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 51.0, "clocks.sm": 1417.0, "power.draw": 11.49, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:25.628+00:00", "free_ram_gb": 3.825, "tree_rss_gb": 7.129, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 53.0, "clocks.sm": 1027.0, "power.draw": 14.07, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:30.547+00:00", "free_ram_gb": 3.879, "tree_rss_gb": 7.131, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1050.0, "power.draw": 14.73, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:35.594+00:00", "free_ram_gb": 4.022, "tree_rss_gb": 7.135, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1087.0, "power.draw": 15.7, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:40.578+00:00", "free_ram_gb": 4.075, "tree_rss_gb": 7.137, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1095.0, "power.draw": 14.75, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:45.589+00:00", "free_ram_gb": 4.205, "tree_rss_gb": 7.14, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 52.0, "clocks.sm": 1020.0, "power.draw": 15.48, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:50.616+00:00", "free_ram_gb": 4.203, "tree_rss_gb": 7.143, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 15.09, "compute_pids": [22644]}}+{"t": "2026-10-23T14:24:55.619+00:00", "free_ram_gb": 4.189, "tree_rss_gb": 7.146, "peak_wset_gb": 7.146, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 15.06, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:00.654+00:00", "free_ram_gb": 4.165, "tree_rss_gb": 7.148, "peak_wset_gb": 7.148, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 14.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:05.690+00:00", "free_ram_gb": 4.17, "tree_rss_gb": 7.151, "peak_wset_gb": 7.151, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 997.0, "power.draw": 14.31, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:10.648+00:00", "free_ram_gb": 4.055, "tree_rss_gb": 7.154, "peak_wset_gb": 7.154, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 15.08, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:15.674+00:00", "free_ram_gb": 4.073, "tree_rss_gb": 7.157, "peak_wset_gb": 7.157, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 982.0, "power.draw": 14.23, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:20.692+00:00", "free_ram_gb": 4.818, "tree_rss_gb": 7.16, "peak_wset_gb": 7.16, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 997.0, "power.draw": 15.15, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:25.709+00:00", "free_ram_gb": 4.777, "tree_rss_gb": 7.163, "peak_wset_gb": 7.163, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 15.4, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:30.709+00:00", "free_ram_gb": 5.109, "tree_rss_gb": 7.166, "peak_wset_gb": 7.166, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 16.28, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:35.708+00:00", "free_ram_gb": 4.79, "tree_rss_gb": 7.17, "peak_wset_gb": 7.17, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 16.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:40.758+00:00", "free_ram_gb": 4.679, "tree_rss_gb": 7.172, "peak_wset_gb": 7.172, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 15.17, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:45.733+00:00", "free_ram_gb": 4.505, "tree_rss_gb": 7.175, "peak_wset_gb": 7.175, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 53.0, "clocks.sm": 1072.0, "power.draw": 15.7, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:50.771+00:00", "free_ram_gb": 4.775, "tree_rss_gb": 7.178, "peak_wset_gb": 7.178, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 1072.0, "power.draw": 15.43, "compute_pids": [22644]}}+{"t": "2026-10-23T14:25:55.736+00:00", "free_ram_gb": 3.834, "tree_rss_gb": 7.181, "peak_wset_gb": 7.181, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 54.0, "clocks.sm": 1005.0, "power.draw": 15.94, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:00.827+00:00", "free_ram_gb": 4.031, "tree_rss_gb": 7.184, "peak_wset_gb": 7.184, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 54.0, "clocks.sm": 1072.0, "power.draw": 16.08, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:05.818+00:00", "free_ram_gb": 4.19, "tree_rss_gb": 7.186, "peak_wset_gb": 7.186, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 54.0, "clocks.sm": 1095.0, "power.draw": 14.94, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:10.826+00:00", "free_ram_gb": 4.353, "tree_rss_gb": 7.189, "peak_wset_gb": 7.189, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 53.0, "clocks.sm": 960.0, "power.draw": 15.93, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:15.848+00:00", "free_ram_gb": 5.032, "tree_rss_gb": 7.192, "peak_wset_gb": 7.192, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 53.0, "clocks.sm": 1072.0, "power.draw": 15.68, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:20.856+00:00", "free_ram_gb": 5.026, "tree_rss_gb": 7.195, "peak_wset_gb": 7.195, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 53.0, "clocks.sm": 1087.0, "power.draw": 16.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:25.855+00:00", "free_ram_gb": 5.118, "tree_rss_gb": 7.198, "peak_wset_gb": 7.198, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1005.0, "power.draw": 14.42, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:30.858+00:00", "free_ram_gb": 5.261, "tree_rss_gb": 7.2, "peak_wset_gb": 7.2, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1042.0, "power.draw": 15.39, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:35.858+00:00", "free_ram_gb": 4.788, "tree_rss_gb": 7.203, "peak_wset_gb": 7.203, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1027.0, "power.draw": 16.63, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:40.866+00:00", "free_ram_gb": 4.467, "tree_rss_gb": 7.206, "peak_wset_gb": 7.206, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 960.0, "power.draw": 15.2, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:45.876+00:00", "free_ram_gb": 4.709, "tree_rss_gb": 7.209, "peak_wset_gb": 7.209, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1035.0, "power.draw": 15.32, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:50.892+00:00", "free_ram_gb": 4.449, "tree_rss_gb": 7.212, "peak_wset_gb": 7.212, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1050.0, "power.draw": 14.88, "compute_pids": [22644]}}+{"t": "2026-10-23T14:26:55.884+00:00", "free_ram_gb": 4.413, "tree_rss_gb": 7.215, "peak_wset_gb": 7.215, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 930.0, "power.draw": 19.95, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:00.942+00:00", "free_ram_gb": 4.476, "tree_rss_gb": 7.218, "peak_wset_gb": 7.218, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 15.19, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:05.926+00:00", "free_ram_gb": 4.15, "tree_rss_gb": 7.221, "peak_wset_gb": 7.221, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 16.32, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:10.945+00:00", "free_ram_gb": 4.438, "tree_rss_gb": 7.224, "peak_wset_gb": 7.224, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 16.32, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:15.990+00:00", "free_ram_gb": 4.669, "tree_rss_gb": 7.227, "peak_wset_gb": 7.227, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 982.0, "power.draw": 15.97, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:20.999+00:00", "free_ram_gb": 5.1, "tree_rss_gb": 7.23, "peak_wset_gb": 7.23, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 952.0, "power.draw": 15.01, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:26.001+00:00", "free_ram_gb": 4.688, "tree_rss_gb": 7.233, "peak_wset_gb": 7.233, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 15.95, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:30.978+00:00", "free_ram_gb": 4.604, "tree_rss_gb": 7.237, "peak_wset_gb": 7.237, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 810.0, "power.draw": 14.2, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:35.999+00:00", "free_ram_gb": 4.878, "tree_rss_gb": 7.239, "peak_wset_gb": 7.239, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 56.0, "clocks.sm": 1305.0, "power.draw": 22.8, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:41.013+00:00", "free_ram_gb": 4.816, "tree_rss_gb": 7.242, "peak_wset_gb": 7.242, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 16.42, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:46.028+00:00", "free_ram_gb": 4.837, "tree_rss_gb": 7.245, "peak_wset_gb": 7.245, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 13.63, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:51.017+00:00", "free_ram_gb": 4.616, "tree_rss_gb": 7.249, "peak_wset_gb": 7.249, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 990.0, "power.draw": 15.11, "compute_pids": [22644]}}+{"t": "2026-10-23T14:27:56.042+00:00", "free_ram_gb": 4.794, "tree_rss_gb": 7.247, "peak_wset_gb": 7.249, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 990.0, "power.draw": 15.24, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:01.042+00:00", "free_ram_gb": 4.559, "tree_rss_gb": 7.25, "peak_wset_gb": 7.25, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 8.49, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:06.085+00:00", "free_ram_gb": 4.632, "tree_rss_gb": 7.253, "peak_wset_gb": 7.253, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 15.56, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:11.044+00:00", "free_ram_gb": 4.925, "tree_rss_gb": 7.256, "peak_wset_gb": 7.256, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 13.61, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:16.085+00:00", "free_ram_gb": 4.488, "tree_rss_gb": 7.26, "peak_wset_gb": 7.26, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 15.82, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:21.071+00:00", "free_ram_gb": 4.849, "tree_rss_gb": 7.263, "peak_wset_gb": 7.263, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 17.85, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:26.094+00:00", "free_ram_gb": 4.887, "tree_rss_gb": 7.266, "peak_wset_gb": 7.266, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 16.02, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:31.117+00:00", "free_ram_gb": 4.626, "tree_rss_gb": 7.269, "peak_wset_gb": 7.269, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 15.68, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:36.133+00:00", "free_ram_gb": 4.605, "tree_rss_gb": 7.273, "peak_wset_gb": 7.273, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 21.26, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:41.144+00:00", "free_ram_gb": 4.461, "tree_rss_gb": 7.276, "peak_wset_gb": 7.276, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 16.4, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:46.203+00:00", "free_ram_gb": 4.517, "tree_rss_gb": 7.279, "peak_wset_gb": 7.279, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 15.52, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:51.204+00:00", "free_ram_gb": 4.345, "tree_rss_gb": 7.282, "peak_wset_gb": 7.282, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 15.95, "compute_pids": [22644]}}+{"t": "2026-10-23T14:28:56.179+00:00", "free_ram_gb": 4.258, "tree_rss_gb": 7.285, "peak_wset_gb": 7.285, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 15.81, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:01.182+00:00", "free_ram_gb": 4.138, "tree_rss_gb": 7.289, "peak_wset_gb": 7.289, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 53.0, "clocks.sm": 1125.0, "power.draw": 16.94, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:06.243+00:00", "free_ram_gb": 3.517, "tree_rss_gb": 7.292, "peak_wset_gb": 7.292, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 55.0, "clocks.sm": 1020.0, "power.draw": 15.57, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:11.264+00:00", "free_ram_gb": 3.472, "tree_rss_gb": 7.295, "peak_wset_gb": 7.295, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 55.0, "clocks.sm": 990.0, "power.draw": 14.95, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:16.279+00:00", "free_ram_gb": 3.646, "tree_rss_gb": 7.298, "peak_wset_gb": 7.298, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 55.0, "clocks.sm": 1395.0, "power.draw": 15.56, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:21.253+00:00", "free_ram_gb": 4.195, "tree_rss_gb": 7.3, "peak_wset_gb": 7.3, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 54.0, "clocks.sm": 1050.0, "power.draw": 17.03, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:26.262+00:00", "free_ram_gb": 4.341, "tree_rss_gb": 7.304, "peak_wset_gb": 7.304, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 982.0, "power.draw": 17.01, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:31.270+00:00", "free_ram_gb": 4.638, "tree_rss_gb": 7.307, "peak_wset_gb": 7.307, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1072.0, "power.draw": 15.56, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:36.297+00:00", "free_ram_gb": 4.988, "tree_rss_gb": 7.309, "peak_wset_gb": 7.309, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 15.49, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:41.342+00:00", "free_ram_gb": 4.985, "tree_rss_gb": 7.313, "peak_wset_gb": 7.313, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 15.8, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:46.320+00:00", "free_ram_gb": 4.965, "tree_rss_gb": 7.315, "peak_wset_gb": 7.315, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 14.06, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:51.372+00:00", "free_ram_gb": 5.04, "tree_rss_gb": 7.318, "peak_wset_gb": 7.318, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 810.0, "power.draw": 14.17, "compute_pids": [22644]}}+{"t": "2026-10-23T14:29:56.384+00:00", "free_ram_gb": 4.983, "tree_rss_gb": 7.32, "peak_wset_gb": 7.32, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 15.23, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:01.430+00:00", "free_ram_gb": 4.86, "tree_rss_gb": 7.323, "peak_wset_gb": 7.323, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1005.0, "power.draw": 15.44, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:06.424+00:00", "free_ram_gb": 4.444, "tree_rss_gb": 7.326, "peak_wset_gb": 7.326, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 52.0, "clocks.sm": 990.0, "power.draw": 16.59, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:11.442+00:00", "free_ram_gb": 4.402, "tree_rss_gb": 7.329, "peak_wset_gb": 7.329, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1005.0, "power.draw": 16.04, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:16.528+00:00", "free_ram_gb": 4.657, "tree_rss_gb": 7.331, "peak_wset_gb": 7.331, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1042.0, "power.draw": 15.78, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:21.515+00:00", "free_ram_gb": 4.842, "tree_rss_gb": 7.413, "peak_wset_gb": 7.413, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 6796.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 990.0, "power.draw": 15.56, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:26.531+00:00", "free_ram_gb": 4.501, "tree_rss_gb": 7.416, "peak_wset_gb": 7.416, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7150.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1102.0, "power.draw": 15.74, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:31.572+00:00", "free_ram_gb": 4.356, "tree_rss_gb": 7.577, "peak_wset_gb": 7.581, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7232.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1042.0, "power.draw": 16.18, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:36.565+00:00", "free_ram_gb": 4.532, "tree_rss_gb": 7.6, "peak_wset_gb": 7.601, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7232.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 51.0, "clocks.sm": 517.0, "power.draw": 7.43, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:41.570+00:00", "free_ram_gb": 4.626, "tree_rss_gb": 7.605, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7232.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 960.0, "power.draw": 14.61, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:46.569+00:00", "free_ram_gb": 4.642, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7234.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 51.0, "clocks.sm": 1192.0, "power.draw": 18.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:51.613+00:00", "free_ram_gb": 4.729, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7234.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 52.0, "clocks.sm": 1132.0, "power.draw": 18.49, "compute_pids": [22644]}}+{"t": "2026-10-23T14:30:56.631+00:00", "free_ram_gb": 4.751, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7234.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 17.03, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:01.693+00:00", "free_ram_gb": 4.6, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7236.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1185.0, "power.draw": 16.29, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:06.665+00:00", "free_ram_gb": 4.484, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7236.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 14.5, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:11.714+00:00", "free_ram_gb": 4.224, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7236.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1057.0, "power.draw": 15.51, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:16.738+00:00", "free_ram_gb": 4.339, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7238.0, "memory.total": 8151.0, "utilization.gpu": 7.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 13.57, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:21.748+00:00", "free_ram_gb": 4.532, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7238.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1125.0, "power.draw": 12.39, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:26.769+00:00", "free_ram_gb": 4.487, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7240.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 15.7, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:31.766+00:00", "free_ram_gb": 4.449, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7240.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 937.0, "power.draw": 14.85, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:36.757+00:00", "free_ram_gb": 4.47, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7240.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 16.35, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:41.809+00:00", "free_ram_gb": 4.323, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7242.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1162.0, "power.draw": 17.08, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:46.777+00:00", "free_ram_gb": 4.372, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7242.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 16.75, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:51.822+00:00", "free_ram_gb": 4.499, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7242.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 15.15, "compute_pids": [22644]}}+{"t": "2026-10-23T14:31:56.789+00:00", "free_ram_gb": 4.465, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7244.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1155.0, "power.draw": 18.05, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:01.814+00:00", "free_ram_gb": 4.372, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7244.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 877.0, "power.draw": 15.95, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:06.838+00:00", "free_ram_gb": 4.313, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7246.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1185.0, "power.draw": 17.17, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:11.876+00:00", "free_ram_gb": 4.351, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7246.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1050.0, "power.draw": 16.62, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:16.882+00:00", "free_ram_gb": 4.406, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7246.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 50.0, "clocks.sm": 1065.0, "power.draw": 13.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:21.922+00:00", "free_ram_gb": 4.434, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7248.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 1020.0, "power.draw": 17.31, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:26.892+00:00", "free_ram_gb": 4.444, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7248.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 16.69, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:31.894+00:00", "free_ram_gb": 4.243, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7248.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 16.12, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:36.908+00:00", "free_ram_gb": 4.563, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7250.0, "memory.total": 8151.0, "utilization.gpu": 23.0, "temperature.gpu": 53.0, "clocks.sm": 1110.0, "power.draw": 22.99, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:41.916+00:00", "free_ram_gb": 4.577, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7250.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 997.0, "power.draw": 16.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:46.979+00:00", "free_ram_gb": 4.398, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7252.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 16.07, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:52.009+00:00", "free_ram_gb": 4.134, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7252.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 945.0, "power.draw": 14.78, "compute_pids": [22644]}}+{"t": "2026-10-23T14:32:56.969+00:00", "free_ram_gb": 4.128, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7254.0, "memory.total": 8151.0, "utilization.gpu": 25.0, "temperature.gpu": 54.0, "clocks.sm": 1200.0, "power.draw": 20.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:02.045+00:00", "free_ram_gb": 4.646, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7254.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 53.0, "clocks.sm": 1057.0, "power.draw": 15.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:07.032+00:00", "free_ram_gb": 4.68, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7256.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 53.0, "clocks.sm": 1117.0, "power.draw": 17.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:12.051+00:00", "free_ram_gb": 4.64, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7256.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 53.0, "clocks.sm": 1717.0, "power.draw": 23.38, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:17.043+00:00", "free_ram_gb": 4.648, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7256.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 14.4, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:22.094+00:00", "free_ram_gb": 4.651, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7258.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 17.14, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:27.067+00:00", "free_ram_gb": 4.379, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7258.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 15.59, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:32.172+00:00", "free_ram_gb": 4.358, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7260.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 14.99, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:37.170+00:00", "free_ram_gb": 4.308, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7260.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1027.0, "power.draw": 14.95, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:42.191+00:00", "free_ram_gb": 4.542, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7260.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1087.0, "power.draw": 15.0, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:47.204+00:00", "free_ram_gb": 4.652, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7262.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 17.12, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:52.238+00:00", "free_ram_gb": 4.308, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7262.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 937.0, "power.draw": 11.68, "compute_pids": [22644]}}+{"t": "2026-10-23T14:33:57.247+00:00", "free_ram_gb": 4.106, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7262.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 990.0, "power.draw": 8.85, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:02.263+00:00", "free_ram_gb": 4.066, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7264.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 15.48, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:07.272+00:00", "free_ram_gb": 4.119, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7264.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 23.46, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:12.272+00:00", "free_ram_gb": 4.173, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7266.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 53.0, "clocks.sm": 1095.0, "power.draw": 9.08, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:17.324+00:00", "free_ram_gb": 4.105, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7266.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1087.0, "power.draw": 16.52, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:22.333+00:00", "free_ram_gb": 4.153, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7266.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1140.0, "power.draw": 15.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:27.324+00:00", "free_ram_gb": 4.129, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7268.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1035.0, "power.draw": 16.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:32.358+00:00", "free_ram_gb": 4.005, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7268.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1050.0, "power.draw": 14.72, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:37.340+00:00", "free_ram_gb": 3.797, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7270.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 53.0, "clocks.sm": 1125.0, "power.draw": 54.94, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:42.395+00:00", "free_ram_gb": 4.192, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7270.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 53.0, "clocks.sm": 952.0, "power.draw": 17.51, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:47.394+00:00", "free_ram_gb": 4.581, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7272.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 15.07, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:52.436+00:00", "free_ram_gb": 4.623, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7272.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 53.0, "clocks.sm": 1125.0, "power.draw": 18.11, "compute_pids": [22644]}}+{"t": "2026-10-23T14:34:57.439+00:00", "free_ram_gb": 4.644, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7272.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 15.64, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:02.465+00:00", "free_ram_gb": 4.567, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7274.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 52.0, "clocks.sm": 1147.0, "power.draw": 18.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:07.433+00:00", "free_ram_gb": 4.564, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7274.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1147.0, "power.draw": 16.53, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:12.440+00:00", "free_ram_gb": 4.258, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7276.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1050.0, "power.draw": 16.74, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:17.446+00:00", "free_ram_gb": 4.56, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7276.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 51.0, "clocks.sm": 997.0, "power.draw": 17.02, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:22.494+00:00", "free_ram_gb": 4.529, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7278.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 15.8, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:27.478+00:00", "free_ram_gb": 4.289, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7278.0, "memory.total": 8151.0, "utilization.gpu": 24.0, "temperature.gpu": 53.0, "clocks.sm": 1192.0, "power.draw": 21.24, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:32.533+00:00", "free_ram_gb": 4.438, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7278.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 52.0, "clocks.sm": 1020.0, "power.draw": 13.53, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:37.527+00:00", "free_ram_gb": 4.447, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7280.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1140.0, "power.draw": 18.31, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:42.530+00:00", "free_ram_gb": 4.459, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7280.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 52.0, "clocks.sm": 1035.0, "power.draw": 13.9, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:47.564+00:00", "free_ram_gb": 4.414, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7282.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 51.0, "clocks.sm": 967.0, "power.draw": 14.35, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:52.610+00:00", "free_ram_gb": 4.478, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7282.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 17.26, "compute_pids": [22644]}}+{"t": "2026-10-23T14:35:57.564+00:00", "free_ram_gb": 4.485, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7282.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 15.26, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:02.610+00:00", "free_ram_gb": 4.513, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7284.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 17.21, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:07.609+00:00", "free_ram_gb": 4.439, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7284.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 17.46, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:12.640+00:00", "free_ram_gb": 4.403, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7286.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 52.0, "clocks.sm": 1027.0, "power.draw": 17.59, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:17.609+00:00", "free_ram_gb": 4.41, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7286.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1102.0, "power.draw": 18.03, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:22.662+00:00", "free_ram_gb": 4.511, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7286.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 982.0, "power.draw": 14.88, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:27.636+00:00", "free_ram_gb": 4.578, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7288.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 17.63, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:32.632+00:00", "free_ram_gb": 4.145, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7288.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 16.75, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:37.662+00:00", "free_ram_gb": 4.051, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7290.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 15.28, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:42.659+00:00", "free_ram_gb": 4.258, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7290.0, "memory.total": 8151.0, "utilization.gpu": 21.0, "temperature.gpu": 51.0, "clocks.sm": 997.0, "power.draw": 18.37, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:47.682+00:00", "free_ram_gb": 4.42, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7290.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1132.0, "power.draw": 14.86, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:52.692+00:00", "free_ram_gb": 4.223, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7292.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 16.88, "compute_pids": [22644]}}+{"t": "2026-10-23T14:36:57.638+00:00", "free_ram_gb": 4.956, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7292.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 53.0, "clocks.sm": 990.0, "power.draw": 18.01, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:02.702+00:00", "free_ram_gb": 5.087, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7294.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 53.0, "clocks.sm": 1072.0, "power.draw": 16.93, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:07.724+00:00", "free_ram_gb": 5.112, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7294.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1072.0, "power.draw": 17.06, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:12.745+00:00", "free_ram_gb": 5.136, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7294.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 14.73, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:17.710+00:00", "free_ram_gb": 5.105, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7296.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1087.0, "power.draw": 17.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:22.732+00:00", "free_ram_gb": 5.124, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7296.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 16.22, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:27.764+00:00", "free_ram_gb": 5.131, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7296.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 14.77, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:32.802+00:00", "free_ram_gb": 5.341, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7298.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1035.0, "power.draw": 16.38, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:37.802+00:00", "free_ram_gb": 4.842, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7298.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 15.31, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:42.816+00:00", "free_ram_gb": 4.143, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7300.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 16.1, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:47.800+00:00", "free_ram_gb": 3.61, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7300.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 17.69, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:52.844+00:00", "free_ram_gb": 3.701, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7300.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 14.6, "compute_pids": [22644]}}+{"t": "2026-10-23T14:37:57.867+00:00", "free_ram_gb": 4.149, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7302.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 17.51, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:02.891+00:00", "free_ram_gb": 5.22, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7302.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1087.0, "power.draw": 22.45, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:07.878+00:00", "free_ram_gb": 5.161, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7302.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 50.0, "clocks.sm": 1042.0, "power.draw": 14.56, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:12.940+00:00", "free_ram_gb": 5.188, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7304.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1072.0, "power.draw": 20.74, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:17.948+00:00", "free_ram_gb": 5.157, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7304.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 16.89, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:22.973+00:00", "free_ram_gb": 5.163, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7306.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1087.0, "power.draw": 17.59, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:27.975+00:00", "free_ram_gb": 5.157, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7306.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 16.4, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:32.983+00:00", "free_ram_gb": 5.221, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7306.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1102.0, "power.draw": 16.55, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:38.011+00:00", "free_ram_gb": 5.288, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7308.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 51.0, "clocks.sm": 1132.0, "power.draw": 18.3, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:42.977+00:00", "free_ram_gb": 5.22, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7308.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 15.48, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:48.014+00:00", "free_ram_gb": 5.359, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7310.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 52.0, "clocks.sm": 1065.0, "power.draw": 16.54, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:53.004+00:00", "free_ram_gb": 5.575, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7310.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1042.0, "power.draw": 15.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:38:58.036+00:00", "free_ram_gb": 5.419, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7312.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 52.0, "clocks.sm": 1125.0, "power.draw": 19.49, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:03.084+00:00", "free_ram_gb": 5.351, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7312.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 960.0, "power.draw": 17.13, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:08.054+00:00", "free_ram_gb": 5.161, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7314.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1110.0, "power.draw": 16.13, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:13.084+00:00", "free_ram_gb": 5.22, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7314.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1162.0, "power.draw": 17.23, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:18.090+00:00", "free_ram_gb": 5.486, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7314.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1072.0, "power.draw": 7.87, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:23.112+00:00", "free_ram_gb": 5.492, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7316.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 52.0, "clocks.sm": 1155.0, "power.draw": 18.76, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:28.124+00:00", "free_ram_gb": 5.568, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7316.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 16.89, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:33.115+00:00", "free_ram_gb": 5.56, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7318.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1177.0, "power.draw": 17.79, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:38.108+00:00", "free_ram_gb": 5.211, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7318.0, "memory.total": 8151.0, "utilization.gpu": 28.0, "temperature.gpu": 54.0, "clocks.sm": 1207.0, "power.draw": 17.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:43.136+00:00", "free_ram_gb": 5.22, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7318.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 51.0, "clocks.sm": 982.0, "power.draw": 14.88, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:48.138+00:00", "free_ram_gb": 5.875, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7320.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 15.58, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:53.171+00:00", "free_ram_gb": 5.679, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7320.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1050.0, "power.draw": 13.89, "compute_pids": [22644]}}+{"t": "2026-10-23T14:39:58.192+00:00", "free_ram_gb": 5.737, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7322.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1117.0, "power.draw": 18.23, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:03.215+00:00", "free_ram_gb": 5.79, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7322.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 52.0, "clocks.sm": 1125.0, "power.draw": 18.22, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:08.176+00:00", "free_ram_gb": 5.684, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7322.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1035.0, "power.draw": 16.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:13.227+00:00", "free_ram_gb": 5.8, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7324.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 52.0, "clocks.sm": 1132.0, "power.draw": 18.0, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:18.258+00:00", "free_ram_gb": 5.92, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7324.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 16.17, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:23.249+00:00", "free_ram_gb": 5.946, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7326.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1200.0, "power.draw": 17.91, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:28.241+00:00", "free_ram_gb": 5.893, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7326.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 975.0, "power.draw": 16.73, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:33.265+00:00", "free_ram_gb": 5.886, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7328.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1005.0, "power.draw": 15.24, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:38.281+00:00", "free_ram_gb": 5.87, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7328.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1125.0, "power.draw": 18.08, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:43.280+00:00", "free_ram_gb": 5.915, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7328.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 15.42, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:48.281+00:00", "free_ram_gb": 6.161, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7330.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1072.0, "power.draw": 17.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:53.301+00:00", "free_ram_gb": 6.308, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7330.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 16.88, "compute_pids": [22644]}}+{"t": "2026-10-23T14:40:58.282+00:00", "free_ram_gb": 6.21, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7332.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1140.0, "power.draw": 18.83, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:03.310+00:00", "free_ram_gb": 6.077, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7332.0, "memory.total": 8151.0, "utilization.gpu": 29.0, "temperature.gpu": 54.0, "clocks.sm": 1177.0, "power.draw": 36.47, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:08.328+00:00", "free_ram_gb": 5.591, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7332.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 1042.0, "power.draw": 14.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:13.335+00:00", "free_ram_gb": 5.588, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7334.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 52.0, "clocks.sm": 1185.0, "power.draw": 18.6, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:18.340+00:00", "free_ram_gb": 5.462, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7334.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 14.41, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:23.339+00:00", "free_ram_gb": 5.384, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7336.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 1027.0, "power.draw": 16.85, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:28.345+00:00", "free_ram_gb": 5.253, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7336.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1125.0, "power.draw": 17.36, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:33.365+00:00", "free_ram_gb": 5.126, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7336.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 967.0, "power.draw": 15.33, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:38.408+00:00", "free_ram_gb": 5.197, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7338.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1117.0, "power.draw": 16.69, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:43.430+00:00", "free_ram_gb": 5.187, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7338.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 52.0, "clocks.sm": 922.0, "power.draw": 16.45, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:48.465+00:00", "free_ram_gb": 5.728, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7338.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 52.0, "clocks.sm": 1005.0, "power.draw": 15.27, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:53.435+00:00", "free_ram_gb": 5.767, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7340.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1102.0, "power.draw": 19.82, "compute_pids": [22644]}}+{"t": "2026-10-23T14:41:58.433+00:00", "free_ram_gb": 5.768, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7340.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 15.53, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:03.466+00:00", "free_ram_gb": 5.798, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7342.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1057.0, "power.draw": 17.78, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:08.460+00:00", "free_ram_gb": 5.818, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7342.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 900.0, "power.draw": 15.52, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:13.484+00:00", "free_ram_gb": 5.944, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7342.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1080.0, "power.draw": 15.62, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:18.489+00:00", "free_ram_gb": 5.874, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7344.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1170.0, "power.draw": 8.19, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:23.492+00:00", "free_ram_gb": 7.194, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7344.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 51.0, "clocks.sm": 1087.0, "power.draw": 17.01, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:28.509+00:00", "free_ram_gb": 7.144, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7346.0, "memory.total": 8151.0, "utilization.gpu": 24.0, "temperature.gpu": 51.0, "clocks.sm": 1125.0, "power.draw": 21.25, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:33.543+00:00", "free_ram_gb": 6.976, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7346.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 15.24, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:38.528+00:00", "free_ram_gb": 7.102, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7348.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1162.0, "power.draw": 18.08, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:43.574+00:00", "free_ram_gb": 7.114, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7348.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 23.82, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:48.584+00:00", "free_ram_gb": 7.12, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7350.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 15.94, "compute_pids": [22644]}}+{"t": "2026-10-23T14:42:53.570+00:00", "free_ram_gb": 7.301, "tree_rss_gb": 7.607, "peak_wset_gb": 7.608, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7350.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 997.0, "power.draw": 17.92, "compute_pids": [22644]}}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/box.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/box.jsonlnew file mode 100644index 00000000..df6ffb23--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/box.jsonl@@ -0,0 +1,2 @@+{"at":"before","utc":"2026-10-23T14:43:25.4572401Z","mem":{"AvailableBytes":15142400000,"CommittedBytes":28662173696,"PagesInputPersec":199454223},"ac":2,"gpu":"0 MiB, 47, 0 MHz"}+{"at":"after","utc":"2026-10-23T15:02:29.8387038Z","mem":{"AvailableBytes":15550382080,"CommittedBytes":28872978432,"PagesInputPersec":201352256},"ac":2,"gpu":"0 MiB, 48, 0 MHz"}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/exit.txt b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/exit.txtnew file mode 100644index 00000000..ca916d09--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/exit.txt@@ -0,0 +1 @@+exit 0diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/stderr.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/stderr.lognew file mode 100644index 00000000..9871e53b--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/stderr.log@@ -0,0 +1,2 @@+W1023 19:43:37.412000 6068 site-packages\torch\utils\flop_counter.py:113] triton not found; flop counting will not work for triton kernels+[transformers] `torch_dtype` is deprecated! Use `dtype` instead!diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/stdout.log b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/stdout.lognew file mode 100644index 00000000..f34554d3--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/stdout.log@@ -0,0 +1 @@+{"step": 0, "seconds": 1120.52, "tokens_per_second": 1.828, "loss": 13.491609573364258, "loss_positions": 512, "loss_finite": true, "grads_finite": true, "grads_present": 610, "expert_fetches": 69120, "expert_bytes_read": 1570336174080, "layer_loads": 120, "os_io": {"process_read_bytes": 1587653703774, "disk_read_bytes": {"PhysicalDrive0": 8110742528, "PhysicalDrive1": 1586785978368}}, "peak_vram_mib": 6091, "peak_reserved_mib": 7176, "peak_host_bytes_so_far": 7757692928}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/step.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/step.jsonnew file mode 100644index 00000000..7ee547e0--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/step.json@@ -0,0 +1,77 @@+{+ "label": "random weights, shape only; routing forced uniform (token t -> experts kt..kt+k-1 mod E)",+ "args": {+ "store": "D:/soup-bench/moe-k2-synth",+ "weights": null,+ "model": null,+ "expert_layout": "per_expert",+ "out": "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\b2\\step.json",+ "seq": 2048,+ "batch": 1,+ "steps": 1,+ "loss_tokens": 512,+ "lora_r": 8,+ "buffers": 2,+ "tier": "disk",+ "no_pin": true,+ "seed": 3,+ "label": "random weights, shape only",+ "route": "uniform"+ },+ "steps": [+ {+ "step": 0,+ "seconds": 1120.52,+ "tokens_per_second": 1.828,+ "loss": 13.491609573364258,+ "loss_positions": 512,+ "loss_finite": true,+ "grads_finite": true,+ "grads_present": 610,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080,+ "layer_loads": 120,+ "os_io": {+ "process_read_bytes": 1587653703774,+ "disk_read_bytes": {+ "PhysicalDrive0": 8110742528,+ "PhysicalDrive1": 1586785978368+ }+ },+ "peak_vram_mib": 6091,+ "peak_reserved_mib": 7176,+ "peak_host_bytes_so_far": 7757692928+ }+ ],+ "store_bytes": 533242299230,+ "store_expert_layout": "per_expert",+ "store_quant": "nf4",+ "store_double_quant": true,+ "build_seconds": 6.25,+ "routers_overridden": 60,+ "trainable_params": 30513664,+ "stats": {+ "n_layers": 61,+ "buffers": 2,+ "buffer_bytes": 2918596920,+ "large_buffer_bytes": 2348810240,+ "store_bytes": 5115134896,+ "pinned": false,+ "pinned_bytes": 0,+ "tier": "disk",+ "read_ahead": 2,+ "disk_bytes": 9779642448,+ "drive_count": 1,+ "direct_io_drives": 1,+ "direct_io_roots": {+ "D:/soup-bench/moe-k2-synth": true+ },+ "layer_loads": 120,+ "large_loads": 3,+ "device": "cuda",+ "total_params": 1026408232448,+ "nf4_scale_cache_bytes": 0,+ "expert_fetches": 69120,+ "expert_bytes_read": 1570336174080+ }+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/watchdog.json b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/watchdog.jsonnew file mode 100644index 00000000..b3a9a489--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/watchdog.json@@ -0,0 +1,39 @@+{+ "started": "2026-10-23T14:43:26.628+00:00",+ "match": null,+ "min_free_gb": 1.5,+ "period_s": 5.0,+ "fired": false,+ "gaps": [],+ "off_ac_samples": 0,+ "foreign_compute_pids": [],+ "pid": 6068,+ "cmdline": [+ "C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\python.exe",+ "benchmarks/harness/moe_stream_step.py",+ "--store",+ "D:/soup-bench/moe-k2-synth",+ "--tier",+ "disk",+ "--no-pin",+ "--route",+ "uniform",+ "--batch",+ "1",+ "--steps",+ "1",+ "--loss-tokens",+ "512",+ "--seq",+ "2048",+ "--out",+ "C:\\Users\\user\\projects\\soup-wt\\moe-seq-scaling\\benchmarks\\results\\probe-rtx5070\\giant-moe\\seq-scaling\\b2\\step.json"+ ],+ "ended": "2026-10-23T15:02:29.805+00:00",+ "samples": 228,+ "min_free_ram_gb": 7.313,+ "max_tree_rss_gb": 7.603,+ "peak_wset_gb": 7.603,+ "max_gpu_memory_used_mib": 7350.0,+ "max_gpu_temperature_c": 52.0+}diff --git a/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/watchdog.jsonl b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/watchdog.jsonlnew file mode 100644index 00000000..9275a432--- /dev/null+++ b/benchmarks/results/probe-rtx5070/giant-moe/seq-scaling/b2/watchdog.jsonl@@ -0,0 +1,228 @@+{"t": "2026-10-23T14:43:26.841+00:00", "free_ram_gb": 15.119, "tree_rss_gb": 0.022, "peak_wset_gb": 0.022, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 46.0, "clocks.sm": 1417.0, "power.draw": 12.58, "compute_pids": []}}+{"t": "2026-10-23T14:43:32.473+00:00", "free_ram_gb": 14.117, "tree_rss_gb": 0.594, "peak_wset_gb": 0.594, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 47.0, "clocks.sm": 7740.0, "power.draw": 11.15, "compute_pids": []}}+{"t": "2026-10-23T14:43:37.666+00:00", "free_ram_gb": 13.892, "tree_rss_gb": 0.805, "peak_wset_gb": 0.805, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 0.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 47.0, "clocks.sm": 0.0, "power.draw": 11.09, "compute_pids": []}}+{"t": "2026-10-23T14:43:41.756+00:00", "free_ram_gb": 13.688, "tree_rss_gb": 1.013, "peak_wset_gb": 1.013, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 232.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 48.0, "clocks.sm": 1417.0, "power.draw": 12.34, "compute_pids": [6068]}}+{"t": "2026-10-23T14:43:46.770+00:00", "free_ram_gb": 8.603, "tree_rss_gb": 6.517, "peak_wset_gb": 6.517, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 3148.0, "memory.total": 8151.0, "utilization.gpu": 0.0, "temperature.gpu": 48.0, "clocks.sm": 1462.0, "power.draw": 11.52, "compute_pids": [6068]}}+{"t": "2026-10-23T14:43:51.810+00:00", "free_ram_gb": 8.049, "tree_rss_gb": 7.127, "peak_wset_gb": 7.142, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 49.0, "clocks.sm": 922.0, "power.draw": 16.6, "compute_pids": [6068]}}+{"t": "2026-10-23T14:43:56.799+00:00", "free_ram_gb": 8.016, "tree_rss_gb": 7.13, "peak_wset_gb": 7.142, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1080.0, "power.draw": 15.24, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:01.837+00:00", "free_ram_gb": 7.874, "tree_rss_gb": 7.132, "peak_wset_gb": 7.142, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 22.83, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:06.881+00:00", "free_ram_gb": 7.567, "tree_rss_gb": 7.135, "peak_wset_gb": 7.142, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 832.0, "power.draw": 14.63, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:11.920+00:00", "free_ram_gb": 7.518, "tree_rss_gb": 7.137, "peak_wset_gb": 7.142, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 990.0, "power.draw": 14.92, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:16.901+00:00", "free_ram_gb": 7.874, "tree_rss_gb": 7.14, "peak_wset_gb": 7.142, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1035.0, "power.draw": 14.04, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:21.930+00:00", "free_ram_gb": 7.858, "tree_rss_gb": 7.144, "peak_wset_gb": 7.144, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 14.61, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:26.925+00:00", "free_ram_gb": 7.773, "tree_rss_gb": 7.146, "peak_wset_gb": 7.146, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 21.0, "temperature.gpu": 51.0, "clocks.sm": 1350.0, "power.draw": 15.46, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:31.918+00:00", "free_ram_gb": 7.802, "tree_rss_gb": 7.149, "peak_wset_gb": 7.149, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 50.0, "clocks.sm": 1140.0, "power.draw": 14.91, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:36.992+00:00", "free_ram_gb": 7.744, "tree_rss_gb": 7.151, "peak_wset_gb": 7.151, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4088.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 50.0, "clocks.sm": 945.0, "power.draw": 13.52, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:41.974+00:00", "free_ram_gb": 7.771, "tree_rss_gb": 7.154, "peak_wset_gb": 7.154, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 50.0, "clocks.sm": 870.0, "power.draw": 14.2, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:47.021+00:00", "free_ram_gb": 7.776, "tree_rss_gb": 7.156, "peak_wset_gb": 7.156, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 997.0, "power.draw": 13.95, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:51.992+00:00", "free_ram_gb": 7.622, "tree_rss_gb": 7.159, "peak_wset_gb": 7.159, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 17.48, "compute_pids": [6068]}}+{"t": "2026-10-23T14:44:57.053+00:00", "free_ram_gb": 7.743, "tree_rss_gb": 7.162, "peak_wset_gb": 7.162, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 945.0, "power.draw": 14.72, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:02.063+00:00", "free_ram_gb": 7.742, "tree_rss_gb": 7.164, "peak_wset_gb": 7.164, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4216.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1050.0, "power.draw": 14.73, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:07.909+00:00", "free_ram_gb": 7.734, "tree_rss_gb": 7.167, "peak_wset_gb": 7.167, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 51.0, "clocks.sm": 1312.0, "power.draw": 22.74, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:12.098+00:00", "free_ram_gb": 7.732, "tree_rss_gb": 7.17, "peak_wset_gb": 7.17, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1117.0, "power.draw": 14.88, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:17.152+00:00", "free_ram_gb": 7.642, "tree_rss_gb": 7.172, "peak_wset_gb": 7.172, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 997.0, "power.draw": 14.92, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:22.151+00:00", "free_ram_gb": 7.641, "tree_rss_gb": 7.175, "peak_wset_gb": 7.175, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 967.0, "power.draw": 13.46, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:27.159+00:00", "free_ram_gb": 7.651, "tree_rss_gb": 7.178, "peak_wset_gb": 7.178, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1290.0, "power.draw": 21.14, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:32.138+00:00", "free_ram_gb": 7.573, "tree_rss_gb": 7.181, "peak_wset_gb": 7.181, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4344.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1065.0, "power.draw": 15.97, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:37.173+00:00", "free_ram_gb": 7.577, "tree_rss_gb": 7.182, "peak_wset_gb": 7.182, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 1027.0, "power.draw": 14.65, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:42.214+00:00", "free_ram_gb": 7.626, "tree_rss_gb": 7.185, "peak_wset_gb": 7.185, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 15.85, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:47.259+00:00", "free_ram_gb": 7.618, "tree_rss_gb": 7.188, "peak_wset_gb": 7.188, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 15.53, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:53.180+00:00", "free_ram_gb": 7.577, "tree_rss_gb": 7.19, "peak_wset_gb": 7.19, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 937.0, "power.draw": 14.04, "compute_pids": [6068]}}+{"t": "2026-10-23T14:45:57.208+00:00", "free_ram_gb": 7.653, "tree_rss_gb": 7.193, "peak_wset_gb": 7.193, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4472.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 14.91, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:02.205+00:00", "free_ram_gb": 7.47, "tree_rss_gb": 7.196, "peak_wset_gb": 7.196, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 15.23, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:07.230+00:00", "free_ram_gb": 7.536, "tree_rss_gb": 7.198, "peak_wset_gb": 7.198, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 51.0, "clocks.sm": 1230.0, "power.draw": 21.83, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:12.238+00:00", "free_ram_gb": 7.39, "tree_rss_gb": 7.2, "peak_wset_gb": 7.2, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1042.0, "power.draw": 15.7, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:17.249+00:00", "free_ram_gb": 7.364, "tree_rss_gb": 7.204, "peak_wset_gb": 7.204, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 15.41, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:22.359+00:00", "free_ram_gb": 7.402, "tree_rss_gb": 7.206, "peak_wset_gb": 7.206, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 51.0, "clocks.sm": 937.0, "power.draw": 13.95, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:27.299+00:00", "free_ram_gb": 7.425, "tree_rss_gb": 7.209, "peak_wset_gb": 7.209, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4600.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 51.0, "clocks.sm": 930.0, "power.draw": 14.69, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:32.579+00:00", "free_ram_gb": 7.473, "tree_rss_gb": 7.212, "peak_wset_gb": 7.212, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 975.0, "power.draw": 15.59, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:37.338+00:00", "free_ram_gb": 7.313, "tree_rss_gb": 7.214, "peak_wset_gb": 7.214, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 990.0, "power.draw": 14.59, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:42.350+00:00", "free_ram_gb": 7.469, "tree_rss_gb": 7.216, "peak_wset_gb": 7.216, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 952.0, "power.draw": 17.32, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:47.378+00:00", "free_ram_gb": 7.714, "tree_rss_gb": 7.218, "peak_wset_gb": 7.218, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 1005.0, "power.draw": 14.84, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:52.386+00:00", "free_ram_gb": 7.724, "tree_rss_gb": 7.221, "peak_wset_gb": 7.221, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4728.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 50.0, "clocks.sm": 930.0, "power.draw": 13.96, "compute_pids": [6068]}}+{"t": "2026-10-23T14:46:57.386+00:00", "free_ram_gb": 7.713, "tree_rss_gb": 7.224, "peak_wset_gb": 7.224, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 997.0, "power.draw": 13.94, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:02.388+00:00", "free_ram_gb": 7.715, "tree_rss_gb": 7.226, "peak_wset_gb": 7.226, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 870.0, "power.draw": 13.85, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:07.412+00:00", "free_ram_gb": 7.725, "tree_rss_gb": 7.228, "peak_wset_gb": 7.228, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 50.0, "clocks.sm": 922.0, "power.draw": 12.86, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:12.413+00:00", "free_ram_gb": 7.723, "tree_rss_gb": 7.231, "peak_wset_gb": 7.231, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 50.0, "clocks.sm": 907.0, "power.draw": 16.83, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:17.455+00:00", "free_ram_gb": 7.751, "tree_rss_gb": 7.233, "peak_wset_gb": 7.233, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 930.0, "power.draw": 13.57, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:22.518+00:00", "free_ram_gb": 7.726, "tree_rss_gb": 7.236, "peak_wset_gb": 7.236, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4856.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 885.0, "power.draw": 13.73, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:27.461+00:00", "free_ram_gb": 7.821, "tree_rss_gb": 7.239, "peak_wset_gb": 7.239, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 975.0, "power.draw": 14.73, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:32.448+00:00", "free_ram_gb": 7.819, "tree_rss_gb": 7.24, "peak_wset_gb": 7.24, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 885.0, "power.draw": 12.77, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:37.506+00:00", "free_ram_gb": 7.821, "tree_rss_gb": 7.243, "peak_wset_gb": 7.243, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 990.0, "power.draw": 14.47, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:42.489+00:00", "free_ram_gb": 7.696, "tree_rss_gb": 7.245, "peak_wset_gb": 7.245, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 50.0, "clocks.sm": 802.0, "power.draw": 12.85, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:47.869+00:00", "free_ram_gb": 7.726, "tree_rss_gb": 7.247, "peak_wset_gb": 7.247, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 855.0, "power.draw": 13.49, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:52.538+00:00", "free_ram_gb": 7.665, "tree_rss_gb": 7.245, "peak_wset_gb": 7.248, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 4984.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 885.0, "power.draw": 13.56, "compute_pids": [6068]}}+{"t": "2026-10-23T14:47:57.556+00:00", "free_ram_gb": 7.802, "tree_rss_gb": 7.247, "peak_wset_gb": 7.248, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 1087.0, "power.draw": 15.17, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:02.914+00:00", "free_ram_gb": 7.99, "tree_rss_gb": 7.25, "peak_wset_gb": 7.25, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 49.0, "clocks.sm": 1005.0, "power.draw": 14.4, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:08.155+00:00", "free_ram_gb": 8.331, "tree_rss_gb": 7.252, "peak_wset_gb": 7.252, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 14.33, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:12.643+00:00", "free_ram_gb": 8.379, "tree_rss_gb": 7.255, "peak_wset_gb": 7.255, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1012.0, "power.draw": 14.43, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:17.701+00:00", "free_ram_gb": 8.207, "tree_rss_gb": 7.257, "peak_wset_gb": 7.257, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 952.0, "power.draw": 11.04, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:22.697+00:00", "free_ram_gb": 8.215, "tree_rss_gb": 7.259, "peak_wset_gb": 7.259, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5112.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 49.0, "clocks.sm": 915.0, "power.draw": 13.31, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:27.696+00:00", "free_ram_gb": 8.173, "tree_rss_gb": 7.262, "peak_wset_gb": 7.262, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 50.0, "clocks.sm": 1117.0, "power.draw": 16.29, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:32.764+00:00", "free_ram_gb": 8.181, "tree_rss_gb": 7.263, "peak_wset_gb": 7.263, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 930.0, "power.draw": 13.43, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:38.084+00:00", "free_ram_gb": 8.278, "tree_rss_gb": 7.266, "peak_wset_gb": 7.266, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1050.0, "power.draw": 15.1, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:42.801+00:00", "free_ram_gb": 8.297, "tree_rss_gb": 7.268, "peak_wset_gb": 7.268, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 907.0, "power.draw": 14.15, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:48.043+00:00", "free_ram_gb": 8.219, "tree_rss_gb": 7.27, "peak_wset_gb": 7.27, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1012.0, "power.draw": 15.77, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:53.242+00:00", "free_ram_gb": 8.123, "tree_rss_gb": 7.273, "peak_wset_gb": 7.273, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 967.0, "power.draw": 13.51, "compute_pids": [6068]}}+{"t": "2026-10-23T14:48:57.915+00:00", "free_ram_gb": 8.176, "tree_rss_gb": 7.275, "peak_wset_gb": 7.275, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5240.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 49.0, "clocks.sm": 825.0, "power.draw": 13.28, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:02.856+00:00", "free_ram_gb": 8.2, "tree_rss_gb": 7.277, "peak_wset_gb": 7.277, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 930.0, "power.draw": 13.82, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:07.844+00:00", "free_ram_gb": 8.196, "tree_rss_gb": 7.279, "peak_wset_gb": 7.279, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1087.0, "power.draw": 15.24, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:12.845+00:00", "free_ram_gb": 8.216, "tree_rss_gb": 7.281, "peak_wset_gb": 7.281, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 975.0, "power.draw": 13.54, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:17.859+00:00", "free_ram_gb": 8.259, "tree_rss_gb": 7.284, "peak_wset_gb": 7.284, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 8.0, "temperature.gpu": 49.0, "clocks.sm": 892.0, "power.draw": 13.22, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:22.839+00:00", "free_ram_gb": 8.272, "tree_rss_gb": 7.286, "peak_wset_gb": 7.286, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 990.0, "power.draw": 14.97, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:28.260+00:00", "free_ram_gb": 8.21, "tree_rss_gb": 7.288, "peak_wset_gb": 7.288, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5368.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 855.0, "power.draw": 13.99, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:32.901+00:00", "free_ram_gb": 8.094, "tree_rss_gb": 7.291, "peak_wset_gb": 7.291, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 960.0, "power.draw": 14.39, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:38.206+00:00", "free_ram_gb": 8.029, "tree_rss_gb": 7.293, "peak_wset_gb": 7.293, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1035.0, "power.draw": 14.2, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:42.934+00:00", "free_ram_gb": 8.121, "tree_rss_gb": 7.296, "peak_wset_gb": 7.296, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 960.0, "power.draw": 14.4, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:47.961+00:00", "free_ram_gb": 8.087, "tree_rss_gb": 7.298, "peak_wset_gb": 7.298, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 975.0, "power.draw": 14.85, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:53.307+00:00", "free_ram_gb": 8.03, "tree_rss_gb": 7.302, "peak_wset_gb": 7.302, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5496.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 1005.0, "power.draw": 14.7, "compute_pids": [6068]}}+{"t": "2026-10-23T14:49:58.446+00:00", "free_ram_gb": 8.057, "tree_rss_gb": 7.304, "peak_wset_gb": 7.304, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 997.0, "power.draw": 14.74, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:02.987+00:00", "free_ram_gb": 8.081, "tree_rss_gb": 7.307, "peak_wset_gb": 7.307, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 990.0, "power.draw": 14.28, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:08.344+00:00", "free_ram_gb": 7.979, "tree_rss_gb": 7.31, "peak_wset_gb": 7.31, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1005.0, "power.draw": 15.03, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:13.494+00:00", "free_ram_gb": 8.029, "tree_rss_gb": 7.312, "peak_wset_gb": 7.312, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 960.0, "power.draw": 14.43, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:18.209+00:00", "free_ram_gb": 8.15, "tree_rss_gb": 7.315, "peak_wset_gb": 7.315, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5624.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 15.59, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:23.003+00:00", "free_ram_gb": 8.147, "tree_rss_gb": 7.318, "peak_wset_gb": 7.318, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 50.0, "clocks.sm": 885.0, "power.draw": 35.06, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:28.184+00:00", "free_ram_gb": 8.19, "tree_rss_gb": 7.32, "peak_wset_gb": 7.32, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 15.86, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:33.383+00:00", "free_ram_gb": 8.197, "tree_rss_gb": 7.323, "peak_wset_gb": 7.323, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 975.0, "power.draw": 15.19, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:38.070+00:00", "free_ram_gb": 8.231, "tree_rss_gb": 7.326, "peak_wset_gb": 7.326, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 960.0, "power.draw": 13.77, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:43.076+00:00", "free_ram_gb": 8.277, "tree_rss_gb": 7.329, "peak_wset_gb": 7.329, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 960.0, "power.draw": 14.32, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:48.521+00:00", "free_ram_gb": 8.259, "tree_rss_gb": 7.33, "peak_wset_gb": 7.33, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 5752.0, "memory.total": 8151.0, "utilization.gpu": 4.0, "temperature.gpu": 49.0, "clocks.sm": 727.0, "power.draw": 12.09, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:53.117+00:00", "free_ram_gb": 8.193, "tree_rss_gb": 7.41, "peak_wset_gb": 7.41, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 6838.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 922.0, "power.draw": 13.27, "compute_pids": [6068]}}+{"t": "2026-10-23T14:50:58.132+00:00", "free_ram_gb": 8.084, "tree_rss_gb": 7.529, "peak_wset_gb": 7.531, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7232.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 960.0, "power.draw": 15.77, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:03.182+00:00", "free_ram_gb": 7.914, "tree_rss_gb": 7.574, "peak_wset_gb": 7.578, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7232.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 885.0, "power.draw": 13.74, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:08.755+00:00", "free_ram_gb": 7.862, "tree_rss_gb": 7.601, "peak_wset_gb": 7.601, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7232.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 51.0, "clocks.sm": 1012.0, "power.draw": 15.94, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:13.528+00:00", "free_ram_gb": 7.808, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7234.0, "memory.total": 8151.0, "utilization.gpu": 23.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 18.26, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:18.257+00:00", "free_ram_gb": 7.775, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7234.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1035.0, "power.draw": 15.64, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:23.661+00:00", "free_ram_gb": 7.855, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7236.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 51.0, "clocks.sm": 1117.0, "power.draw": 16.66, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:28.708+00:00", "free_ram_gb": 7.815, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7236.0, "memory.total": 8151.0, "utilization.gpu": 21.0, "temperature.gpu": 52.0, "clocks.sm": 1185.0, "power.draw": 20.02, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:33.273+00:00", "free_ram_gb": 8.088, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7236.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 14.37, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:38.763+00:00", "free_ram_gb": 8.109, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7238.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 52.0, "clocks.sm": 1110.0, "power.draw": 17.57, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:44.187+00:00", "free_ram_gb": 8.103, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7238.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1095.0, "power.draw": 16.2, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:48.281+00:00", "free_ram_gb": 8.107, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7240.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1065.0, "power.draw": 18.04, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:53.327+00:00", "free_ram_gb": 8.147, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7240.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1050.0, "power.draw": 15.65, "compute_pids": [6068]}}+{"t": "2026-10-23T14:51:58.773+00:00", "free_ram_gb": 8.139, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7242.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 51.0, "clocks.sm": 1162.0, "power.draw": 18.68, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:03.289+00:00", "free_ram_gb": 8.134, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7242.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1057.0, "power.draw": 15.9, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:08.337+00:00", "free_ram_gb": 8.146, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7244.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 51.0, "clocks.sm": 1207.0, "power.draw": 18.65, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:13.318+00:00", "free_ram_gb": 8.103, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7244.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1725.0, "power.draw": 24.75, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:18.826+00:00", "free_ram_gb": 8.088, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7244.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 51.0, "clocks.sm": 1110.0, "power.draw": 16.12, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:23.883+00:00", "free_ram_gb": 8.186, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7246.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1170.0, "power.draw": 18.14, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:28.881+00:00", "free_ram_gb": 8.165, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7246.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1042.0, "power.draw": 15.56, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:33.921+00:00", "free_ram_gb": 7.919, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7248.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 17.25, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:38.460+00:00", "free_ram_gb": 8.039, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7248.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 952.0, "power.draw": 16.06, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:43.461+00:00", "free_ram_gb": 8.056, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7250.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 52.0, "clocks.sm": 1132.0, "power.draw": 17.75, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:48.473+00:00", "free_ram_gb": 8.036, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7250.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 52.0, "clocks.sm": 967.0, "power.draw": 17.28, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:53.489+00:00", "free_ram_gb": 8.011, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7250.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 952.0, "power.draw": 16.65, "compute_pids": [6068]}}+{"t": "2026-10-23T14:52:58.992+00:00", "free_ram_gb": 8.18, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7252.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 52.0, "clocks.sm": 1132.0, "power.draw": 18.36, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:03.605+00:00", "free_ram_gb": 8.156, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7252.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 52.0, "clocks.sm": 1110.0, "power.draw": 15.97, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:09.540+00:00", "free_ram_gb": 8.197, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7254.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1057.0, "power.draw": 18.06, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:14.307+00:00", "free_ram_gb": 8.157, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7254.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 51.0, "clocks.sm": 1020.0, "power.draw": 15.82, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:19.062+00:00", "free_ram_gb": 8.166, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7256.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 51.0, "clocks.sm": 1170.0, "power.draw": 18.7, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:23.631+00:00", "free_ram_gb": 8.188, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7256.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 51.0, "clocks.sm": 1072.0, "power.draw": 17.17, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:29.108+00:00", "free_ram_gb": 8.18, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7258.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 18.43, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:34.084+00:00", "free_ram_gb": 8.172, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7258.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 900.0, "power.draw": 23.21, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:38.598+00:00", "free_ram_gb": 8.192, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7260.0, "memory.total": 8151.0, "utilization.gpu": 22.0, "temperature.gpu": 51.0, "clocks.sm": 1177.0, "power.draw": 17.73, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:43.638+00:00", "free_ram_gb": 8.071, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7260.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 52.0, "clocks.sm": 1665.0, "power.draw": 15.93, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:48.677+00:00", "free_ram_gb": 8.172, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7262.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 1050.0, "power.draw": 15.4, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:53.638+00:00", "free_ram_gb": 8.159, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7262.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1110.0, "power.draw": 18.63, "compute_pids": [6068]}}+{"t": "2026-10-23T14:53:58.670+00:00", "free_ram_gb": 8.14, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7262.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 15.73, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:03.869+00:00", "free_ram_gb": 8.126, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7264.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 975.0, "power.draw": 17.17, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:08.692+00:00", "free_ram_gb": 8.151, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7264.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1110.0, "power.draw": 16.17, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:13.690+00:00", "free_ram_gb": 8.043, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7266.0, "memory.total": 8151.0, "utilization.gpu": 21.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 18.05, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:18.742+00:00", "free_ram_gb": 8.079, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7266.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1020.0, "power.draw": 17.54, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:23.782+00:00", "free_ram_gb": 8.121, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7268.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1087.0, "power.draw": 17.79, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:28.733+00:00", "free_ram_gb": 8.131, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7268.0, "memory.total": 8151.0, "utilization.gpu": 24.0, "temperature.gpu": 51.0, "clocks.sm": 1612.0, "power.draw": 23.98, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:33.724+00:00", "free_ram_gb": 8.121, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7270.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 990.0, "power.draw": 15.47, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:38.760+00:00", "free_ram_gb": 8.119, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7270.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1155.0, "power.draw": 18.58, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:43.830+00:00", "free_ram_gb": 8.141, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7270.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 960.0, "power.draw": 15.77, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:48.808+00:00", "free_ram_gb": 8.11, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7272.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 907.0, "power.draw": 16.68, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:53.811+00:00", "free_ram_gb": 8.12, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7272.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 16.06, "compute_pids": [6068]}}+{"t": "2026-10-23T14:54:58.802+00:00", "free_ram_gb": 8.083, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7274.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1222.0, "power.draw": 18.78, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:03.792+00:00", "free_ram_gb": 8.136, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7274.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 22.01, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:08.852+00:00", "free_ram_gb": 8.128, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7276.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1057.0, "power.draw": 15.61, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:13.867+00:00", "free_ram_gb": 8.037, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7276.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1162.0, "power.draw": 18.51, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:19.704+00:00", "free_ram_gb": 8.015, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7276.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1110.0, "power.draw": 16.1, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:23.843+00:00", "free_ram_gb": 8.056, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7278.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 17.76, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:28.909+00:00", "free_ram_gb": 8.092, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7278.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1072.0, "power.draw": 16.06, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:33.891+00:00", "free_ram_gb": 8.096, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7280.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1057.0, "power.draw": 17.81, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:38.900+00:00", "free_ram_gb": 8.031, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7280.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1027.0, "power.draw": 17.45, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:43.925+00:00", "free_ram_gb": 8.031, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7282.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 50.0, "clocks.sm": 1147.0, "power.draw": 18.69, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:48.935+00:00", "free_ram_gb": 8.093, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7282.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 967.0, "power.draw": 23.47, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:54.015+00:00", "free_ram_gb": 8.041, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7284.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1222.0, "power.draw": 17.15, "compute_pids": [6068]}}+{"t": "2026-10-23T14:55:58.945+00:00", "free_ram_gb": 7.94, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7284.0, "memory.total": 8151.0, "utilization.gpu": 23.0, "temperature.gpu": 51.0, "clocks.sm": 1147.0, "power.draw": 18.85, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:03.943+00:00", "free_ram_gb": 8.112, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7284.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1065.0, "power.draw": 15.52, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:09.797+00:00", "free_ram_gb": 8.049, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7286.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1125.0, "power.draw": 18.13, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:14.337+00:00", "free_ram_gb": 8.046, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7286.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 1057.0, "power.draw": 15.63, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:18.985+00:00", "free_ram_gb": 7.926, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7288.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1027.0, "power.draw": 17.46, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:24.006+00:00", "free_ram_gb": 7.957, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7288.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1012.0, "power.draw": 17.15, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:29.013+00:00", "free_ram_gb": 7.962, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7290.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1117.0, "power.draw": 18.8, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:34.039+00:00", "free_ram_gb": 7.921, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7290.0, "memory.total": 8151.0, "utilization.gpu": 39.0, "temperature.gpu": 51.0, "clocks.sm": 1485.0, "power.draw": 44.42, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:39.043+00:00", "free_ram_gb": 8.014, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7290.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1057.0, "power.draw": 16.05, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:44.041+00:00", "free_ram_gb": 8.197, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7292.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 50.0, "clocks.sm": 1132.0, "power.draw": 18.81, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:49.728+00:00", "free_ram_gb": 8.163, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7292.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 15.6, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:54.467+00:00", "free_ram_gb": 8.135, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7294.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1162.0, "power.draw": 17.3, "compute_pids": [6068]}}+{"t": "2026-10-23T14:56:59.206+00:00", "free_ram_gb": 8.118, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7294.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 952.0, "power.draw": 15.04, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:04.371+00:00", "free_ram_gb": 8.11, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7296.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 1102.0, "power.draw": 14.8, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:09.068+00:00", "free_ram_gb": 8.091, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7296.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 49.0, "clocks.sm": 1117.0, "power.draw": 17.38, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:14.319+00:00", "free_ram_gb": 8.092, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7296.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 14.41, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:19.148+00:00", "free_ram_gb": 8.129, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7298.0, "memory.total": 8151.0, "utilization.gpu": 23.0, "temperature.gpu": 50.0, "clocks.sm": 1087.0, "power.draw": 18.0, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:24.148+00:00", "free_ram_gb": 8.126, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7298.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 16.12, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:29.232+00:00", "free_ram_gb": 8.083, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7300.0, "memory.total": 8151.0, "utilization.gpu": 22.0, "temperature.gpu": 50.0, "clocks.sm": 1027.0, "power.draw": 20.44, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:34.669+00:00", "free_ram_gb": 8.129, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7300.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 14.29, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:39.362+00:00", "free_ram_gb": 8.135, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7302.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 50.0, "clocks.sm": 1132.0, "power.draw": 19.24, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:44.288+00:00", "free_ram_gb": 8.073, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7302.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1012.0, "power.draw": 16.87, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:49.753+00:00", "free_ram_gb": 8.054, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7302.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1072.0, "power.draw": 15.38, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:54.286+00:00", "free_ram_gb": 8.068, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7304.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 49.0, "clocks.sm": 1050.0, "power.draw": 17.63, "compute_pids": [6068]}}+{"t": "2026-10-23T14:57:59.279+00:00", "free_ram_gb": 8.075, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7304.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1035.0, "power.draw": 14.88, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:04.322+00:00", "free_ram_gb": 8.011, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7306.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1170.0, "power.draw": 17.06, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:09.304+00:00", "free_ram_gb": 7.908, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7306.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 930.0, "power.draw": 22.87, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:14.347+00:00", "free_ram_gb": 7.81, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7308.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1020.0, "power.draw": 16.5, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:19.347+00:00", "free_ram_gb": 7.968, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7308.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 960.0, "power.draw": 15.92, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:24.767+00:00", "free_ram_gb": 7.97, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7308.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 1012.0, "power.draw": 15.17, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:29.385+00:00", "free_ram_gb": 7.926, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7310.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1192.0, "power.draw": 18.54, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:34.349+00:00", "free_ram_gb": 7.961, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7310.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 982.0, "power.draw": 17.22, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:39.351+00:00", "free_ram_gb": 7.962, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7312.0, "memory.total": 8151.0, "utilization.gpu": 23.0, "temperature.gpu": 50.0, "clocks.sm": 1147.0, "power.draw": 18.94, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:44.867+00:00", "free_ram_gb": 7.963, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7312.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1027.0, "power.draw": 20.84, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:49.415+00:00", "free_ram_gb": 7.924, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7314.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 50.0, "clocks.sm": 1110.0, "power.draw": 18.48, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:54.400+00:00", "free_ram_gb": 7.973, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7314.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 50.0, "clocks.sm": 982.0, "power.draw": 24.86, "compute_pids": [6068]}}+{"t": "2026-10-23T14:58:59.976+00:00", "free_ram_gb": 7.984, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7314.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1080.0, "power.draw": 15.59, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:04.531+00:00", "free_ram_gb": 8.013, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7316.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 50.0, "clocks.sm": 1200.0, "power.draw": 19.24, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:09.502+00:00", "free_ram_gb": 8.006, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7316.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1042.0, "power.draw": 15.87, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:14.516+00:00", "free_ram_gb": 8.015, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7318.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 49.0, "clocks.sm": 1147.0, "power.draw": 18.39, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:19.511+00:00", "free_ram_gb": 8.016, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7318.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 50.0, "clocks.sm": 1005.0, "power.draw": 15.77, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:24.541+00:00", "free_ram_gb": 8.035, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7320.0, "memory.total": 8151.0, "utilization.gpu": 16.0, "temperature.gpu": 49.0, "clocks.sm": 1125.0, "power.draw": 17.86, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:29.532+00:00", "free_ram_gb": 8.069, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7320.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 17.26, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:35.062+00:00", "free_ram_gb": 8.078, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7322.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 17.91, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:39.601+00:00", "free_ram_gb": 8.071, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7322.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 930.0, "power.draw": 17.19, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:44.704+00:00", "free_ram_gb": 8.088, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7324.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 49.0, "clocks.sm": 1065.0, "power.draw": 17.9, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:49.599+00:00", "free_ram_gb": 8.094, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7324.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 51.0, "clocks.sm": 1695.0, "power.draw": 24.44, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:54.572+00:00", "free_ram_gb": 8.095, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7326.0, "memory.total": 8151.0, "utilization.gpu": 20.0, "temperature.gpu": 50.0, "clocks.sm": 1050.0, "power.draw": 16.92, "compute_pids": [6068]}}+{"t": "2026-10-23T14:59:59.705+00:00", "free_ram_gb": 8.091, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7326.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1110.0, "power.draw": 18.02, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:05.233+00:00", "free_ram_gb": 8.09, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7326.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1117.0, "power.draw": 14.87, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:09.590+00:00", "free_ram_gb": 8.104, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7328.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 49.0, "clocks.sm": 1125.0, "power.draw": 17.33, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:14.633+00:00", "free_ram_gb": 8.094, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7328.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1042.0, "power.draw": 16.27, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:19.739+00:00", "free_ram_gb": 8.09, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7330.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 49.0, "clocks.sm": 1252.0, "power.draw": 18.34, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:24.635+00:00", "free_ram_gb": 8.086, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7330.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1132.0, "power.draw": 15.65, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:29.658+00:00", "free_ram_gb": 8.079, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7332.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 49.0, "clocks.sm": 1140.0, "power.draw": 16.83, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:34.799+00:00", "free_ram_gb": 8.08, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7332.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 49.0, "clocks.sm": 1147.0, "power.draw": 18.58, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:39.872+00:00", "free_ram_gb": 8.081, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7332.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 990.0, "power.draw": 15.53, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:44.669+00:00", "free_ram_gb": 8.068, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7334.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 49.0, "clocks.sm": 1140.0, "power.draw": 18.24, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:49.676+00:00", "free_ram_gb": 8.051, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7334.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1117.0, "power.draw": 15.22, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:55.357+00:00", "free_ram_gb": 8.087, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7336.0, "memory.total": 8151.0, "utilization.gpu": 19.0, "temperature.gpu": 49.0, "clocks.sm": 1057.0, "power.draw": 17.59, "compute_pids": [6068]}}+{"t": "2026-10-23T15:00:59.727+00:00", "free_ram_gb": 8.064, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7336.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1020.0, "power.draw": 16.61, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:04.740+00:00", "free_ram_gb": 8.044, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7338.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 49.0, "clocks.sm": 1057.0, "power.draw": 16.96, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:09.761+00:00", "free_ram_gb": 8.028, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7338.0, "memory.total": 8151.0, "utilization.gpu": 22.0, "temperature.gpu": 50.0, "clocks.sm": 990.0, "power.draw": 17.75, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:14.973+00:00", "free_ram_gb": 8.031, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7338.0, "memory.total": 8151.0, "utilization.gpu": 7.0, "temperature.gpu": 49.0, "clocks.sm": 975.0, "power.draw": 13.51, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:19.992+00:00", "free_ram_gb": 8.088, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7340.0, "memory.total": 8151.0, "utilization.gpu": 10.0, "temperature.gpu": 49.0, "clocks.sm": 712.0, "power.draw": 14.16, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:24.783+00:00", "free_ram_gb": 8.092, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7340.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 675.0, "power.draw": 13.05, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:29.821+00:00", "free_ram_gb": 8.138, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7340.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 50.0, "clocks.sm": 967.0, "power.draw": 14.35, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:34.866+00:00", "free_ram_gb": 8.192, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7342.0, "memory.total": 8151.0, "utilization.gpu": 18.0, "temperature.gpu": 50.0, "clocks.sm": 1095.0, "power.draw": 17.53, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:39.867+00:00", "free_ram_gb": 8.132, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7342.0, "memory.total": 8151.0, "utilization.gpu": 13.0, "temperature.gpu": 50.0, "clocks.sm": 1687.0, "power.draw": 23.97, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:44.861+00:00", "free_ram_gb": 8.134, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7342.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1102.0, "power.draw": 15.62, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:49.862+00:00", "free_ram_gb": 8.135, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7344.0, "memory.total": 8151.0, "utilization.gpu": 22.0, "temperature.gpu": 50.0, "clocks.sm": 1080.0, "power.draw": 17.83, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:55.098+00:00", "free_ram_gb": 8.141, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7344.0, "memory.total": 8151.0, "utilization.gpu": 17.0, "temperature.gpu": 49.0, "clocks.sm": 1035.0, "power.draw": 15.96, "compute_pids": [6068]}}+{"t": "2026-10-23T15:01:59.870+00:00", "free_ram_gb": 8.134, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7346.0, "memory.total": 8151.0, "utilization.gpu": 15.0, "temperature.gpu": 50.0, "clocks.sm": 1050.0, "power.draw": 17.75, "compute_pids": [6068]}}+{"t": "2026-10-23T15:02:04.902+00:00", "free_ram_gb": 8.127, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7346.0, "memory.total": 8151.0, "utilization.gpu": 12.0, "temperature.gpu": 49.0, "clocks.sm": 1080.0, "power.draw": 15.56, "compute_pids": [6068]}}+{"t": "2026-10-23T15:02:09.917+00:00", "free_ram_gb": 8.129, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7348.0, "memory.total": 8151.0, "utilization.gpu": 9.0, "temperature.gpu": 49.0, "clocks.sm": 1185.0, "power.draw": 17.77, "compute_pids": [6068]}}+{"t": "2026-10-23T15:02:14.929+00:00", "free_ram_gb": 8.123, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7348.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1080.0, "power.draw": 15.64, "compute_pids": [6068]}}+{"t": "2026-10-23T15:02:19.936+00:00", "free_ram_gb": 8.118, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7350.0, "memory.total": 8151.0, "utilization.gpu": 14.0, "temperature.gpu": 49.0, "clocks.sm": 1192.0, "power.draw": 17.65, "compute_pids": [6068]}}+{"t": "2026-10-23T15:02:25.085+00:00", "free_ram_gb": 8.114, "tree_rss_gb": 7.603, "peak_wset_gb": 7.603, "on_ac": true, "disk_free_gb": 251.47, "gpu": {"memory.used": 7350.0, "memory.total": 8151.0, "utilization.gpu": 11.0, "temperature.gpu": 48.0, "clocks.sm": 382.0, "power.draw": 18.41, "compute_pids": [6068]}}diff --git a/tests/test_moe_stream_step.py b/tests/test_moe_stream_step.pynew file mode 100644index 00000000..0380695d--- /dev/null+++ b/tests/test_moe_stream_step.py@@ -0,0 +1,244 @@+"""benchmarks/harness/moe_stream_step.py: the loss window and the per-step I/O counters.++``--loss-tokens N`` computes the loss on the last N positions only, so a long step does+not hold vocabulary-sized logits for every position (seq-scaling probe,+``benchmarks/probe-rtx5070-giant-moe-seq-scaling.md``). The per-step ``os_io`` record+is what that probe uses to show where the bytes came from.+"""++from __future__ import annotations++import importlib.util+import json+import math+import sys+import types+from pathlib import Path++import pytest++CUDA = pytest.mark.gpu(reason="NF4 dequantisation needs bitsandbytes on CUDA")+++def _harness(name):+ harness = Path(__file__).resolve().parent.parent / "benchmarks" / "harness"+ if str(harness) not in sys.path:+ sys.path.insert(0, str(harness))+ spec = importlib.util.spec_from_file_location(name, harness / f"{name}.py")+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module+++def _ids(rows, seq):+ numpy = pytest.importorskip("numpy")+ return numpy.arange(rows * seq).reshape(rows, seq)+++# -- --loss-tokens ----------------------------------------------------------+++def test_loss_tokens_defaults_to_the_full_loss():+ step = _harness("moe_stream_step")+ args = step._args(["--store", "s", "--out", "o"])+ assert args.loss_tokens == 0+ ids = _ids(1, 8)+ labels, extra = step._loss_inputs(ids, args.loss_tokens)+ assert labels is ids+ assert extra == {}++[email protected]("window", [8, 9, 4096])+def test_a_window_at_or_past_the_sequence_is_the_full_loss(window):+ step = _harness("moe_stream_step")+ ids = _ids(2, 8)+ labels, extra = step._loss_inputs(ids, window)+ assert labels is ids+ assert extra == {}+++def test_a_short_window_keeps_the_last_positions_of_every_row():+ step = _harness("moe_stream_step")+ ids = _ids(2, 8)+ labels, extra = step._loss_inputs(ids, 3)+ assert labels.tolist() == [[5, 6, 7], [13, 14, 15]]+ assert extra == {"logits_to_keep": 3}++[email protected]("value", ["-1", "1.5", "many"])+def test_loss_tokens_refuses_a_value_that_is_not_a_count(value, capsys):+ step = _harness("moe_stream_step")+ with pytest.raises(SystemExit):+ step._args(["--store", "s", "--out", "o", "--loss-tokens", value])+ assert "--loss-tokens must be a whole number >= 0" in capsys.readouterr().err+++def test_forward_loss_passes_the_window_to_the_model():+ step = _harness("moe_stream_step")+ seen = {}++ def model(**kwargs):+ seen.update(kwargs)+ return types.SimpleNamespace(loss="L")++ ids = _ids(1, 8)+ loss, positions = step._forward_loss(model, ids, 2)+ assert loss == "L"+ assert positions == 2+ assert seen["input_ids"] is ids+ assert seen["labels"].tolist() == [[6, 7]]+ assert seen["logits_to_keep"] == 2++ seen.clear()+ loss, positions = step._forward_loss(model, ids, 0)+ assert positions == 8+ assert seen["labels"] is ids+ assert "logits_to_keep" not in seen+++def test_the_window_is_the_next_token_loss_of_the_last_positions():+ """Through a real (tiny, CPU) DeepSeek-V3: the windowed loss is the cross entropy of+ the last N logits against the ids one position later, and the gradient still+ reaches a position outside the window."""+ torch = pytest.importorskip("torch")+ transformers = pytest.importorskip("transformers")+ step = _harness("moe_stream_step")+ synth = _harness("synth_moe_store")+ config = transformers.DeepseekV3Config(**synth.config_dict(synth.SHAPES["tiny"]))+ torch.manual_seed(0)+ model = transformers.AutoModelForCausalLM.from_config(config).float().eval()+ seq, window = 24, 6+ ids = torch.arange(seq).reshape(1, seq) * 5 % config.vocab_size++ loss, positions = step._forward_loss(model, ids, window)+ assert positions == window+ assert math.isfinite(float(loss.detach()))+ with torch.no_grad():+ logits = model(input_ids=ids).logits[:, -window:, :]+ expected = torch.nn.functional.cross_entropy(logits[0, :-1].float(), ids[0, seq - window + 1 :])+ assert float(loss.detach()) == pytest.approx(float(expected), rel=1e-5, abs=1e-6)++ loss.backward()+ grad = model.get_input_embeddings().weight.grad+ first = int(ids[0, 0])+ assert first not in ids[0, seq - window :].tolist()+ assert float(grad[first].abs().sum()) > 0.0+++# -- per-step I/O counters --------------------------------------------------+++class _FakePsutil(types.ModuleType):+ class Error(Exception):+ pass++ def __init__(self, process_bytes, disks):+ super().__init__("psutil")+ self._process_bytes = process_bytes+ self._disks = disks++ def Process(self): # noqa: N802 - psutil's name+ outer = self++ class _Process:+ def io_counters(self):+ return types.SimpleNamespace(read_bytes=outer._process_bytes)++ return _Process()++ def disk_io_counters(self, perdisk=False):+ assert perdisk is True+ return {+ name: types.SimpleNamespace(read_bytes=value) for name, value in self._disks.items()+ }+++def test_io_snapshot_reads_the_process_and_every_drive(monkeypatch):+ step = _harness("moe_stream_step")+ fake = _FakePsutil(1_000, {"PhysicalDrive0": 10, "PhysicalDrive1": 20})+ monkeypatch.setitem(sys.modules, "psutil", fake)+ assert step._io_snapshot() == {+ "process_read_bytes": 1_000,+ "disk_read_bytes": {"PhysicalDrive0": 10, "PhysicalDrive1": 20},+ }+++def test_io_snapshot_without_psutil_says_none(monkeypatch):+ step = _harness("moe_stream_step")+ monkeypatch.setitem(sys.modules, "psutil", None)+ assert step._io_snapshot() == {"process_read_bytes": None, "disk_read_bytes": None}+++def test_io_snapshot_survives_counters_the_platform_refuses(monkeypatch):+ step = _harness("moe_stream_step")+ fake = _FakePsutil(0, {})++ def refuse(perdisk=False):+ raise RuntimeError("couldn't find any physical disk")++ def no_process():+ raise fake.Error("access denied")++ fake.disk_io_counters = refuse+ fake.Process = no_process+ monkeypatch.setitem(sys.modules, "psutil", fake)+ assert step._io_snapshot() == {"process_read_bytes": None, "disk_read_bytes": None}+++def test_io_delta_is_the_bytes_read_between_two_snapshots():+ step = _harness("moe_stream_step")+ before = {"process_read_bytes": 100, "disk_read_bytes": {"PhysicalDrive0": 5, "D1": 7}}+ after = {"process_read_bytes": 1_600, "disk_read_bytes": {"PhysicalDrive0": 5, "D1": 1_507}}+ assert step._io_delta(before, after) == {+ "process_read_bytes": 1_500,+ "disk_read_bytes": {"PhysicalDrive0": 0, "D1": 1_500},+ }+++def test_io_delta_drops_a_drive_seen_only_once_and_keeps_none():+ step = _harness("moe_stream_step")+ before = {"process_read_bytes": None, "disk_read_bytes": {"A": 1}}+ after = {"process_read_bytes": 9, "disk_read_bytes": {"A": 4, "B": 50}}+ assert step._io_delta(before, after) == {+ "process_read_bytes": None,+ "disk_read_bytes": {"A": 3},+ }+ none = {"process_read_bytes": None, "disk_read_bytes": None}+ assert step._io_delta(none, after) == {"process_read_bytes": None, "disk_read_bytes": None}+++# -- the wiring, end to end (GPU only) --------------------------------------+++@CUDA+def test_one_windowed_step_records_the_window_and_the_io(tmp_path):+ pytest.importorskip("bitsandbytes")+ store = str(tmp_path / "store")+ _harness("synth_moe_store").write_store(store, "tiny", log=lambda _line: None)+ out = tmp_path / "step.json"+ assert (+ _harness("moe_stream_step").main(+ [+ "--store",+ store,+ "--out",+ str(out),+ "--seq",+ "16",+ "--steps",+ "1",+ "--no-pin",+ "--route",+ "uniform",+ "--loss-tokens",+ "4",+ ]+ )+ == 0+ )+ report = json.loads(out.read_text(encoding="utf-8"))+ assert report["args"]["loss_tokens"] == 4+ (entry,) = report["steps"]+ assert entry["loss_finite"] and entry["grads_finite"]+ assert entry["loss_positions"] == 4+ assert entry["os_io"]["process_read_bytes"] >= entry["expert_bytes_read"] > 0