Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
feat(stream): training.stream_recompute keeps streamed activations instead of recomputing (#2152)
Alpamys Makazhan2d562107a67b
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 79a5b372..d65a1612 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -57,6 +57,7 @@ They are the evidence behind the preprint:| [`gate-1418-ab-cs-accept.md`](gate-1418-ab-cs-accept.md) | `soup ab`'s accept rule ([#1418](https://github.com/MakazhanAlpamys/Soup/issues/1418)): `accept_h0` once the always-valid confidence sequence for the difference, the #1265 mixture inverted at level alpha, lies inside +-effect_size, instead of the Bayes factor falling to `log(beta/(1-alpha))`; `--beta` retired | **Accept 2.0-2.4x sooner with no difference** at alpha 0.05: 307 pairs at effect_size / sigma 0.3 (was 621), 117 at 0.5 (was 281), runs to 1000 rows. `reject_h0` rule unchanged; Type-I only falls (worst 0.0328 to 0.0289 at alpha 0.05). Wrong `accept_h0` at a true difference of exactly effect_size at most **0.0083** at alpha 0.05 and 0.0019 at 0.01 (bound alpha; was 0.0042 / 0.0032), power there within 0.005 of before at alpha 0.05 and no lower from 0.01 down. Level beta measured too and not taken (power 0.965 at beta 0.20) || [`gate-1702-arena-packing.md`](gate-1702-arena-packing.md) | Largest-first packing of the pinned host-memory arenas on the layer-streaming RAM tier ([#1702](https://github.com/MakazhanAlpamys/Soup/issues/1702), shipped in [#1713](https://github.com/MakazhanAlpamys/Soup/pull/1713)); Qwen3-8B NF4 on the RTX 5070 Laptop 8 GB | **Memory saving confirmed, no speed claim.** The RAM tier page-locks 6,442,450,944 bytes where the in-order plan page-locks 8,589,934,592, and three deterministic optimizer steps are byte-identical between the two plans (870 of 870 digests). The 40-run speed series measured +1.96% on the RAM tier, inside the +/-15.13% band its own control set, so by the rule written beforehand no speed-up and no slow-down is claimed; the band is wide and cannot exclude a smaller change || [`gate-1892-nf4-scale-cache.md`](gate-1892-nf4-scale-cache.md) | End-to-end A/B of the opt-in `training.stream_nf4_scale_cache` ([#1892](https://github.com/MakazhanAlpamys/Soup/pull/1892)) on a streamed Mistral-7B NF4 run | **Confirmed on one laptop:** median paired reduction 3.387 % [2.389, 6.936] over five AB/BA pairs against a rule of >= 2 % with lower bound > 0; +0.446 GB VRAM. The companion `--train-threads 1` test showed no gain (median -0.555 %) and is dropped. Exactness is covered by the PR tests, not by this record. |+| [`gate-stream-recompute.md`](gate-stream-recompute.md) | `training.stream_recompute`: streamed layers that keep their activations instead of the checkpoint recompute, on a streamed Mistral-7B NF4 run (RTX 5070 Laptop 8 GB, a new baseline) | **Measured on one laptop:** -22 % / -17 % / -8.5 % step time on the RAM tier at 512 / 1024 / 2048 tokens per step (32 / 24 / 12 of 32 layers fit), loss and 256 of 256 LoRA gradients bit-identical under deterministic algorithms; no gain on a read-bound disk tier. `soup train` with `auto` chose 32/32 layers at max_length 512 and 10/32 at 2048, train_loss equal to `on`. || [`gate-fp8-shard.md`](gate-fp8-shard.md) | Sharding a DeepSeek-V3 FP8 block-quantised checkpoint by dequantising it at shard time (CPU only): dequant exactness, streamed-from-FP8 vs resident parity on a tiny `DeepseekV3ForCausalLM`, and Moonlight-16B-A3B converted to the format and sharded | **G1, G2, G3 pass.** Bit-exact against an independent per-block reference (and transformers' own dequant where its block rule applies); logits, loss and LoRA gradients bit-exact in both expert layouts, caches tensor-for-tensor equal incl. NF4; the 27-layer NF4 cache of the FP8 copy came within 0.0002% (0.000115%) of the pre-flight estimate, and every checked FP8-derived tensor equals the reference. No GPU run || [`probe-rtx5070-moonlight-fp8-source.md`](probe-rtx5070-moonlight-fp8-source.md) | The GPU loss check `gate-fp8-shard.md` leaves to the maintainer: does a `soup train` LoRA run on the FP8-block copy of Moonlight-16B-A3B train like the same run on its bf16 source? Two arms, the same code and config apart from `base` and `output`, fused NF4 caches; limits on the step-1, per-step, epoch-3 and evaluation loss differences, GPU peak and seconds per step, set from a matched-size reference pair measured before the run | **FAIL (T2_max_step, T6_step_time)**, 2026-10-20. Both arms complete on the 8 GB card at the same 3.1 GiB GPU peak. Step-1, final-evaluation and epoch-3-mean losses agree to 0.081, 0.072 and 0.101, within their limits, but the 5-token row diverges by 3.27 nats at step 7 (limit 1.6), and B's median step took 1.91x A's (limit 1.10). A diagnosis after the verdict finds the cache layouts identical and the reader on direct I/O in both arms, with B's files reading at A's rate or faster on a rested drive; the most likely cause of T6 is the D: drive's sustained-read plateau (section 6), with B's timed steps starting 17 s after its own shard I/O. The rule's cool-down covered the GPU, not the drive. Speed parity and per-step parity are not shown |@@ -129,6 +130,8 @@ only in a scratchpad on a machine that is gone, which is| [`fp8_loss_check_compare.py`](harness/fp8_loss_check_compare.py) | [`probe-rtx5070-moonlight-fp8-source.md`](probe-rtx5070-moonlight-fp8-source.md) §2 | The verdict from the two arm folders: COMPLETES per arm, T1-T6 with the committed limits, the INCONCLUSIVE conditions, per-step and per-epoch tables | CPU; seconds || [`fp8_loss_check_layout.py`](harness/fp8_loss_check_layout.py) | [`probe-rtx5070-moonlight-fp8-source.md`](probe-rtx5070-moonlight-fp8-source.md) §6 | Do two shard caches differ in anything a disk reader sees? Per safetensors file: header length, data start, size, metadata, tensor order, every tensor's dtype, shape, length and offset (mod 4096 and 512 for the large ones), then `index.json` field by field. Headers only, read-only | CPU; seconds || [`fp8_loss_check_readbench.py`](harness/fp8_loss_check_readbench.py) | [`probe-rtx5070-moonlight-fp8-source.md`](probe-rtx5070-moonlight-fp8-source.md) §6 | GB/s of one layer file read the way the disk tier reads it (`open_direct`, 4 aligned ranges, 4 threads), alternating two caches; refuses below 10 GB free. No torch import | Windows NVMe; ~1 GB per read, 9 GB as committed |+| [`stream_recompute_ab.py`](harness/stream_recompute_ab.py) | [`gate-stream-recompute.md`](gate-stream-recompute.md) section 4 | Interleaved ON vs OFF-for-the-last-K blocks over several batch x seq shapes in one process, K from one layer's retained bytes after the forward under an allocated-peak cap | CUDA GPU required; Mistral-7B NF4: minutes per shape |+| [`stream_recompute_exactness.py`](harness/stream_recompute_exactness.py) | [`gate-stream-recompute.md`](gate-stream-recompute.md) section 3 | Loss and every LoRA gradient, recompute ON vs OFF, bit for bit under deterministic algorithms and MATH sdpa, with an ON-vs-ON control and a one-token negative control; exits 1 on a failed gate | CUDA GPU required; Mistral-7B NF4 at 1 x 512: about a minute after sharding |## Hardwarediff --git a/benchmarks/gate-stream-recompute.md b/benchmarks/gate-stream-recompute.mdnew file mode 100644index 00000000..bd02afbd--- /dev/null+++ b/benchmarks/gate-stream-recompute.md@@ -0,0 +1,154 @@+<!--+Measurement record, published as measured.++Hardware: RTX 5070 Laptop GPU 8 GB (Blackwell sm_120) — a NEW baseline, never compared+with RTX 3050 numbers. Intel i9-14900HX, Windows 11, AC power.+Stack: torch 2.14.0+cu130, bitsandbytes 0.50.2, transformers 5.17.0.+Model: unsloth/mistral-7b-instruct-v0.3, NF4 with double quantisation, layer-streamed,+RAM tier unless a row says otherwise; one synthetic Qwen2.5-14B row on the disk tier.+Soup: the measurements in sections 1-4 ran on the local experiment branch+exp/1c-nf4-dequant (base 1cff0ae45, 2026-10-21/22); section 5 on the feature as ported.+Harness: benchmarks/harness/stream_recompute_exactness.py (section 3) and+benchmarks/harness/stream_recompute_ab.py (section 4), ported here from the experiment+branch's exp1c/exact_e2e.py and exp1c/ab_shapes.py; the micro-benchmarks behind sections 1+and 2 (micro1.py, micro2_zero.py, micro3b_recompute.py) and the e2e A/B scripts behind the+section 3 speed table (ab_e2e.py, ab_e2e_sc.py, ab_e2e_tail.py) stay on that branch.+-->++# Gate record — `training.stream_recompute`: streamed layers without the checkpoint recompute++**Status: MEASURED on one laptop (RTX 5070 Laptop 8 GB).** On the RAM tier, skipping the+checkpoint recompute takes **22 % / 17 % / 8.5 %** off a Mistral-7B NF4 step at 512 /+1024 / 2048 tokens per step, as far as the activations fit (all 32 layers at 512, 24 at+1024, 12 at 2048). Loss and all 256 LoRA gradients are **bit-identical** to the recompute+under deterministic algorithms. On a read-bound disk tier there is **no gain**, and at a+high VRAM fill it is slower. `soup train` with `stream_recompute: auto` chose 32/32 layers+at max_length 512 and 10/32 at 2048, with the same train_loss as `on` per shape.++Unit convention: **decimal GB**, matching the other records.++---++## 1. Where the dequantisation ceiling is (micro, synthetic Mistral-7B shapes, M = 512)++| | per decoder layer |+|---|---|+| copy roofline (device) | 333 GB/s (fill 357 GB/s) |+| bnb `dequantize_4bit`, nested scales | 2.29 ms |+| same, expanded fp32 scales (#1892) | 2.00 ms (gate/up/down at 290-294 GB/s = ~88 % of copy roofline) |+| 7 forward GEMMs (cuBLAS, bf16) | 5.39 ms (40-43 TFLOPS) |+| dequant on a side stream overlapped with the GEMMs | no gain (6.90 vs 6.65 ms serial) |++- bitsandbytes' dequantisation is already near the bandwidth roofline: a faster dequant+ kernel cannot buy much.+- 64 visits x 2.29 ms = 147 ms, about 14 % of a 1.05 s step.+- An earlier "-23 % without dequantisation" figure multiplied by an ALL-ZERO cached weight.+ On this power-limited card a zero-weight GEMM runs 7.6 % faster than a real one (5.01 vs+ 5.42 ms per layer, 4 interleaved rounds, consistent), so part of that figure was a+ power/clock artefact; the dequantisation's real share is about 14-18 %.++## 2. The bigger lever: the recompute++A streamed layer runs its forward twice: in the forward, and again as the checkpoint+recompute in the backward. When the activations fit in VRAM, the recompute can be dropped.+Pool-slot tensors must still never be saved as live views (#331): the layer runs under+`torch.autograd.graph.saved_tensors_hooks`, saves pool tensors as tokens, and at its first+backward unpack does the same `pool.wait` + `prefetcher.advance` the recompute did. NF4+linears save only the packed bytes (`packed_dequant_nf4_linear`, `save_for_backward`) and+dequantise again for `grad_x`, so no dense weight is kept.++One layer, 1 x 512: checkpoint 23.1 ms; no recompute with the packed save 18.6 ms (-19.5 %,+keeps 76 MB per layer); no recompute with a dense save 16.9 ms (-26.5 %, keeps 492 MB per+layer). All three bit-identical under deterministic mode and the MATH sdpa backend. The+packed save is what ships: the dense one costs 6.5x the VRAM per layer.++## 3. Real streamed Mistral-7B NF4, RAM tier, 1 x 512++**Exactness** (`stream_recompute_exactness.py`: deterministic algorithms, MATH sdpa, LoRA B+given non-zero values so every gradient is non-zero): ON vs ON, OFF vs ON and OFF vs OFF —+loss equal, **0 of 256 LoRA gradients differ**; the negative control (one input token+changed) differs in 256 of 256. A first run reported "equal" vacuously because the switch+found 0 layers (the wrapper is transparent to `named_modules()`); the harness now refuses+unless every layer was switched.++**Speed** (default sdpa, forward + backward + PagedAdamW8bit, arms interleaved in ONE+process, 3 warm-up + 8 timed steps per block, 2 rounds):++| arm | step | tok/s | vs recompute | peak VRAM |+|---|---|---|---|---|+| recompute (`on`) | 0.764-0.770 s | 665-670 | — | 1.34 GB |+| `on` + `stream_nf4_scale_cache` | 0.761 s | 673 | -1.1 % | — |+| no recompute (`off`) | 0.608-0.610 s | 839-842 | **-20.5 / -21.0 %** | 4.20 GB |+| `off` + scale cache | 0.598-0.600 s | 853-856 | **-21.8 / -22.1 %** | 4.65 GB |+| `off` + dense weight kept for the last 2 / 4 layers | 0.594 / 0.597 s | 862 / 858 | -22.3 / -21.9 % (noise) | 5.5 / 6.4 GB |++Round-to-round spread within an arm: under 0.5 %. The dense-weight option measured no gain+over the packed save and is not part of the feature.++## 4. Wider shapes, the disk tier, a model that does not fit++`stream_recompute_ab.py`: one process per model and tier, scale cache on, arms ON / OFF+interleaved, 2 rounds. OFF skips the recompute only for the LAST K layers+(`set_stream_recompute(..., retain=K)`), K the largest that keeps the allocated peak under a+cap. One layer's retained bytes are what stays allocated after the forward (OFF with K = 2+minus ON). An earlier version used the difference of PEAKS, which under-estimated a layer+3x at 1024 tokens, picked K = 32, overflowed the card (8.26 GB) and WDDM spilled into host+RAM (OFF 2.3x slower); those two points are discarded.++| model, tier | tokens/step | retained bytes/layer | K of L | ON step | OFF step | gain |+|---|---|---|---|---|---|---|+| Mistral-7B, RAM | 1x512 (section 3) | 0.10 GB | 32/32 | 0.770 s | 0.600 s | 22 % |+| Mistral-7B, RAM | 1x1024 | 0.20 GB | 24/32 | 1.405 s | 1.167 s | 16.9 % |+| Mistral-7B, RAM | 2x512 | 0.20 GB | 24/32 | 1.382 s | 1.139 s | 17.6 % |+| Mistral-7B, RAM | 1x2048 | 0.34 GB* | 12/32 | 3.455 s | 3.163 s | 8.5 % |+| Mistral-7B, RAM | 4x512 | 0.34 GB* | 12/32 | 3.158 s | 2.890 s | 8.5 % |+| Mistral-7B, disk (warm page cache) | 1x512 | 0.10 GB | 32/32 | 1.649 s | 1.503 s | 8.9 % |+| Mistral-7B, disk (warm page cache) | 1x1024 | 0.20 GB | 24/32 | 1.702 s | 1.550 s | 9.0 % |+| synthetic Qwen2.5-14B, disk | 1x512 | 0.11 GB | 24/48 (cap 7 GB) | 3.73 s | 4.37 s | -17 % |+| synthetic Qwen2.5-14B, disk | 1x512 | 0.11 GB | 6/48 (cap 5 GB) | 5.22 s | 5.32 s | -1.8 % (noise) |++\* peak-difference estimate (the discarded method), but K = 12 kept the peak at 7.4 GB, no+spill.++- Compute-bound (RAM tier): the gain is about 22 % x K/L, so it shrinks with tokens per step+ as fewer layers fit.+- Read-bound (the 14B, whose 9.3 GB store does not stay in the page cache): no gain — every+ layer is still read twice per step, since the backward visit re-reads it — and at a high+ VRAM fill it hurts. Disk-tier ON times varied 3.7 vs 5.2 s between runs (other processes+ on C:), so the disk numbers are rough.+- What follows for the feature: `auto` engages only on the RAM tier and budgets K from+ measured retained bytes and the RESERVED peak, with headroom below the card size; `off`+ on the disk tier warns.++## 5. The feature as shipped: `soup train` smoke++`stream_recompute: auto` decides at setup, before step 1: one synthetic forward+backward at+the configured `batch_size` x `data.max_length` (the worst case any batch can pad up to)+with the last layer retained measures that layer's allocated bytes and the step's reserved+peak. A layer is charged at its bytes times the step's reserved/allocated ratio when that+is above 1; the budget is the smallest of 85 % of the card, free + reserved VRAM and+`stream_vram_override`, minus that peak; K = budget // charge. This differs from section+4's method (which capped the allocated peak); the GPU check below is of the shipped planner.++`soup train`, 24 rows, Mistral-7B NF4 streamed, RAM tier:++| | max_length 512 | max_length 2048 |+|---|---|---|+| `auto` K | 32/32 | 10/32 |+| charged per layer | 99.4 MB | 422.8 MB |+| budget | 5.83 GB | 4.47 GB |+| train_loss `auto` vs `on` | equal | equal |++Train runtime: `on` 27.3 s (a first, cold run read 38.3 s), `auto` 21.0 s, `off` 21.2 s.+These three agree with max_length 512, where `auto` chose all 32 layers, the same as `off`.++## 6. Limits++- One laptop, one model family measured end to end (Mistral-7B NF4); task `sft` only, which+ is what the config allows.+- K errs low by up to one layer: the synthetic step's peak already holds the measured+ layer.+- `auto` costs one extra step at setup.+- The CPU suite (`tests/test_stream_recompute.py`) proves the bit-exactness and the #331+ negative controls on a tiny 4-layer / 2-buffer model; speed and VRAM are measured only+ here.diff --git a/benchmarks/harness/stream_probe.py b/benchmarks/harness/stream_probe.pyindex dc2faa17..6025b741 100644--- a/benchmarks/harness/stream_probe.py+++ b/benchmarks/harness/stream_probe.py@@ -167,6 +167,15 @@ def parse_args() -> argparse.Namespace:"came from the shipped code"),)+ parser.add_argument(+ "--recompute",+ choices=("on", "off"),+ default="on",+ help=(+ "training.stream_recompute: off = every streamed layer keeps its activations "+ "instead of the checkpoint recompute"+ ),+ )parser.add_argument("--out", required=True, help="JSON results file, rewritten per point")parser.add_argument("--label", default="", help="free text stored beside the results")return parser.parse_args()@@ -508,6 +517,7 @@ class Instruments:"self.events[slot].record(stream)","_release_source(source, idx, self.events[slot])","_release_source(source, idx, None)",+ "self.generation[slot] += 1",),)inst = self@@ -541,6 +551,9 @@ class Instruments:pool.buffers[slot][name].copy_(source.get(idx, name))_release_source(source, idx, None)pool.owner[slot] = idx+ # The no-recompute backward refuses a slot whose load generation+ # moved since its first visit; a replacement load must bump it too.+ pool.generation[slot] += 1pool.loads += 1return slot@@ -1097,9 +1110,15 @@ def main() -> int:inst = Instruments(runtime, model, args.quant)if args.quant == "nf4":print(f"{'nf4 linears':<16}{inst.patched_linears} patched for the ablation")+ if args.recompute == "off":+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ switched = set_stream_recompute(model, False)+ print(f"{'recompute':<16}OFF on {switched} streamed layers (stream_recompute='off')")meta = {**facts,+ "recompute": args.recompute,"label": args.label,"weights": args.weights,"weights_dir": weights_dir,diff --git a/benchmarks/harness/stream_recompute_ab.py b/benchmarks/harness/stream_recompute_ab.pynew file mode 100644index 00000000..f876a1d8--- /dev/null+++ b/benchmarks/harness/stream_recompute_ab.py@@ -0,0 +1,227 @@+#!/usr/bin/env python3+"""Interleaved A/B of training.stream_recompute: ON for every layer vs OFF for the last K.++One process, one streamed model (NF4, the scale cache on, as in the record), over several+``batch x seq`` shapes. Per shape, K is the largest number of last layers whose retained+activations keep the allocated peak under ``--cap-gb``: one layer's retained bytes are+measured as what stays allocated after the FORWARD with K=2 minus the same with every layer+recomputing, halved. Not a difference of peaks: peaks are not additive, and a peak+difference under-estimated a Mistral-7B layer 3x at 1024 tokens, picked K=32 and spilled+the card (benchmarks/gate-stream-recompute.md). ``--rounds`` interleave ON and OFF blocks;+each block is ``--warmup`` discarded and ``--steps`` timed steps of forward, backward and a+PagedAdamW8bit step, synchronised, so a step time is a wall clock of one full step.++This is the method behind the record's shape table. ``soup train`` itself chooses K with+``plan_stream_recompute_auto`` (a synthetic step at the configured shape and a reserved-peak+budget); this harness does not use that planner.++Typical invocation::++ python benchmarks/harness/stream_recompute_ab.py \+ --weights unsloth/mistral-7b-instruct-v0.3 --cap-gb 7 --shapes 1x512,1x1024,1x2048++A machine without CUDA is an intentional skip and exits 0.+"""++import argparse+import contextlib+import json+import os+import statistics+import sys+import time+from pathlib import Path+from typing import Any, Dict, Iterator, List, Optional, Tuple++REPO = Path(__file__).resolve().parents[2]+++def _stream_probe() -> Any:+ """Import the sibling probe, whichever directory this ran from."""+ here = str(Path(__file__).resolve().parent)+ if here not in sys.path:+ sys.path.insert(0, here)+ import stream_probe++ return stream_probe+++def _assert_this_tree() -> str:+ """The measured code must be this checkout's, not a same-version copy elsewhere."""+ import soup_cli++ origin = os.path.normcase(os.path.realpath(soup_cli.__file__))+ src = os.path.normcase(os.path.realpath(REPO / "src"))+ assert origin.startswith(src), (+ f"soup_cli is imported from {soup_cli.__file__}, not from {src}; put this checkout's "+ "src first on PYTHONPATH"+ )+ return soup_cli.__file__+++def _shapes(text: str) -> List[Tuple[int, int]]:+ shapes = []+ for item in text.split(","):+ batch, seq = item.strip().lower().split("x")+ shapes.append((int(batch), int(seq)))+ return shapes+++def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:+ parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])+ parser.add_argument("--weights", default="unsloth/mistral-7b-instruct-v0.3")+ parser.add_argument("--shards", default=None, help="an already-sharded directory")+ parser.add_argument("--tier", choices=("ram", "disk"), default="ram")+ parser.add_argument("--cap-gb", type=float, default=7.0, help="allocated-peak cap for K")+ parser.add_argument("--shapes", type=_shapes, default=_shapes("1x1024,2x512,1x2048,4x512"))+ parser.add_argument("--rounds", type=int, default=2)+ parser.add_argument("--warmup", type=int, default=2)+ parser.add_argument("--steps", type=int, default=6)+ parser.add_argument("--lora-r", type=int, default=8)+ parser.add_argument("--lora-targets", default="q_proj,k_proj,v_proj,o_proj")+ parser.add_argument("--seed", type=int, default=3)+ parser.add_argument("--input-seed", type=int, default=17)+ parser.add_argument("--out", default=None, help="JSON results file (default: stdout only)")+ return parser.parse_args(argv)+++def _nf4_scale_cache() -> Iterator[None]:+ """stream_probe.build() has no switch for training.stream_nf4_scale_cache; the record's+ rows were measured with it on, so the builder is wrapped for the build only."""+ import soup_cli.utils.layer_stream_runtime as runtime_module++ original = runtime_module.build_streamed_model++ def with_cache(*args: Any, **kwargs: Any) -> Any:+ kwargs["nf4_scale_cache"] = True+ return original(*args, **kwargs)++ runtime_module.build_streamed_model = with_cache+ try:+ yield+ finally:+ runtime_module.build_streamed_model = original+++def main(argv: Optional[List[str]] = None) -> int:+ cli = parse_args(argv)+ stream_probe = _stream_probe()+ if not stream_probe.cuda_available():+ print("SKIP: no CUDA device")+ return 0+ import bitsandbytes as bnb+ import torch++ from soup_cli.utils.layer_stream import resolve_stream_dtype+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ print("soup_cli from", _assert_this_tree())+ device = "cuda"+ args = argparse.Namespace(+ weights=cli.weights,+ shards=cli.shards,+ quant="nf4",+ tier=cli.tier,+ no_pin=False,+ buffers=2,+ read_ahead=2,+ lora_r=cli.lora_r,+ lora_targets=cli.lora_targets,+ seed=cli.seed,+ lazy_shard_handles=False,+ control_sync_source=False,+ )+ with _nf4_scale_cache():+ model, _runtime, config, *_ = stream_probe.build(+ args, device, resolve_stream_dtype(device)+ )+ model.train()+ n_layers = int(config.num_hidden_layers)+ trainable = [param for param in model.parameters() if param.requires_grad]+ optimizer = bnb.optim.PagedAdamW8bit(trainable, lr=1e-4)+ generator = torch.Generator(device=device).manual_seed(cli.input_seed)++ def block(ids: Any, retain: Optional[int], warmup: int, steps: int) -> Dict[str, float]:+ if retain is None:+ assert set_stream_recompute(model, True) == n_layers+ else:+ assert set_stream_recompute(model, False, retain=retain) == n_layers+ torch.cuda.synchronize()+ torch.cuda.reset_peak_memory_stats()+ times = []+ kept = 0.0+ for index in range(warmup + steps):+ torch.cuda.synchronize()+ started = time.perf_counter()+ base = torch.cuda.memory_allocated()+ out = model(input_ids=ids, labels=ids)+ kept = (torch.cuda.memory_allocated() - base) / 1e9+ out.loss.backward()+ optimizer.step()+ optimizer.zero_grad(set_to_none=True)+ torch.cuda.synchronize()+ if index >= warmup:+ times.append(time.perf_counter() - started)+ return {+ "seconds": statistics.median(times),+ "peak_gb": torch.cuda.max_memory_allocated() / 1e9,+ "reserved_gb": torch.cuda.max_memory_reserved() / 1e9,+ "kept_after_forward_gb": kept,+ }++ results: Dict[str, Any] = {+ "cap_gb": cli.cap_gb,+ "card_total_gb": torch.cuda.mem_get_info()[1] / 1e9,+ "gpu": torch.cuda.get_device_name(),+ "weights": cli.weights,+ "tier": cli.tier,+ "shapes": [],+ }+ for batch, seq in cli.shapes:+ ids = torch.randint(+ 0, int(config.vocab_size), (batch, seq), generator=generator, device=device+ )+ on = block(ids, None, warmup=1, steps=2)+ two = block(ids, 2, warmup=1, steps=1)+ per_layer = max((two["kept_after_forward_gb"] - on["kept_after_forward_gb"]) / 2, 1e-3)+ k = int(max(0, min(n_layers, (cli.cap_gb - on["peak_gb"]) // per_layer)))+ row: Dict[str, Any] = {+ "batch": batch,+ "seq": seq,+ "per_layer_gb": per_layer,+ "K": k,+ "n_layers": n_layers,+ "rounds": [],+ }+ for _ in range(cli.rounds):+ row["rounds"].append(+ {+ "on": block(ids, None, cli.warmup, cli.steps),+ "off": block(ids, k, cli.warmup, cli.steps),+ }+ )+ on_s = statistics.median(r["on"]["seconds"] for r in row["rounds"])+ off_s = statistics.median(r["off"]["seconds"] for r in row["rounds"])+ row.update(+ {+ "on_s": on_s,+ "off_s": off_s,+ "faster_pct": 100 * (1 - off_s / on_s),+ "on_peak_gb": max(r["on"]["peak_gb"] for r in row["rounds"]),+ "off_peak_gb": max(r["off"]["peak_gb"] for r in row["rounds"]),+ "off_reserved_gb": max(r["off"]["reserved_gb"] for r in row["rounds"]),+ }+ )+ results["shapes"].append(row)+ summary = {key: value for key, value in row.items() if key != "rounds"}+ print(json.dumps(summary), flush=True)+ text = json.dumps(results, indent=1)+ if cli.out:+ Path(cli.out).write_text(text + "\n", encoding="utf-8")+ print(text)+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmarks/harness/stream_recompute_exactness.py b/benchmarks/harness/stream_recompute_exactness.pynew file mode 100644index 00000000..8365e935--- /dev/null+++ b/benchmarks/harness/stream_recompute_exactness.py@@ -0,0 +1,194 @@+#!/usr/bin/env python3+"""Exactness gate for training.stream_recompute on a REAL streamed model: ON vs OFF.++One process. Deterministic algorithms and the MATH sdpa backend, the same input ids, and+the loss and every LoRA gradient compared bit for bit. Two controls make "equal" mean+something: ON vs ON again must be equal (the harness is deterministic), and a NEGATIVE+control, one input token changed, must differ. LoRA B starts at zero, which makes every A+gradient zero, so B is given small random values first. The run refuses if the switch+found a different number of streamed layers than the config has: a first version compared+ON with ON and reported "equal" because the wrapper is transparent to named_modules().++Typical invocation::++ python benchmarks/harness/stream_recompute_exactness.py \+ --weights unsloth/mistral-7b-instruct-v0.3 --seq 512++A machine without CUDA is an intentional skip and exits 0.+"""++import argparse+import json+import os+import sys+import time+from pathlib import Path+from typing import Any, Dict, List, Optional++REPO = Path(__file__).resolve().parents[2]+++def _stream_probe() -> Any:+ """Import the sibling probe, whichever directory this ran from."""+ here = str(Path(__file__).resolve().parent)+ if here not in sys.path:+ sys.path.insert(0, here)+ import stream_probe++ return stream_probe+++def _assert_this_tree() -> str:+ """The measured code must be this checkout's, not a same-version copy elsewhere."""+ import soup_cli++ origin = os.path.normcase(os.path.realpath(soup_cli.__file__))+ src = os.path.normcase(os.path.realpath(REPO / "src"))+ assert origin.startswith(src), (+ f"soup_cli is imported from {soup_cli.__file__}, not from {src}; put this checkout's "+ "src first on PYTHONPATH"+ )+ return soup_cli.__file__+++def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:+ parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])+ parser.add_argument("--weights", default="unsloth/mistral-7b-instruct-v0.3")+ parser.add_argument("--shards", default=None, help="an already-sharded directory")+ parser.add_argument("--quant", choices=("none", "nf4"), default="nf4")+ parser.add_argument("--tier", choices=("ram", "disk"), default="ram")+ parser.add_argument("--seq", type=int, default=512)+ parser.add_argument("--batch", type=int, default=1)+ parser.add_argument("--lora-r", type=int, default=8)+ parser.add_argument("--lora-targets", default="q_proj,k_proj,v_proj,o_proj")+ parser.add_argument("--seed", type=int, default=3)+ parser.add_argument("--input-seed", type=int, default=17)+ parser.add_argument("--out", default=None, help="JSON results file (default: stdout only)")+ return parser.parse_args(argv)+++def main(argv: Optional[List[str]] = None) -> int:+ cli = parse_args(argv)+ # Before the first CUDA context: cuBLAS reads it at handle creation.+ os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")+ stream_probe = _stream_probe()+ if not stream_probe.cuda_available():+ print("SKIP: no CUDA device")+ return 0+ import torch+ from torch.nn.attention import SDPBackend, sdpa_kernel++ from soup_cli.utils.layer_stream import resolve_stream_dtype+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ print("soup_cli from", _assert_this_tree())+ device = "cuda"+ args = argparse.Namespace(+ weights=cli.weights,+ shards=cli.shards,+ quant=cli.quant,+ tier=cli.tier,+ no_pin=False,+ buffers=2,+ read_ahead=2,+ lora_r=cli.lora_r,+ lora_targets=cli.lora_targets,+ seed=cli.seed,+ lazy_shard_handles=False,+ control_sync_source=False,+ )+ model, _runtime, config, *_ = stream_probe.build(args, device, resolve_stream_dtype(device))+ model.train()+ lora = [(name, param) for name, param in model.named_parameters() if param.requires_grad]+ b_generator = torch.Generator(device=device).manual_seed(5)+ with torch.no_grad():+ for name, param in lora:+ if "lora_B" in name:+ noise = torch.randn(+ param.shape, generator=b_generator, device=device, dtype=param.dtype+ )+ param.copy_(noise * 0.01)+ n_layers = int(config.num_hidden_layers)+ switched = set_stream_recompute(model, False)+ assert switched == n_layers, f"the switch found {switched} streamed layers of {n_layers}"+ generator = torch.Generator(device=device).manual_seed(cli.input_seed)+ vocab = int(config.vocab_size)+ ids = torch.randint(0, vocab, (cli.batch, cli.seq), generator=generator, device=device)+ ids_negative = ids.clone()+ position = min(100, cli.seq - 1)+ ids_negative[0, position] = (ids_negative[0, position] + 1) % vocab+ torch.use_deterministic_algorithms(True)++ def run(recompute: bool, inputs: Any) -> Dict[str, Any]:+ set_stream_recompute(model, recompute)+ for _, param in lora:+ param.grad = None+ torch.cuda.reset_peak_memory_stats()+ started = time.perf_counter()+ with sdpa_kernel([SDPBackend.MATH]):+ base = torch.cuda.memory_allocated()+ out = model(input_ids=inputs, labels=inputs)+ torch.cuda.synchronize()+ kept = torch.cuda.memory_allocated() - base+ out.loss.backward()+ torch.cuda.synchronize()+ return {+ "loss": out.loss.detach().clone(),+ "grads": [param.grad.detach().clone() for _, param in lora],+ "seconds": time.perf_counter() - started,+ "kept_after_forward_gb": kept / 1e9,+ "peak_gb": torch.cuda.max_memory_allocated() / 1e9,+ }++ def compare(a: Dict[str, Any], b: Dict[str, Any]) -> Dict[str, Any]:+ differing = sum(+ 0 if torch.equal(x, y) else 1 for x, y in zip(a["grads"], b["grads"])+ )+ return {+ "loss_equal": bool(torch.equal(a["loss"], b["loss"])),+ "grads_differing": differing,+ "n_grads": len(a["grads"]),+ }++ on_1 = run(True, ids)+ on_2 = run(True, ids)+ off_1 = run(False, ids)+ off_2 = run(False, ids)+ negative = run(False, ids_negative)+ results = {+ "gpu": torch.cuda.get_device_name(),+ "weights": cli.weights,+ "shape": [cli.batch, cli.seq],+ "switched_layers": switched,+ "on_vs_on": compare(on_1, on_2),+ "off_vs_on": compare(off_1, on_1),+ "off_vs_off": compare(off_1, off_2),+ "negative_control_vs_on (must differ)": compare(negative, on_1),+ "seconds": {+ "on_1": on_1["seconds"],+ "on_2": on_2["seconds"],+ "off_1": off_1["seconds"],+ "off_2": off_2["seconds"],+ },+ "kept_after_forward_gb": {+ "on": on_2["kept_after_forward_gb"],+ "off": off_2["kept_after_forward_gb"],+ },+ "peak_gb": {"on": on_2["peak_gb"], "off": off_2["peak_gb"]},+ "nonzero_grads": sum(int(bool(grad.abs().sum() > 0)) for grad in on_1["grads"]),+ }+ text = json.dumps(results, indent=1)+ if cli.out:+ Path(cli.out).write_text(text + "\n", encoding="utf-8")+ print(text)+ ok = (+ results["on_vs_on"]["grads_differing"] == 0+ and results["off_vs_on"]["grads_differing"] == 0+ and results["off_vs_on"]["loss_equal"]+ and results["negative_control_vs_on (must differ)"]["grads_differing"] > 0+ )+ return 0 if ok else 1+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/changelog.d/0.76.0/2152.added.md b/changelog.d/0.76.0/2152.added.mdnew file mode 100644index 00000000..680ae248--- /dev/null+++ b/changelog.d/0.76.0/2152.added.md@@ -0,0 +1,6 @@+- Add `training.stream_recompute: on | off | auto` for layer streaming (SFT). `off` keeps each+ streamed layer's activations instead of re-running its forward as the checkpoint recompute;+ `auto` measures one step at the configured shape at setup and keeps activations for as many of+ the last layers as fit in VRAM (RAM tier only). Gradients are bit-identical to the recompute+ path; on an RTX 5070 Laptop 8 GB a streamed Mistral-7B NF4 step is 22% / 17% / 8.5% faster at+ 512 / 1024 / 2048 tokens. The default `on` keeps the shipped behaviour. (#2152)diff --git a/docs/config-reference.md b/docs/config-reference.mdindex 41ecb9b6..8e6307e5 100644--- a/docs/config-reference.md+++ b/docs/config-reference.md@@ -321,6 +321,7 @@ Sections:| `training.stream_vram_override` | `int \| null` | `null` | `>= 0` | Bytes to assume free instead of measuring torch.cuda.mem_get_info(). mem_get_info() is a device-level driver query, so it cannot see a per-process cap (set_per_process_memory_fraction, a MIG slice, a card another process is also using), so on that hardware the pre-flight sees the whole device's free VRAM regardless of what this process can actually use. Set this to replace the measured figure in either direction: raise it to let a known-safe over-prediction through, or lower it to make the pre-flight enforce a cap the driver itself cannot report. || `training.stream_vram_probe` | `bool` | `false` | | Decide the layer-streaming VRAM pre-flight on a MEASUREMENT instead of the fitted formula. One real forward+backward runs at the configured shape after the streamed model is built, and its peak decides whether the run proceeds; the prediction is printed beside it so a divergence is visible. The formula under-predicts past seq 4352 (measured 0.934x the real peak at seq 5120 and 0.787x at 6144), which is the direction that does not announce itself — an OOM on Linux, a silent spill to host memory on Windows. Against the probe the same formula reads 0.992x at seq 4096 and 0.830x at 5120, because the probe itself runs 12.5-14.3% above the real training step — the direction that makes it safe as a gate. Off by default: it costs one step (1-5 s measured) and it can refuse a run the formula accepts. task='sft' only: the probe runs a plain causal-LM forward+backward, which IS the SFT step but not a preference loss. At the one preference shape measured it reads +13.5% high (probe 6.021 GB against a real DPO step's 5.304 GB) — the same safe direction it shows for SFT — but one shape is not a validation, so the restriction stands until it is measured across shapes. || `training.stream_nf4_scale_cache` | `bool` | `false` | | Keep each streamed NF4 weight's expanded fp32 block scales on the compute device after its first use. A double-quantised weight stores its scales as 8-bit codes plus a nested scale and offset, and bitsandbytes' dequantise + linear path expands them on every visit (forward and checkpoint recompute): two extra kernels and a temporary per projection. The base is frozen, so the expansion is done once per (layer, weight) and reused; results are bit-identical. Costs 4 bytes per 64-element block of VRAM for the whole run (416 MiB on Mistral-7B), which the pre-flight charges up front. bitsandbytes' fused 4-bit GEMM path expands the scales inside its kernel and is left unchanged. Needs quantization='4bit' with double quantization (the default). Off by default. |+| `training.stream_recompute` | `str` | `"on"` | one of `"on"`, `"off"`, `"auto"` | Whether a streamed decoder layer runs its forward a second time in the backward. 'on' (the default) is the shipped checkpoint recompute: activations are dropped after the forward and recomputed. 'off': every streamed layer keeps its activations instead (the backward still re-reads each layer's weights): a faster step where the step is compute-bound (the RAM tier), at the cost of every layer's activations in VRAM, which the formula pre-flight does not count (stream_vram_probe measures it). 'auto': before training, one synthetic forward+backward at the configured batch_size x data.max_length measures one layer's retained bytes and the step's reserved peak; the LAST K layers keep their activations and the rest recompute, K = (the smallest of 85% of the card, free + reserved VRAM and stream_vram_override) minus that peak, divided by the layer's bytes times the step's reserved/allocated ratio. Costs one step at setup; RAM tier and CUDA only, otherwise the recompute is kept. Gradients are the recompute's: measured bit-identical under deterministic algorithms. task='sft' only for 'off' and 'auto', and refused beside activation_offloading. YAML reads an unquoted on/off (and true/false, yes/no) as a boolean; true is taken as 'on' and false as 'off'. || `training.stream_expert_layout` | `str` | `"fused"` | one of `"fused"`, `"per_expert"` | How a MoE base's routed experts are streamed. 'fused' (the default) keeps each MoE layer's experts inside its layer file, so the largest layer bounds the card. 'per_expert' writes them to their own file, one pair of matrices per expert (NF4 per matrix under quantization='4bit'), and reads an expert only when the router picks it, again in the backward: the decoder-layer buffers hold the dense part alone, so the VRAM bound no longer grows with the expert count. Each routed expert is read about three times per step from the cache volume, so expect slower steps (an NVMe volume is recommended). Under quantization='4bit' the routed experts are NF4 here but stay at the stream dtype with 'fused', so the two layouts train against numerically different bases. Admitted for DeepSeek-V3-family bases (model_type deepseek_v3); adapters on the routed experts (moe_lora) are refused with it. || `training.stream_loss_chunk` | `int \| null` | `null` | `>= 16`, `<= 8192` | Tokens per chunk for the output head and the loss under layer streaming (#1693). Unset, the head builds the full \[batch, seq, vocab] logits and the loss works on all of them, which is what caps the context length on a small card. Set, each chunk runs the head, takes its cross-entropy and the gradient on its hidden states, and drops its logits, so the logits term is vocab x chunk x batch whatever data.max_length is. It costs one more \[batch, seq, hidden] tensor and a loop over the head. It gives up outputs.logits on a labelled forward, and equality to the last bit with an unchunked step: the per-token terms are the same, the order of the final sum is not. task='sft' only; refused beside use_cut_ce. || `training.stream_disk_kind` | `str \| null` | `null` | one of `"nvme"`, `"ssd"`, `"hdd"` | Override the auto-detected disk media type for the streaming disk overflow tier. Detection classifies a device by a measured sequential read when the kernel's rotational flag is unreliable — virtio and other paravirtual disks report spinning with no media hint, so a fast cloud disk is otherwise refused (#365). Set this for the case where even that is wrong: 'nvme' forces the disk tier on, 'ssd'/'hdd' force it off. The resolved value is printed beside what was detected. |diff --git a/docs/performance-and-quantization.md b/docs/performance-and-quantization.mdindex 5dab0a39..39d4b3c1 100644--- a/docs/performance-and-quantization.md+++ b/docs/performance-and-quantization.md@@ -450,6 +450,7 @@ training:# stream_vram_override: 4_000_000_000 # Bytes to assume free (v0.73.x); see below# stream_vram_probe: true # Decide the fit by MEASURING one step (sft only); see below# stream_nf4_scale_cache: true # Keep expanded NF4 scales resident (4bit + double quant); see below+ # stream_recompute: auto # Keep activations instead of the checkpoint recompute: on/off/auto; see below# stream_loss_chunk: 128 # Output head and loss N tokens at a time, for long contexts (sft only)# stream_disk_kind: nvme # Override auto disk-kind detection: nvme/ssd/hdd; see below```@@ -580,6 +581,29 @@ cache stays empty. Needs `quantization: 4bit` with double quantization (thedefault). It trades VRAM for time, and the time saved has not been measuredacross machines yet, so it is off by default.+**`stream_recompute` (`on` by default).** A streamed layer normally runs its+forward twice: once in the forward and again as the checkpoint recompute in the+backward. `off` keeps every layer's activations instead. The backward still+re-reads each layer's weights into the buffer pool before it uses them, so pool+slots are never read stale (#331), and the gradients are the recompute's+(bit-identical under deterministic algorithms, measured on Mistral-7B NF4 and in+the CPU tests). On the RAM tier it measured 22% / 17% / 8.5% off the step at 512 /+1024 / 2048 tokens per step on an 8 GB card, as far as the activations fit; it+costs every layer's activations in VRAM, which the formula pre-flight does not+count (`stream_vram_probe: true` measures it, and runs after the switch). `auto`+decides at setup, before step 1: one synthetic forward+backward at the configured+`batch_size` x `data.max_length` (the worst case any batch can pad up to) measures+what one layer keeps allocated and the step's reserved peak. The last K layers then+keep their activations and the rest recompute, with K = (the smallest of 85% of+the card, free + reserved VRAM and `stream_vram_override`, minus that peak) divided+by the layer's bytes times the step's reserved/allocated ratio. It prints K and the+figures it used, and keeps the recompute (saying why) when it cannot measure. On+the disk tier `auto` keeps the recompute and `off` warns: the step is read-bound+there and skipping the recompute does not skip the backward read (no gain measured+on a read-bound 14B, worse at a high VRAM fill). `off` and `auto` are `task: sft`+only and refused beside `activation_offloading`, whose hooks would never see the+activations a retained layer keeps.+**Qwen3.8-Flash-Next / Qwen4-Exp PLE.** The frozen PLE N-gram table is not adecoder-layer weight for storage purposes: putting it in the PLE layer's shardwould make the shared layer buffer as large as the whole table. Soup keeps the@@ -885,7 +909,7 @@ prints this advice when it sees you accumulating.- `unfrozen_parameters`, `lisa_enabled`, `packing`, `multipack`, `use_fsdp2_compile`, `train_router_only`, `expand_layers` → each independently rewrites or re-freezes the same layers- `use_cut_ce` is refused with `stream_layers` (#1206): streamed setup does not install the resident CCE patch. Disable CCE to stream with the conservative full-logits budget, or disable streaming to use resident CCE. A tiny local streamed Llama was exercised through sharding, forward/loss and backward on CPU; this is no CUDA memory measurement.- `use_liger` / `use_flash_attn` → **not reached at all** under streaming (#1692). Both are read by the *resident* setup: Liger patches the MLP forward there, and flash-attn is handed to `from_pretrained` as `attn_implementation`. A streamed model is built by `from_config` with no attention argument and no Liger patch, and `_liger_applied` is never set, so `use_liger_kernel` misses `TrainingArguments` too. These are the switches you reach for when a streamed run will not fit, so they are refused rather than dropped; each is offered again once it is wired and shown to engage on a streamed model. On every task outside `{sft, tts}` they are already refused at config load (#806)-- `stream_source` / `stream_ngram_source` / `stream_buffers` / `stream_read_ahead` / `stream_vram_override` / `stream_vram_probe` / `stream_nf4_scale_cache` / `stream_expert_layout` / `stream_disk_kind` / `stream_pin` / `stream_loss_chunk` set while `stream_layers: false` → a footgun, refused+- `stream_source` / `stream_ngram_source` / `stream_buffers` / `stream_read_ahead` / `stream_vram_override` / `stream_vram_probe` / `stream_nf4_scale_cache` / `stream_recompute` / `stream_expert_layout` / `stream_disk_kind` / `stream_pin` / `stream_loss_chunk` set while `stream_layers: false` → a footgun, refused- `stream_expert_layout: per_expert` with `moe_lora` → refused: a routed expert streamed one at a time has nowhere to keep an adapter. On a base that is not `model_type: deepseek_v3` → refused at setup, before the weights are resolved- an FP8 checkpoint in any format other than DeepSeek-V3's block FP8, or an FP8 base that is not `model_type: deepseek_v3` → refused naming the supported format: at setup when `config.json` gives it away, otherwise before the cache is written. The block format itself is dequantised at shard time (see "FP8 block-quantised bases" above)- any other quantised checkpoint → refused by name, before a byte of the cache is written, and in the same words by the sharder, the disk pre-flight and the setup. The sharder checks the source first, from its `config.json` and headers, so a refusal creates no cache folder, leaves an existing cache and its `index.json` untouched, and never reuses a cache an older Soup sharded from such a source. Layer streaming reads plain floating-point weights, DeepSeek-V3's block FP8 and, on qwen4_exp only, oMLX/oQ affine bundles; the message says which format it found, that layer streaming cannot dequantise it, and to point `training.base` at a bf16 or FP8-block copy. Refused:diff --git a/src/soup_cli/config/schema.py b/src/soup_cli/config/schema.pyindex 44e40f87..d4b74dc2 100644--- a/src/soup_cli/config/schema.py+++ b/src/soup_cli/config/schema.py@@ -3840,6 +3840,49 @@ class TrainingConfig(BaseModel):"double quantization (the default). Off by default."),)+ stream_recompute: Literal["on", "off", "auto"] = Field(+ default="on",+ description=(+ "Whether a streamed decoder layer runs its forward a second time in "+ "the backward. 'on' (the default) is the shipped checkpoint recompute: "+ "activations are dropped after the forward and recomputed. 'off': "+ "every streamed layer keeps its activations instead (the backward "+ "still re-reads each layer's weights): a faster step where the step "+ "is compute-bound (the RAM tier), at the cost of every layer's "+ "activations in VRAM, which the formula pre-flight does not count "+ "(stream_vram_probe measures it). 'auto': before training, one "+ "synthetic forward+backward at the configured batch_size x "+ "data.max_length measures one layer's retained bytes and the step's "+ "reserved peak; the LAST K layers keep their activations and the rest "+ "recompute, K = (the smallest of 85% of the card, free + reserved "+ "VRAM and stream_vram_override) minus that peak, divided by the "+ "layer's bytes times the step's reserved/allocated ratio. Costs one "+ "step at setup; RAM tier and CUDA only, otherwise the recompute is "+ "kept. Gradients are the recompute's: measured bit-identical under "+ "deterministic algorithms. task='sft' only for 'off' and 'auto', and "+ "refused beside activation_offloading. YAML reads an unquoted on/off "+ "(and true/false, yes/no) as a boolean; true is taken as 'on' and "+ "false as 'off'."+ ),+ )++ @field_validator("stream_recompute", mode="before")+ @classmethod+ def _validate_stream_recompute_mode(cls, v: Any) -> Any:+ """YAML 1.1 reads an unquoted ``on``/``off`` (and yes/no/true/false) as a+ bool. Here the bool agrees with the mode (``stream_recompute: off`` = no+ recompute), so it is mapped back; an int, including 0/1, is refused."""+ if v is True:+ return "on"+ if v is False:+ return "off"+ if not isinstance(v, str):+ raise ValueError(+ "training.stream_recompute must be one of 'on', 'off', 'auto'; "+ f"got {type(v).__name__} {v!r}"+ )+ return v+stream_expert_layout: Literal["fused", "per_expert"] = Field(default="fused",description=(@@ -6988,6 +7031,7 @@ class SoupConfig(BaseModel):or tcfg.stream_vram_override is not Noneor tcfg.stream_vram_probeor tcfg.stream_nf4_scale_cache+ or tcfg.stream_recompute != "on"or tcfg.stream_expert_layout != "fused"or tcfg.stream_disk_kind is not Noneor tcfg.stream_pin is not None@@ -6997,7 +7041,8 @@ class SoupConfig(BaseModel):"training.stream_source / training.stream_ngram_source / ""training.stream_buffers / training.stream_read_ahead / ""training.stream_vram_override / training.stream_vram_probe "- "/ training.stream_nf4_scale_cache / training.stream_expert_layout "+ "/ training.stream_nf4_scale_cache / training.stream_recompute "+ "/ training.stream_expert_layout ""/ training.stream_disk_kind / training.stream_pin ""/ training.stream_loss_chunk set but stream_layers is false; set ""stream_layers=true to stream the base layer-by-layer."@@ -7090,6 +7135,31 @@ class SoupConfig(BaseModel):f"per-token log-probs of chosen and rejected instead, and that "f"reduction is not chunked. Remove training.stream_loss_chunk.")+ if tcfg.stream_recompute != "on" and self.task != "sft":+ # Only the SFT step is measured. A loss with two grad-enabled+ # forwards before one backward would unpack a layer's saved pool+ # tensors after its slot was refilled, which the owner check+ # refuses mid-step rather than training on stale weights.+ raise ValueError(+ f"training.stream_recompute={tcfg.stream_recompute!r} supports "+ f"task='sft' only; got task={self.task!r}. Keeping activations "+ f"instead of the checkpoint recompute is measured on the SFT step "+ f"only; a loss that runs more than one grad-enabled forward per "+ f"backward is not supported yet. Remove training.stream_recompute "+ f"or set it to 'on'."+ )+ if tcfg.stream_recompute != "on" and tcfg.activation_offloading is not None:+ # A retained layer saves its activations through its own+ # saved_tensors_hooks, the innermost ones, so the offloading hooks+ # never see them: they would stay on the card, silently.+ raise ValueError(+ f"training.stream_recompute={tcfg.stream_recompute!r} cannot be "+ f"combined with training.activation_offloading="+ f"{tcfg.activation_offloading!r}: a streamed layer that keeps its "+ f"activations saves them through its own hooks, so they would stay "+ f"in VRAM and never be offloaded. Set stream_recompute to 'on' or "+ f"unset activation_offloading."+ )if tcfg.stream_loss_chunk is not None and self.data.format == "raft":raise ValueError("training.stream_loss_chunk cannot be combined with data.format="diff --git a/src/soup_cli/trainer/stream_setup.py b/src/soup_cli/trainer/stream_setup.pyindex ff1c2ac0..d53993df 100644--- a/src/soup_cli/trainer/stream_setup.py+++ b/src/soup_cli/trainer/stream_setup.py@@ -1553,6 +1553,23 @@ class StreamingSetupMixin:# not started yet, so release the runtime here.self._close_stream_runtime()raise+ # training.stream_recompute -- before the probe, so that the probe+ # measures the mode that will train. The default ('on') adds nothing to+ # the setup: no attribute is read and nothing is called, so the shape+ # and task arguments below are only evaluated for 'off' / 'auto'.+ if getattr(tcfg, "stream_recompute", "on") != "on":+ try:+ self._configure_stream_recompute(+ model,+ tcfg,+ tier=tier,+ on_cuda=on_cuda,+ task=cfg.task,+ step_shape=self._stream_step_shape(cfg, tcfg, model_config),+ )+ except Exception:+ self._close_stream_runtime()+ raiseif probe_plan is not None:self._run_stream_vram_probe(model, probe_plan)stats = runtime.stats()@@ -1584,6 +1601,92 @@ class StreamingSetupMixin:return 2 * slot_bytesreturn slot_bytes+ def _configure_stream_recompute(+ self,+ model,+ tcfg,+ *,+ tier: str,+ on_cuda: bool,+ task: str,+ step_shape: tuple = (0, 0, 0),+ ) -> None:+ """``training.stream_recompute``: 'on' leaves the checkpoint recompute;+ 'off' switches every streamed layer; 'auto' (RAM tier, CUDA) measures one+ synthetic step at ``step_shape`` (rows, seq_len, vocab_size) and switches+ the last layers that fit, all before training starts."""+ from soup_cli.utils.layer_stream import STREAM_RECOMPUTE_HEADROOM, TIER_DISK++ mode = getattr(tcfg, "stream_recompute", "on")+ if mode == "on":+ return+ if task != "sft":+ # The config refuses this first; repeated here so that no other+ # caller of the setup can switch a loss with two grad-enabled+ # forwards per backward.+ raise ValueError(+ f"training.stream_recompute={mode!r} supports task='sft' only; "+ f"got task={task!r}."+ )+ if mode == "off":+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ switched = set_stream_recompute(model, False)+ console.print(+ f"[cyan]training.stream_recompute='off':[/] {switched} streamed layers "+ "keep their activations instead of recomputing them in the backward "+ "(faster on the RAM tier, more VRAM; the formula pre-flight above does "+ "not count them, training.stream_vram_probe measures them)"+ )+ if tier == TIER_DISK:+ console.print(+ "[yellow]training.stream_recompute='off' on the disk tier:[/] a "+ "disk-tier step is usually read-bound, and skipping the recompute "+ "does not skip the backward read of each layer, so it is likely no "+ "faster and holds more VRAM; the recompute ('on') may be cheaper here."+ )+ return+ if tier == TIER_DISK:+ console.print(+ "[cyan]training.stream_recompute='auto' keeps the recompute on the disk "+ "tier:[/] a disk-tier step is read-bound, and skipping the recompute does "+ "not skip the backward read of each layer."+ )+ return+ if not on_cuda:+ console.print(+ "[cyan]training.stream_recompute='auto' keeps the recompute:[/] no CUDA "+ "device to measure VRAM on."+ )+ return+ from soup_cli.utils.layer_stream_runtime import plan_stream_recompute_auto++ rows, seq_len, vocab_size = step_shape+ console.print(+ f"[cyan]training.stream_recompute='auto':[/] measuring one step at batch {rows} "+ f"x seq {seq_len} to choose how many of the last layers keep their activations "+ f"within {STREAM_RECOMPUTE_HEADROOM:.0%} of the card."+ )+ plan_stream_recompute_auto(+ model,+ rows=rows,+ seq_len=seq_len,+ vocab_size=vocab_size,+ notify=console.print,+ override_bytes=getattr(tcfg, "stream_vram_override", None),+ )++ def _stream_step_shape(self, cfg, tcfg, model_config) -> tuple:+ """``(rows, seq_len, vocab_size)`` of the worst-case step: the shape+ :meth:`_stream_budget_lines` budgets and the VRAM probe measures."""+ shape_cfg = self._stream_shape_config(model_config)+ batch = tcfg.batch_size if isinstance(tcfg.batch_size, int) else 1+ return (+ batch * self._STREAM_ROWS_PER_EXAMPLE,+ int(cfg.data.max_length),+ int(getattr(shape_cfg, "vocab_size", 0) or 0),+ )+def _attach_streamed_save_guard(self) -> None:"""Check every ``checkpoint-*`` adapter a streamed run writes (#1011)."""if getattr(self, "_stream_runtime", None) is None:diff --git a/src/soup_cli/utils/layer_stream.py b/src/soup_cli/utils/layer_stream.pyindex e4ab77c3..54c57bc3 100644--- a/src/soup_cli/utils/layer_stream.py+++ b/src/soup_cli/utils/layer_stream.py@@ -1228,6 +1228,85 @@ def nf4_scale_cache_bytes(quant_specs: Mapping[str, Any], n_layers: int) -> int:return per_layer * n_layers+#: training.stream_recompute='auto' — the share of the card's total memory the+#: retained activations may grow the step into.+STREAM_RECOMPUTE_HEADROOM = 0.85++#: The three caps of :func:`stream_recompute_budget`, by the name it reports.+RECOMPUTE_CAP_CARD = "card"+RECOMPUTE_CAP_FREE = "free"+RECOMPUTE_CAP_OVERRIDE = "override"+++@dataclass(frozen=True)+class RecomputeBudget:+ """The VRAM the retained activations of ``stream_recompute='auto'`` may use."""++ #: The smallest of the caps, in bytes.+ cap_bytes: int+ #: Which cap that was: RECOMPUTE_CAP_CARD / _FREE / _OVERRIDE.+ bound: str+ #: ``cap_bytes`` minus the measured reserved peak; <= 0 means none.+ budget_bytes: int+++def stream_recompute_budget(+ *,+ total_bytes: int,+ free_bytes: int,+ reserved_bytes: int,+ reserved_peak_bytes: int,+ override_bytes: Optional[int] = None,+ headroom: float = STREAM_RECOMPUTE_HEADROOM,+) -> RecomputeBudget:+ """``min(headroom x card, free + reserved now, stream_vram_override) - reserved peak``.++ The card term keeps headroom below the physical size; the free term is what+ this process can still reach on a card other processes share+ (``mem_get_info`` free plus what the allocator already holds); the override,+ when set, is the user's own cap for what the driver cannot report. The peak+ is the RESERVED one, not the allocated one: the caching allocator's+ fragmentation is real VRAM.+ """+ if (+ isinstance(headroom, bool)+ or not isinstance(headroom, (int, float))+ or not 0.0 < headroom <= 1.0+ ):+ raise ValueError(f"headroom must be a float in (0, 1]; got {headroom!r}")+ caps = {+ RECOMPUTE_CAP_CARD: int(headroom * total_bytes),+ RECOMPUTE_CAP_FREE: int(free_bytes) + int(reserved_bytes),+ }+ if override_bytes is not None:+ caps[RECOMPUTE_CAP_OVERRIDE] = int(override_bytes)+ bound = min(caps, key=lambda name: caps[name])+ cap = caps[bound]+ return RecomputeBudget(cap_bytes=cap, bound=bound, budget_bytes=cap - int(reserved_peak_bytes))+++def retained_layer_charge(+ allocated_bytes: int, peak_reserved_bytes: int, peak_allocated_bytes: int+) -> int:+ """One retained layer's charge: its allocated bytes times the step's+ reserved/allocated ratio when that is above 1 (the allocator's rounding and+ fragmentation grow with what it holds), rounded up."""+ allocated_bytes = int(allocated_bytes)+ if peak_allocated_bytes <= 0 or peak_reserved_bytes <= peak_allocated_bytes:+ return allocated_bytes+ return -(-allocated_bytes * int(peak_reserved_bytes) // int(peak_allocated_bytes))+++def plan_retained_layers(budget_bytes: int, per_layer_bytes: int, n_layers: int) -> int:+ """How many of the last streamed layers may skip the checkpoint recompute:+ ``budget_bytes // per_layer_bytes`` clamped to ``[0, n_layers]``. A zero or+ negative budget, or a layer charged at zero bytes or less, plans nothing,+ which is the checkpoint recompute everywhere."""+ if budget_bytes <= 0 or per_layer_bytes <= 0:+ return 0+ return max(0, min(int(n_layers), int(budget_bytes) // int(per_layer_bytes)))++def estimate_loss_chunk_bytes(*, hidden_size: int, seq_len: int, batch_size: int = 1, dtype: str = "bfloat16") -> int:diff --git a/src/soup_cli/utils/layer_stream_runtime.py b/src/soup_cli/utils/layer_stream_runtime.pyindex cec24ec5..11ae8dec 100644--- a/src/soup_cli/utils/layer_stream_runtime.py+++ b/src/soup_cli/utils/layer_stream_runtime.py@@ -364,6 +364,118 @@ def _checkpoint_visible_nf4_function():return CheckpointVisibleNF4Matmul+_PACKED_DEQUANT_NF4_FUNCTION = None+++def _packed_dequant_nf4_function():+ """Lazy autograd Function: dense-dequant forward, PACKED-only saved state.++ For a streamed layer that runs WITHOUT checkpoint recompute. ``F.linear`` on+ a dequantised weight would save the dense bf16 weight for the backward (437 MB+ per Mistral-7B layer); this saves only the packed bytes and the quantisation+ tensors through ``save_for_backward`` and dequantises again for ``grad_x``.+ The forward arithmetic is the compatibility path's (dequantise + ``F.linear``)+ and the backward's ``grad_y @ W`` is the same GEMM ``F.linear``'s backward+ issues, so outputs and gradients stay bit-identical to the recompute path.+ """+ global _PACKED_DEQUANT_NF4_FUNCTION+ if _PACKED_DEQUANT_NF4_FUNCTION is not None:+ return _PACKED_DEQUANT_NF4_FUNCTION++ import bitsandbytes.functional as bnb_functional+ import torch+ import torch.nn.functional as functional++ def _state(ctx, absmax, state2_absmax, state2_code, offset):+ state2 = None+ q_offset = None+ if ctx.nested:+ state2 = bnb_functional.QuantState(+ absmax=state2_absmax,+ code=state2_code,+ blocksize=ctx.state2_blocksize,+ dtype=torch.float32,+ )+ q_offset = offset+ return bnb_functional.QuantState(+ absmax=absmax,+ shape=torch.Size(ctx.shape),+ code=ctx.code,+ dtype=ctx.weight_dtype,+ blocksize=ctx.blocksize,+ quant_type=ctx.quant_type,+ offset=q_offset,+ state2=state2,+ )++ class PackedDequantNF4Linear(torch.autograd.Function):+ @staticmethod+ def forward(ctx, x, packed, absmax, state2_absmax, state2_code, offset, bias, meta):+ shape, weight_dtype, blocksize, quant_type, nested, state2_blocksize, code = meta+ ctx.shape = tuple(int(dim) for dim in shape)+ ctx.weight_dtype = weight_dtype+ ctx.blocksize = int(blocksize)+ ctx.quant_type = str(quant_type)+ ctx.nested = bool(nested)+ ctx.state2_blocksize = int(state2_blocksize)+ ctx.code = code+ ctx.bias_dtype = None if bias is None else bias.dtype+ ctx.owner_check = getattr(packed, "_soup_stream_owner_check", None)+ weight = bnb_functional.dequantize_4bit(+ packed, _state(ctx, absmax, state2_absmax, state2_code, offset)+ ).to(x.dtype)+ ctx.save_for_backward(packed, absmax, state2_absmax, state2_code, offset)+ return functional.linear(x, weight, bias)++ @staticmethod+ def backward(ctx, grad_output):+ packed, absmax, state2_absmax, state2_code, offset = ctx.saved_tensors+ if ctx.owner_check is not None:+ ctx.owner_check()+ grad_x = None+ if ctx.needs_input_grad[0]:+ weight = bnb_functional.dequantize_4bit(+ packed, _state(ctx, absmax, state2_absmax, state2_code, offset)+ ).to(grad_output.dtype)+ grad_x = torch.matmul(grad_output, weight)+ grad_bias = None+ if ctx.needs_input_grad[6] and ctx.bias_dtype is not None:+ grad_bias = grad_output.reshape(-1, grad_output.shape[-1]).sum(+ dim=0, dtype=ctx.bias_dtype+ )+ return grad_x, None, None, None, None, None, grad_bias, None++ _PACKED_DEQUANT_NF4_FUNCTION = PackedDequantNF4Linear+ return PackedDequantNF4Linear+++def packed_dequant_nf4_linear(x: Any, packed: Any, quant_state: Any, bias: Any = None) -> Any:+ """Dense-dequant NF4 linear that saves only packed state for its backward+ (training.stream_recompute 'off' / 'auto')."""+ empty = packed.new_empty((0,))+ state2_absmax = empty+ state2_code = empty+ offset = empty+ state2_blocksize = 0+ if quant_state.nested:+ state2_absmax = quant_state.state2.absmax+ state2_code = quant_state.state2.code+ offset = quant_state.offset+ state2_blocksize = int(quant_state.state2.blocksize)+ meta = (+ tuple(quant_state.shape),+ quant_state.dtype,+ int(quant_state.blocksize),+ str(quant_state.quant_type),+ bool(quant_state.nested),+ state2_blocksize,+ quant_state.code,+ )+ return _packed_dequant_nf4_function().apply(+ x, packed, quant_state.absmax, state2_absmax, state2_code, offset, bias, meta+ )++def checkpoint_visible_nf4_linear(x: Any, packed: Any, quant_state: Any, bias: Any = None) -> Any:@@ -474,6 +586,7 @@ def install_dequant_forward(module: Any) -> int:import typesimport bitsandbytes as bnb+ import torchimport torch.nn.functional as functionalfrom bitsandbytes.functional import dequantize_4bit@@ -508,6 +621,11 @@ def install_dequant_forward(module: Any) -> int:scale_cache = getattr(self.weight, "_soup_scale_cache", None)if scale_cache is not None:quant_state = scale_cache.dense_state(self.weight._soup_scale_cache_key, quant_state)+ if getattr(self, "_soup_packed_backward", False) and torch.is_grad_enabled():+ # training.stream_recompute — no checkpoint recompute around this+ # layer: never let F.linear save the dense weight; save the packed+ # bytes instead.+ return packed_dequant_nf4_linear(x, self.weight, quant_state, bias).to(inp_dtype)weight = dequantize_4bit(self.weight, quant_state).to(x.dtype)return functional.linear(x, weight, bias).to(inp_dtype)@@ -1173,6 +1291,9 @@ class LayerBufferPool:]self.events = [torch.cuda.Event() for _ in range(self.n)] if self.is_cuda else []self.owner: list = [None] * self.n+ # Bumped on every load into a slot, so a reader can tell a slot that was+ # refilled (even with the same layer) from one that was not.+ self.generation: list = [0] * self.nif active_keys_by_layer is None:shared = tuple(layer_spec)self.active_keys_by_layer = [shared]@@ -1215,6 +1336,7 @@ class LayerBufferPool:# into a small reusable pool refills sooner for it._release_source(source, idx, None)self.owner[slot] = idx+ self.generation[slot] += 1self.loads += 1return slot@@ -1491,6 +1613,21 @@ def _build_streamed_layer_class():self.prefetcher = prefetcherself.name_map = dict(name_map or {})self.use_checkpoint = bool(use_checkpoint)+ # training.stream_recompute — False: no checkpoint recompute.+ # Activations stay saved; pool-slot tensors are saved as tokens and+ # the layer is re-visited (wait + prefetch) at its first backward+ # unpack instead of at the recompute. See _body_retained /+ # set_stream_recompute.+ self.recompute = True+ self._bwd_visited = True+ # The pool slot's load generation at the first backward visit; a+ # later unpack refuses a slot that was reloaded since, even with the+ # same owner.+ self._bwd_generation = None+ self._slot_ptrs: Dict[int, frozenset] = {}+ # training.stream_recompute='auto' — a RetainedLayerProbe armed on+ # this layer measures its next no-recompute forward, then disarms.+ self._retained_probe = Noneself.quant_specs = dict(quant_specs or {})self.codes = dict(codes or {})# training.stream_nf4_scale_cache — an NF4ScaleCache owned by the@@ -1726,9 +1863,77 @@ def _build_streamed_layer_class():def forward(self, hidden_states: Any, *args: Any, **kwargs: Any) -> Any:if self.use_checkpoint and torch.is_grad_enabled():+ if not self.recompute:+ return self._body_retained(hidden_states, *args, **kwargs)return checkpoint(self._body, hidden_states, *args, use_reentrant=False, **kwargs)return self._body(hidden_states, *args, **kwargs)+ def _pool_ptrs(self) -> frozenset:+ slot = self.pool.slot_for(self.idx)+ ptrs = self._slot_ptrs.get(slot)+ if ptrs is None:+ ptrs = frozenset(+ buf.untyped_storage().data_ptr() for buf in self.pool.buffers[slot].values()+ )+ self._slot_ptrs[slot] = ptrs+ return ptrs++ def _backward_visit(self) -> None:+ # The first unpack of this layer's pool tensors in the backward is+ # where the recompute used to run: same wait + prefetch, same order.+ slot = self.pool.slot_for(self.idx)+ if not self._bwd_visited:+ self._bwd_visited = True+ self.pool.wait(self.idx)+ # Recorded after the wait (the slot holds this layer now) and+ # before the prefetch, which loads a DIFFERENT slot.+ self._bwd_generation = self.pool.generation[slot]+ self.prefetcher.advance(self.idx)+ return+ if self.pool.owner[slot] != self.idx:+ raise RuntimeError(+ f"streamed backward (no recompute) found slot {slot} holding layer "+ f"{self.pool.owner[slot]} while layer {self.idx} still unpacks from it"+ )+ if self.pool.generation[slot] != self._bwd_generation:+ raise RuntimeError(+ f"streamed backward (no recompute) found slot {slot} reloaded "+ f"(load generation {self._bwd_generation} -> "+ f"{self.pool.generation[slot]}) while layer {self.idx} still unpacks "+ "from it"+ )++ def _body_retained(self, hidden_states: Any, *args: Any, **kwargs: Any) -> Any:+ ptrs = self._pool_ptrs()+ token = _POOL_TOKEN++ def pack(tensor: Any) -> Any:+ try:+ ptr = tensor.untyped_storage().data_ptr()+ except (AttributeError, RuntimeError):+ return tensor+ if ptr in ptrs:+ return (token, tensor)+ return tensor++ def unpack(obj: Any) -> Any:+ if type(obj) is tuple and len(obj) == 2 and obj[0] is token:+ self._backward_visit()+ return obj[1]+ return obj++ self._bwd_visited = False+ probe = self._retained_probe+ if probe is not None:+ # One forward only: the probe restores the recompute after it.+ self._retained_probe = None+ probe.start(self)+ with torch.autograd.graph.saved_tensors_hooks(pack, unpack):+ out = self._body(hidden_states, *args, **kwargs)+ if probe is not None:+ probe.finish(self, out)+ return out+def _substituted_weights(self, buffers: Any) -> Any:# Weights arrive with requires_grad=False, so autograd allocates no# grad buffers for them — but W STAYS IN THE GRAPH for W^T . dL/dy,@@ -1787,6 +1992,385 @@ def _build_streamed_layer_class():_STREAMED_LAYER_CLASS = None+_POOL_TOKEN = object()+++def set_stream_recompute(model: Any, enabled: bool, *, retain: Optional[int] = None) -> int:+ """training.stream_recompute — switch every streamed decoder layer between the+ checkpoint recompute (True, the default) and retained activations (False).+ Returns the number of streamed layers found.++ ``retain`` (only with ``enabled=False``): how many layers skip the recompute+ (the LAST ``retain`` layers); the rest keep the checkpoint recompute.+ ``None`` = all layers. Peak VRAM grows by one layer's activations per+ retained layer, whichever layers they are. A retained layer's NF4 linears+ save only the packed bytes for the backward (``packed_dequant_nf4_linear``)."""+ if retain is not None and (+ isinstance(retain, bool) or not isinstance(retain, int) or retain < 0+ ):+ raise ValueError(f"retain must be None or an int >= 0; got {retain!r}")+ streamed = streamed_decoder_layers(model)+ order = [module.idx for module in streamed]+ kept = set(order if retain is None else (order[-retain:] if retain > 0 else []))+ for module in streamed:+ skip = not enabled and module.idx in kept+ _set_layer_recompute(module, skip=skip)+ module._bwd_visited = True+ return len(streamed)+++def streamed_decoder_layers(model: Any) -> list:+ """Every ``StreamedDecoderLayer`` under ``model``, sorted by layer index.++ The wrapper is transparent to ``named_modules()`` (it yields only its inner+ layer's modules), so it is reached through its parent's ``_modules``.+ """+ cls = _streamed_layer_class()+ streamed = []+ seen = set()+ for parent in model.modules():+ for child in parent._modules.values():+ if isinstance(child, cls) and id(child) not in seen:+ seen.add(id(child))+ streamed.append(child)+ return sorted(streamed, key=lambda module: module.idx)+++def _linear4bit_class() -> Any:+ """``bitsandbytes.nn.Linear4bit``. Only reached for a layer whose NF4+ linears were patched, which already required bitsandbytes."""+ import bitsandbytes as bnb++ return bnb.nn.Linear4bit+++def _set_layer_recompute(module: Any, *, skip: bool) -> None:+ """One streamed layer: ``skip`` = no checkpoint recompute. Its NF4 linears+ then save the packed bytes, not the dense weight, for the backward. A layer+ with no patched NF4 linear (``n_dequant_forward == 0``) has none to flag, and+ bitsandbytes is not imported for it."""+ module.recompute = not skip+ if not getattr(module, "n_dequant_forward", 0):+ return+ linear4bit = _linear4bit_class()+ for child in module.inner.modules():+ if isinstance(child, linear4bit):+ child._soup_packed_backward = skip+++def _output_nbytes(output: Any) -> int:+ """Bytes of every tensor in a layer's output (a tensor, or nested containers)."""+ if hasattr(output, "untyped_storage") and hasattr(output, "element_size"):+ return int(output.numel() * output.element_size())+ if isinstance(output, Mapping):+ return sum(_output_nbytes(value) for value in output.values())+ if isinstance(output, (tuple, list)):+ return sum(_output_nbytes(value) for value in output)+ return 0+++class RetainedLayerProbe:+ """training.stream_recompute='auto' — what ONE streamed layer keeps allocated+ when it skips the checkpoint recompute.++ ``arm(model)`` switches the LAST streamed layer to no-recompute for its next+ grad-enabled forward only, and that forward reads the allocator before and+ after the layer's body: ``after - before - output bytes`` is what the layer+ holds for its backward. The input was allocated before and is saved by the+ recompute path too, so it is not counted; pool-slot tensors are saved as+ tokens and cost nothing. Bytes the NF4 scale cache fills during that forward+ are persistent, not retained, and are subtracted. Allocated memory is read,+ never a difference of peaks: peaks are not additive, and a peak difference+ under-estimated a Mistral-7B layer 3x at 1024 tokens (benchmarks/exp-1c-+ streaming-without-recompute.md, section 6). The layer's mode is restored+ right after the measured forward; its backward still runs the no-recompute+ path, bit-identical to the recompute.++ ``allocated`` reads the allocator; None = ``torch.cuda.memory_allocated`` on+ the layer's device, which needs a CUDA pool. Every failure is recorded in+ ``error`` instead of being raised into the training step.+ """++ def __init__(self, allocated: Optional[Callable[[], int]] = None):+ self._allocated = allocated+ self.per_layer_bytes: Optional[int] = None+ self.layer_idx: Optional[int] = None+ self.error: Optional[str] = None+ self._before: Optional[int] = None+ self._cache_before = 0++ def arm(self, model: Any) -> Optional[int]:+ """Arm the last streamed layer; returns its index, or None (``error`` says why)."""+ layers = streamed_decoder_layers(model)+ if not layers:+ self.error = "the model has no streamed decoder layers"+ return None+ last = layers[-1]+ if self._allocated is None:+ pool = getattr(last, "pool", None)+ if not getattr(pool, "is_cuda", False):+ self.error = "no CUDA device to measure the retained bytes on"+ return None+ import torch++ device = pool.device+ self._allocated = lambda: int(torch.cuda.memory_allocated(device))+ _set_layer_recompute(last, skip=True)+ last._retained_probe = self+ self.error = None+ return last.idx++ def disarm(self, model: Any) -> None:+ """Drop an armed probe whose forward never ran, restoring that layer."""+ for layer in streamed_decoder_layers(model):+ if layer._retained_probe is self:+ layer._retained_probe = None+ _set_layer_recompute(layer, skip=False)++ @staticmethod+ def _cache_bytes(layer: Any) -> int:+ return int(getattr(getattr(layer, "scale_cache", None), "nbytes", 0) or 0)++ def start(self, layer: Any) -> None:+ try:+ self._cache_before = self._cache_bytes(layer)+ self._before = int(self._allocated())+ except Exception as exc: # noqa: BLE001 — a measurement must not stop training+ self._before = None+ self.error = f"the measurement raised {type(exc).__name__}: {exc}"++ def finish(self, layer: Any, output: Any) -> None:+ try:+ if self._before is not None:+ after = int(self._allocated())+ filled = self._cache_bytes(layer) - self._cache_before+ self.per_layer_bytes = after - self._before - filled - _output_nbytes(output)+ self.layer_idx = layer.idx+ except Exception as exc: # noqa: BLE001 — a measurement must not stop training+ self.per_layer_bytes = None+ self.error = f"the measurement raised {type(exc).__name__}: {exc}"+ finally:+ # The flags are read in the forward, so restoring them now leaves this+ # step's backward on the no-recompute path it was recorded for.+ _set_layer_recompute(layer, skip=False)+++def _fmt_bytes(count: int) -> str:+ if abs(count) >= 10**9:+ return f"{count / 1e9:.2f} GB"+ if abs(count) >= 10**6:+ return f"{count / 1e6:.1f} MB"+ return f"{count} bytes"+++class CudaMemoryReader:+ """The allocator figures ``stream_recompute='auto'`` reads, on one CUDA device."""++ def __init__(self, device: Any):+ self.device = device++ def reset_peak(self) -> None: # pragma: no cover - CUDA only+ import torch++ torch.cuda.synchronize(self.device)+ torch.cuda.reset_peak_memory_stats(self.device)++ def synchronize(self) -> None: # pragma: no cover - CUDA only+ import torch++ torch.cuda.synchronize(self.device)++ def allocated(self) -> int: # pragma: no cover - CUDA only+ import torch++ return int(torch.cuda.memory_allocated(self.device))++ def peak_allocated(self) -> int: # pragma: no cover - CUDA only+ import torch++ return int(torch.cuda.max_memory_allocated(self.device))++ def peak_reserved(self) -> int: # pragma: no cover - CUDA only+ import torch++ return int(torch.cuda.max_memory_reserved(self.device))++ def reserved(self) -> int: # pragma: no cover - CUDA only+ import torch++ return int(torch.cuda.memory_reserved(self.device))++ def free(self) -> int: # pragma: no cover - CUDA only+ import torch++ return int(torch.cuda.mem_get_info(self.device)[0])++ def total(self) -> int: # pragma: no cover - CUDA only+ import torch++ return int(torch.cuda.get_device_properties(self.device).total_memory)++ def release(self) -> None: # pragma: no cover - CUDA only+ import torch++ torch.cuda.empty_cache()+++def _synthetic_step(model: Any, *, rows: int, seq_len: int, vocab_size: int, device: Any) -> None:+ """One forward+backward on random ids at the configured shape, gradients dropped.++ Synthetic for the reason :func:`measure_step_peak_bytes` gives: the CONFIGURED+ shape is the worst case any real batch can pad up to, a real batch measures+ whatever length it happened to have.+ """+ import torch++ ids = torch.randint(0, vocab_size, (rows, seq_len), device=device)+ out = None+ try:+ out = model(input_ids=ids, labels=ids)+ out.loss.backward()+ finally:+ del ids, out+ _zero_probe_grads(model)+++def plan_stream_recompute_auto(+ model: Any,+ *,+ rows: int,+ seq_len: int,+ vocab_size: int,+ notify: Callable[[str], Any],+ override_bytes: Optional[int] = None,+ headroom: Optional[float] = None,+ memory: Any = None,+ step: Optional[Callable[..., None]] = None,+) -> int:+ """training.stream_recompute='auto' — decide, before training, how many of the+ LAST streamed layers keep their activations, and switch them. Returns ``K``.++ One synthetic forward+backward at the configured shape (``rows`` x+ ``seq_len``, the shape the VRAM probe measures) runs with the checkpoint+ recompute on every layer but the last, which a :class:`RetainedLayerProbe`+ measures: its allocated bytes, charged at the step's reserved/allocated ratio+ when that is above 1. The step's RESERVED peak (which therefore already holds+ the measured layer, so ``K`` errs low by up to one) is taken from+ :func:`~soup_cli.utils.layer_stream.stream_recompute_budget`'s cap, and ``K =+ budget // charge`` clamped to the layer count. Anything that cannot be+ measured keeps the checkpoint recompute everywhere and says why; that+ includes an out-of-memory in the synthetic step. Any other exception of the+ step propagates, as the VRAM probe treats one (:func:`_classify_probe_exception`).++ ``memory`` reads the allocator (:class:`CudaMemoryReader` on the streamed+ layers' device when None, which needs a CUDA pool); ``step`` runs the+ synthetic step (:func:`_synthetic_step` when None).+ """+ from soup_cli.utils.layer_stream import (+ RECOMPUTE_CAP_CARD,+ RECOMPUTE_CAP_FREE,+ STREAM_RECOMPUTE_HEADROOM,+ plan_retained_layers,+ retained_layer_charge,+ stream_recompute_budget,+ )+ from soup_cli.utils.terminal import for_terminal++ headroom = STREAM_RECOMPUTE_HEADROOM if headroom is None else headroom+ layers = streamed_decoder_layers(model)+ n_layers = len(layers)++ def keep(why: str) -> int:+ set_stream_recompute(model, True)+ notify(+ f"[yellow]training.stream_recompute='auto' keeps the recompute on all "+ f"{n_layers} streamed layers:[/] {for_terminal(why)}"+ )+ return 0++ if not layers:+ return keep("the model has no streamed decoder layers")+ if min(rows, seq_len, vocab_size) < 1:+ return keep(+ f"no step shape to measure (batch {rows} x seq {seq_len}, vocab_size {vocab_size})"+ )+ device = layers[-1].pool.device+ if memory is None:+ if not getattr(layers[-1].pool, "is_cuda", False):+ return keep("no CUDA device to measure VRAM on")+ memory = CudaMemoryReader(device)+ run = _synthetic_step if step is None else step+ shape = f"batch {rows} x seq {seq_len}"+ probe = RetainedLayerProbe(allocated=memory.allocated)+ probe.arm(model)+ try:+ memory.reset_peak()+ run(model, rows=rows, seq_len=seq_len, vocab_size=vocab_size, device=device)+ memory.synchronize()+ peak_reserved = memory.peak_reserved()+ peak_allocated = memory.peak_allocated()+ free = memory.free()+ reserved = memory.reserved()+ total = memory.total()+ except Exception as exc:+ # An out-of-memory is a result, as it is for the VRAM probe: this shape+ # does not fit even with one layer retained. Anything else (an illegal+ # access, a device assert, a bug) may leave the context unusable, so it+ # propagates and the setup closes the runtime and refuses the run.+ if not _is_out_of_memory(exc):+ raise+ return keep(+ f"the synthetic step at {shape} ran out of memory, so the configured shape "+ "does not fit with one layer retained; set training.stream_vram_probe: true "+ "to measure whether it fits at all"+ )+ finally:+ probe.disarm(model)+ try:+ memory.release()+ except Exception: # noqa: BLE001 — a cache flush must not mask the result+ logger.warning("stream_recompute='auto': releasing the allocator cache failed")+ per_layer = probe.per_layer_bytes+ if per_layer is None:+ return keep(probe.error or "the measured forward did not run")+ if per_layer <= 0:+ return keep(f"one layer measured at {_fmt_bytes(per_layer)} retained (<= 0)")+ charge = retained_layer_charge(per_layer, peak_reserved, peak_allocated)+ budget = stream_recompute_budget(+ total_bytes=total,+ free_bytes=free,+ reserved_bytes=reserved,+ reserved_peak_bytes=peak_reserved,+ override_bytes=override_bytes,+ headroom=headroom,+ )+ if budget.bound == RECOMPUTE_CAP_CARD:+ cap = f"{headroom:.0%} of the card's {_fmt_bytes(total)}"+ elif budget.bound == RECOMPUTE_CAP_FREE:+ cap = f"free + reserved VRAM {_fmt_bytes(free + reserved)}"+ else:+ cap = f"training.stream_vram_override {_fmt_bytes(budget.cap_bytes)}"+ if budget.budget_bytes <= 0:+ return keep(+ f"no VRAM headroom: {cap} minus the reserved peak of the {shape} step, "+ f"{_fmt_bytes(peak_reserved)}, is {_fmt_bytes(budget.budget_bytes)}"+ )+ k = plan_retained_layers(budget.budget_bytes, charge, n_layers)+ if k == 0:+ return keep(+ f"one layer ({_fmt_bytes(charge)} charged) does not fit the budget of "+ f"{_fmt_bytes(budget.budget_bytes)}"+ )+ set_stream_recompute(model, False, retain=k)+ ratio = peak_reserved / peak_allocated if peak_allocated > 0 else 1.0+ notify(+ f"[cyan]training.stream_recompute='auto':[/] the last {k} of {n_layers} streamed "+ f"layers keep their activations, the rest recompute (measured at {shape}: one "+ f"layer {_fmt_bytes(per_layer)} retained, charged {_fmt_bytes(charge)} at "+ f"reserved/allocated {ratio:.2f}; budget {_fmt_bytes(budget.budget_bytes)} = "+ f"{cap} minus the reserved peak {_fmt_bytes(peak_reserved)})"+ )+ return kdef _streamed_layer_class():diff --git a/tests/fixtures/recipe_config_snapshots.json b/tests/fixtures/recipe_config_snapshots.jsonindex 3c8de65e..4bf9571d 100644--- a/tests/fixtures/recipe_config_snapshots.json+++ b/tests/fixtures/recipe_config_snapshots.json@@ -274,11 +274,12 @@"training.stream_disk_kind": null,"training.stream_expert_layout": "fused","training.stream_layers": false,- "training.stream_nf4_scale_cache": false,"training.stream_loss_chunk": null,+ "training.stream_nf4_scale_cache": false,"training.stream_ngram_source": "auto","training.stream_pin": null,"training.stream_read_ahead": 2,+ "training.stream_recompute": "on","training.stream_source": "auto","training.stream_vram_override": null,"training.stream_vram_probe": false,diff --git a/tests/test_stream_recompute.py b/tests/test_stream_recompute.pynew file mode 100644index 00000000..a330cd25--- /dev/null+++ b/tests/test_stream_recompute.py@@ -0,0 +1,1393 @@+"""Layer streaming without the checkpoint recompute (``training.stream_recompute``).++A streamed decoder layer normally runs its forward twice: once in the forward+and again as the checkpoint recompute in the backward, which is also where the+layer's weights are re-read into the buffer pool. With the recompute off the+layer keeps its activations instead. Pool-slot tensors are still never saved as+live views (#331): ``_body_retained`` saves them as tokens and, at the first+backward unpack of a layer, does the same ``pool.wait`` + ``prefetcher.advance``+the recompute did (``_backward_visit``). Later unpacks of that layer check the+slot still holds it.++Every model here streams 4 layers through 2 buffers, so slots really are+recycled between a layer's forward and its backward. The config field itself+is in ``tests/test_stream_recompute_config.py``.+"""++from __future__ import annotations++import io+import re+from types import SimpleNamespace+from unittest.mock import MagicMock++import pytest++from soup_cli.utils.layer_stream import plan_retained_layers++_N_LAYERS = 4+_BUFFERS = 2+_HIDDEN = 64 # tests.test_v07202._tiny_llama_dir+_TOKENS = 8+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*m")+++def _plain(text: str) -> str:+ """ANSI-stripped, whitespace-collapsed console output, safe to match."""+ return " ".join(_ANSI_RE.sub("", text).split())+++def _console():+ from rich.console import Console++ buffer = io.StringIO()+ return Console(file=buffer, width=200), buffer+++# --------------------------------------------------------------------------+# fixtures: the smallest CPU streamed models the existing suites build+# --------------------------------------------------------------------------+def _stream(tmp_path, quant):+ pytest.importorskip("torch")+ pytest.importorskip("peft")+ if quant == "nf4":+ pytest.importorskip("bitsandbytes")+ from tests.test_v07202 import _nf4_stream++ model, runtime, _weights, _index, _shards = _nf4_stream(+ tmp_path, n_layers=_N_LAYERS, buffers=_BUFFERS+ )+ else:+ from soup_cli.utils.layer_shard import shard_checkpoint+ from soup_cli.utils.layer_stream_runtime import build_streamed_model+ from tests.test_v07202 import _tiny_llama_dir, _tiny_lora++ weights, _resident, _config = _tiny_llama_dir(tmp_path, n_layers=_N_LAYERS)+ shards = str(tmp_path / "shards")+ index = shard_checkpoint(weights, shards, dtype="float32", arch="llama")+ model, runtime = build_streamed_model(+ model_id=weights,+ shard_dir=shards,+ index=index,+ lora_config=_tiny_lora(),+ device="cpu",+ dtype="float32",+ buffers=_BUFFERS,+ pin=False,+ seed=3,+ )+ from tests.test_v07202 import _randomise_lora_b++ # PEFT starts lora_B at zero, which zeroes every lora_A gradient and makes+ # an equality check vacuous for half the adapter.+ _randomise_lora_b(model)+ return model, runtime+++def _ids():+ import torch++ return torch.randint(0, 64, (1, _TOKENS), generator=torch.Generator().manual_seed(5))+++def _step(model, ids):+ model.zero_grad(set_to_none=True)+ out = model(input_ids=ids, labels=ids)+ out.loss.backward()+ grads = {+ name: param.grad.detach().clone()+ for name, param in model.named_parameters()+ if param.requires_grad+ }+ return out.loss.detach().clone(), grads+++def _layer_of(name: str) -> int:+ match = re.search(r"\.layers\.(\d+)\.", name)+ assert match, name+ return int(match.group(1))+++def _layer_class():+ from soup_cli.utils.layer_stream_runtime import _streamed_layer_class++ return _streamed_layer_class()+++def _count_retained_forwards(monkeypatch):+ """Which layer indices ran ``_body_retained`` (the no-recompute forward)."""+ cls = _layer_class()+ real = cls._body_retained+ seen: list = []++ def counting(self, *args, **kwargs):+ seen.append(self.idx)+ return real(self, *args, **kwargs)++ monkeypatch.setattr(cls, "_body_retained", counting)+ return seen+++def _assert_same(run_a, run_b):+ import torch++ loss_a, grads_a = run_a+ loss_b, grads_b = run_b+ assert torch.equal(loss_a, loss_b)+ assert grads_a.keys() == grads_b.keys()+ assert grads_a, "vacuous: nothing is trainable"+ for name in grads_a:+ assert torch.equal(grads_a[name], grads_b[name]), name+ assert all(bool(grad.abs().sum() > 0) for grad in grads_a.values()), (+ "vacuous: a LoRA gradient is all zero"+ )+++# --------------------------------------------------------------------------+# the arithmetic+# --------------------------------------------------------------------------+class TestRecomputeBudget:+ """min(85% of the card, free + reserved now, stream_vram_override) - reserved peak."""++ def _budget(self, **overrides):+ from soup_cli.utils.layer_stream import stream_recompute_budget++ args = dict(+ total_bytes=100_000,+ free_bytes=50_000,+ reserved_bytes=40_000,+ reserved_peak_bytes=30_000,+ )+ args.update(overrides)+ return stream_recompute_budget(**args)++ def test_the_card_cap_binds(self):+ # card int(0.85 * 100_000) = 85_000 < free + reserved 90_000+ budget = self._budget()+ assert (budget.bound, budget.cap_bytes, budget.budget_bytes) == ("card", 85_000, 55_000)++ def test_the_free_cap_binds_on_a_shared_card(self):+ # free + reserved = 20_000 + 40_000 = 60_000 < card 85_000+ budget = self._budget(free_bytes=20_000)+ assert (budget.bound, budget.cap_bytes, budget.budget_bytes) == ("free", 60_000, 30_000)++ def test_the_override_binds_when_it_is_the_smallest(self):+ budget = self._budget(override_bytes=50_000)+ assert (budget.bound, budget.cap_bytes, budget.budget_bytes) == (+ "override",+ 50_000,+ 20_000,+ )++ def test_an_override_above_the_others_does_not_raise_the_cap(self):+ budget = self._budget(override_bytes=10**9)+ assert (budget.bound, budget.cap_bytes) == ("card", 85_000)++ def test_a_peak_above_the_cap_is_a_negative_budget(self):+ assert self._budget(reserved_peak_bytes=90_000).budget_bytes == -5_000++ def test_the_headroom_scales_the_card_term(self):+ budget = self._budget(headroom=0.5)+ assert (budget.bound, budget.cap_bytes) == ("card", 50_000)++ @pytest.mark.parametrize("headroom", [0.0, -0.1, 1.01, True])+ def test_a_headroom_outside_0_1_is_refused(self, headroom):+ with pytest.raises(ValueError, match="headroom must be a float in"):+ self._budget(headroom=headroom)+++class TestRetainedLayerCharge:+ def test_charged_at_the_reserved_to_allocated_ratio(self):+ from soup_cli.utils.layer_stream import retained_layer_charge++ assert retained_layer_charge(10_000, 60_000, 50_000) == 12_000++ def test_rounded_up(self):+ from soup_cli.utils.layer_stream import retained_layer_charge++ # 10_001 * 1.2 = 12_001.2+ assert retained_layer_charge(10_001, 60_000, 50_000) == 12_002++ def test_never_below_the_allocated_bytes(self):+ from soup_cli.utils.layer_stream import retained_layer_charge++ assert retained_layer_charge(10_000, 40_000, 50_000) == 10_000++ def test_no_allocated_peak_charges_the_bytes(self):+ from soup_cli.utils.layer_stream import retained_layer_charge++ assert retained_layer_charge(10_000, 60_000, 0) == 10_000+++class TestPlanRetainedLayers:+ def test_exact_floor(self):+ assert plan_retained_layers(15_000, 4_000, 32) == 3++ def test_a_budget_one_byte_short_of_a_layer_floors_down(self):+ assert plan_retained_layers(11_999, 4_000, 32) == 2+ assert plan_retained_layers(12_000, 4_000, 32) == 3++ def test_clamped_at_n_layers(self):+ assert plan_retained_layers(10**12, 1, 32) == 32++ @pytest.mark.parametrize("budget", [0, -1, -(10**9)])+ def test_zero_or_negative_budget_is_zero(self, budget):+ assert plan_retained_layers(budget, 4_000, 32) == 0++ @pytest.mark.parametrize("per_layer", [0, -1])+ def test_a_layer_charge_at_or_below_zero_is_zero(self, per_layer):+ assert plan_retained_layers(100_000, per_layer, 32) == 0+++# --------------------------------------------------------------------------+# the switch+# --------------------------------------------------------------------------+class TestSetStreamRecompute:+ def test_returns_the_layer_count_and_switches_every_layer(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import (+ set_stream_recompute,+ streamed_decoder_layers,+ )++ model, _runtime = _stream(tmp_path, "none")+ layers = streamed_decoder_layers(model)+ assert [layer.idx for layer in layers] == list(range(_N_LAYERS))+ assert all(layer.recompute for layer in layers)++ assert set_stream_recompute(model, False) == _N_LAYERS+ assert [layer.recompute for layer in layers] == [False] * _N_LAYERS++ @pytest.mark.parametrize("retain", [0, 1, 2, 3, 4, 9])+ def test_retain_switches_exactly_the_last_k(self, tmp_path, retain):+ from soup_cli.utils.layer_stream_runtime import (+ set_stream_recompute,+ streamed_decoder_layers,+ )++ model, _runtime = _stream(tmp_path, "none")+ assert set_stream_recompute(model, False, retain=retain) == _N_LAYERS+ kept = {layer.idx for layer in streamed_decoder_layers(model) if not layer.recompute}+ assert kept == set(range(max(0, _N_LAYERS - retain), _N_LAYERS))++ def test_enabled_true_restores_every_layer(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import (+ set_stream_recompute,+ streamed_decoder_layers,+ )++ model, _runtime = _stream(tmp_path, "none")+ set_stream_recompute(model, False, retain=3)+ assert set_stream_recompute(model, True) == _N_LAYERS+ assert all(layer.recompute for layer in streamed_decoder_layers(model))++ @pytest.mark.parametrize("retain", [-1, True, 1.5])+ def test_a_bad_retain_is_refused(self, tmp_path, retain):+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, "none")+ with pytest.raises(ValueError, match="retain must be None or an int >= 0"):+ set_stream_recompute(model, False, retain=retain)++ def test_a_model_without_streamed_layers_switches_nothing(self):+ torch = pytest.importorskip("torch")+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ assert set_stream_recompute(torch.nn.Linear(2, 2), False) == 0++ def test_packed_backward_flags_only_on_switched_nf4_layers(self, tmp_path):+ import bitsandbytes as bnb++ from soup_cli.utils.layer_stream_runtime import (+ set_stream_recompute,+ streamed_decoder_layers,+ )++ model, _runtime = _stream(tmp_path, "nf4")++ def flags():+ out = {}+ for layer in streamed_decoder_layers(model):+ children = [+ child for child in layer.inner.modules()+ if isinstance(child, bnb.nn.Linear4bit)+ ]+ assert children, "vacuous: no Linear4bit under a streamed layer"+ out[layer.idx] = {+ bool(getattr(child, "_soup_packed_backward", False)) for child in children+ }+ return out++ assert flags() == {idx: {False} for idx in range(_N_LAYERS)}+ set_stream_recompute(model, False, retain=2)+ assert flags() == {0: {False}, 1: {False}, 2: {True}, 3: {True}}+ set_stream_recompute(model, True)+ assert flags() == {idx: {False} for idx in range(_N_LAYERS)}++ def test_retain_is_keyword_only(self, tmp_path):+ """``retain`` is keyword-only: a positional count must fail loudly rather+ than be read as how many layers keep their activations."""+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, "none")+ with pytest.raises(TypeError):+ set_stream_recompute(model, False, 2)++ def test_a_stream_without_nf4_never_imports_bitsandbytes_here(self, tmp_path, monkeypatch):+ import soup_cli.utils.layer_stream_runtime as rt++ model, _runtime = _stream(tmp_path, "none")+ assert all(layer.n_dequant_forward == 0 for layer in rt.streamed_decoder_layers(model))++ def refuse():+ raise AssertionError("bitsandbytes was reached for a layer with no NF4 linear")++ monkeypatch.setattr(rt, "_linear4bit_class", refuse)+ assert rt.set_stream_recompute(model, False) == _N_LAYERS+ assert rt.set_stream_recompute(model, True) == _N_LAYERS+++# --------------------------------------------------------------------------+# exactness, and the control that proves the harness would see #331+# --------------------------------------------------------------------------[email protected]("quant", ["none", "nf4"])+class TestExactness:+ def test_off_is_bit_identical_to_the_recompute(self, tmp_path, monkeypatch, quant):+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, runtime = _stream(tmp_path, quant)+ ids = _ids()+ loads_before = runtime.pool.loads+ shipped = _step(model, ids)+ # 4 layers through 2 slots: layers 2 and 3 are read in the forward and+ # 1 and 0 again in the backward (layer 0 also in a first forward).+ assert runtime.pool.loads - loads_before >= _N_LAYERS++ retained = _count_retained_forwards(monkeypatch)+ assert set_stream_recompute(model, False) == _N_LAYERS+ loads_before = runtime.pool.loads+ no_recompute = _step(model, ids)+ assert sorted(retained) == list(range(_N_LAYERS)), "the no-recompute path did not run"+ # The backward visit re-reads layers 1 and 0 exactly as the recompute did.+ assert runtime.pool.loads - loads_before == _N_LAYERS+ _assert_same(shipped, no_recompute)++ def test_partial_retain_is_bit_identical(self, tmp_path, monkeypatch, quant):+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, quant)+ ids = _ids()+ shipped = _step(model, ids)+ retained = _count_retained_forwards(monkeypatch)+ set_stream_recompute(model, False, retain=2)+ mixed = _step(model, ids)+ assert sorted(retained) == [2, 3]+ _assert_same(shipped, mixed)++ def test_back_on_is_bit_identical(self, tmp_path, monkeypatch, quant):+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, quant)+ ids = _ids()+ shipped = _step(model, ids)+ set_stream_recompute(model, False)+ _step(model, ids)+ set_stream_recompute(model, True)+ retained = _count_retained_forwards(monkeypatch)+ again = _step(model, ids)+ assert retained == []+ _assert_same(shipped, again)+++class TestTheHarnessSeesAliasing:+ """#331-class negative controls: with the backward visit gone, the pool slot+ of a lower layer still holds an upper layer when its backward reads it."""++ def test_plain_weights_give_wrong_grads_without_the_backward_visit(+ self, tmp_path, monkeypatch+ ):+ import torch++ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, "none")+ ids = _ids()+ loss_on, grads_on = _step(model, ids)+ set_stream_recompute(model, False)+ monkeypatch.setattr(_layer_class(), "_backward_visit", lambda self: None)+ loss_off, grads_off = _step(model, ids)++ assert torch.equal(loss_on, loss_off) # the forward is unaffected+ differing = {+ _layer_of(name) for name in grads_on if not torch.equal(grads_on[name], grads_off[name])+ }+ # Layers 3 and 2 still own their slots; 1 and 0 read layers 3 and 2.+ assert differing == {0, 1}++ def test_nf4_owner_check_fires_without_the_backward_visit(self, tmp_path, monkeypatch):+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, "nf4")+ set_stream_recompute(model, False)+ monkeypatch.setattr(_layer_class(), "_backward_visit", lambda self: None)+ with pytest.raises(+ RuntimeError, match="streamed NF4 backward found a recycled weight slot"+ ):+ _step(model, _ids())++ def test_a_slot_recycled_mid_backward_is_refused(self, tmp_path, monkeypatch):+ """Layer 3's first backward unpack visits it; a load into its slot right+ after must stop the backward at layer 3's next unpack."""+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, runtime = _stream(tmp_path, "none")+ set_stream_recompute(model, False)+ real_advance = runtime.prefetcher.advance+ state = {"backward": False}++ def advance(idx):+ real_advance(idx)+ if state["backward"] and idx == _N_LAYERS - 1:+ runtime.pool.load_async(1, runtime.source) # slot 1 = layer 3's slot++ monkeypatch.setattr(runtime.prefetcher, "advance", advance)+ model.zero_grad(set_to_none=True)+ out = model(input_ids=_ids(), labels=_ids())+ state["backward"] = True+ with pytest.raises(+ RuntimeError,+ match=(+ r"streamed backward \(no recompute\) found slot 1 holding layer 1 "+ r"while layer 3 still unpacks from it"+ ),+ ):+ out.loss.backward()+++class TestBackwardVisit:+ def _layer(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import streamed_decoder_layers++ model, runtime = _stream(tmp_path, "none")+ return streamed_decoder_layers(model)[0], runtime++ def test_the_first_visit_waits_and_prefetches_once(self, tmp_path, monkeypatch):+ layer, runtime = self._layer(tmp_path)+ calls = []+ monkeypatch.setattr(runtime.pool, "wait", lambda idx: calls.append(("wait", idx)))+ monkeypatch.setattr(runtime.prefetcher, "advance", lambda idx: calls.append(("adv", idx)))+ runtime.pool.owner[runtime.pool.slot_for(0)] = 0+ layer._bwd_visited = False+ layer._backward_visit()+ layer._backward_visit()+ assert calls == [("wait", 0), ("adv", 0)]+ assert layer._bwd_visited is True++ def test_a_later_unpack_from_a_recycled_slot_raises(self, tmp_path):+ layer, runtime = self._layer(tmp_path)+ layer._bwd_visited = True+ runtime.pool.owner[runtime.pool.slot_for(layer.idx)] = 2+ with pytest.raises(+ RuntimeError,+ match=(+ r"streamed backward \(no recompute\) found slot 0 holding layer 2 "+ r"while layer 0 still unpacks from it"+ ),+ ):+ layer._backward_visit()++ def _visited(self, tmp_path):+ """Layer 0 after its first backward visit, its slot holding it."""+ layer, runtime = self._layer(tmp_path)+ runtime.pool.load_async(0, runtime.source)+ layer._bwd_visited = False+ layer._backward_visit()+ return layer, runtime++ def test_a_later_unpack_from_its_own_slot_passes(self, tmp_path):+ layer, _runtime = self._visited(tmp_path)+ layer._backward_visit()+ layer._backward_visit()++ def test_a_slot_reloaded_with_the_same_layer_is_refused(self, tmp_path):+ """The owner alone cannot see slot 0 refilled with layer 2 and then layer 0+ again between two unpacks; the load generation can."""+ layer, runtime = self._visited(tmp_path)+ before = runtime.pool.generation[0]+ runtime.pool.load_async(2, runtime.source)+ runtime.pool.load_async(0, runtime.source)+ assert runtime.pool.owner[0] == 0+ with pytest.raises(+ RuntimeError,+ match=(+ rf"streamed backward \(no recompute\) found slot 0 reloaded \(load "+ rf"generation {before} -> {before + 2}\) while layer 0 still unpacks from it"+ ),+ ):+ layer._backward_visit()++ def test_every_load_bumps_the_slot_generation(self, tmp_path):+ _layer, runtime = self._layer(tmp_path)+ pool = runtime.pool+ start = list(pool.generation)+ pool.load_async(0, runtime.source)+ pool.load_async(1, runtime.source)+ pool.load_async(2, runtime.source)+ assert pool.generation == [start[0] + 2, start[1] + 1]+++class TestExactnessWithOtherFeatures:+ def test_off_with_the_nf4_scale_cache_is_bit_identical(self, tmp_path):+ """The off step runs FIRST, so the scale cache is filled under the+ no-recompute path; the recompute step then reuses it."""+ pytest.importorskip("bitsandbytes")+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute+ from tests.test_stream_nf4_scale_cache import _nf4_stream+ from tests.test_v07202 import _randomise_lora_b++ model, runtime, _index = _nf4_stream(tmp_path, cache=True, n_layers=_N_LAYERS)+ _randomise_lora_b(model)+ ids = _ids()+ set_stream_recompute(model, False)+ no_recompute = _step(model, ids)+ assert len(runtime.scale_cache) > 0, "vacuous: the scale cache was never filled"+ set_stream_recompute(model, True)+ _assert_same(_step(model, ids), no_recompute)++ def test_two_accumulated_micro_steps_are_bit_identical(self, tmp_path):+ import torch++ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ model, _runtime = _stream(tmp_path, "nf4")+ generator = torch.Generator().manual_seed(11)+ batches = [torch.randint(0, 64, (1, _TOKENS), generator=generator) for _ in range(2)]++ def accumulate():+ model.zero_grad(set_to_none=True)+ losses = []+ for ids in batches:+ out = model(input_ids=ids, labels=ids)+ (out.loss / len(batches)).backward()+ losses.append(out.loss.detach().clone())+ grads = {+ name: param.grad.detach().clone()+ for name, param in model.named_parameters()+ if param.requires_grad+ }+ return torch.stack(losses), grads++ shipped = accumulate()+ set_stream_recompute(model, False)+ _assert_same(shipped, accumulate())++ def test_off_through_the_real_setup_with_fast_lora(self, tmp_path, monkeypatch):+ """``stream_recompute: off`` from the config through the real setup(),+ with the fast LoRA kernels on: every layer is switched, and the LoRA+ gradients equal the recompute's on the same model."""+ import torch++ from soup_cli.utils.layer_stream_runtime import (+ set_stream_recompute,+ streamed_decoder_layers,+ )+ from tests.test_fast_lora_flag import TARGETS+ from tests.test_v07204 import _build_streamed_wrapper++ wrapper, _, _ = _build_streamed_wrapper(+ tmp_path,+ monkeypatch,+ task="sft",+ n_layers=_N_LAYERS,+ device="cpu",+ tie=False,+ stream_buffers=_BUFFERS,+ fast_lora=True,+ stream_recompute="off",+ lora={"r": 4, "alpha": 8, "dropout": 0.0, "target_modules": TARGETS},+ )+ try:+ model = wrapper.model+ assert [layer.recompute for layer in streamed_decoder_layers(model)] == [+ False+ ] * _N_LAYERS+ assert sum("forward" in vars(module) for module in model.modules()) > 0, (+ "vacuous: the fast LoRA kernels are not on"+ )+ with torch.no_grad():+ for name, param in model.named_parameters():+ if "lora_B" in name:+ torch.manual_seed(abs(hash(name)) % 2**31)+ param.normal_(std=0.1)+ ids = torch.tensor([[1, 5, 9, 13, 2, 7, 11]])+ no_recompute = _step(model, ids)+ set_stream_recompute(model, True)+ _assert_same(_step(model, ids), no_recompute)+ finally:+ wrapper._close_stream_runtime()+++# --------------------------------------------------------------------------+# 'auto': one synthetic step at the configured shape, at setup+# --------------------------------------------------------------------------+_OUTPUT_BYTES = 1 * _TOKENS * _HIDDEN * 4 # the measured layer's fp32 output+++def _readings(*values):+ """A fake ``memory_allocated`` returning ``values`` in order, then failing."""+ it = iter(values)+ calls = []++ def allocated():+ calls.append(1)+ return next(it)++ return allocated, calls+++class TestRetainedLayerProbe:+ def test_measures_the_last_layer_once(self, tmp_path, monkeypatch):+ from soup_cli.utils.layer_stream_runtime import RetainedLayerProbe, streamed_decoder_layers++ model, _runtime = _stream(tmp_path, "none")+ ids = _ids()+ shipped = _step(model, ids)++ allocated, calls = _readings(1_000, 1_000 + 7_000)+ probe = RetainedLayerProbe(allocated=allocated)+ retained = _count_retained_forwards(monkeypatch)+ assert probe.arm(model) == _N_LAYERS - 1+ measured_step = _step(model, ids)+ assert retained == [_N_LAYERS - 1]+ assert probe.per_layer_bytes == 7_000 - _OUTPUT_BYTES+ assert probe.layer_idx == _N_LAYERS - 1+ assert probe.error is None+ # The measured step trains exactly like the shipped one.+ _assert_same(shipped, measured_step)+ # The next forward is the shipped path again, and nothing is re-read.+ assert all(layer.recompute for layer in streamed_decoder_layers(model))+ _step(model, ids)+ assert retained == [_N_LAYERS - 1]+ assert len(calls) == 2++ def test_bytes_the_scale_cache_fills_are_not_counted_as_retained(self, tmp_path):+ """The first visit of a layer fills its share of stream_nf4_scale_cache+ inside the measured forward; that is persistent VRAM, charged up front by+ the pre-flight, not activations the layer keeps for its backward."""+ pytest.importorskip("bitsandbytes")+ from soup_cli.utils.layer_stream import nf4_scale_cache_bytes+ from soup_cli.utils.layer_stream_runtime import RetainedLayerProbe+ from tests.test_stream_nf4_scale_cache import _nf4_stream++ model, runtime, index = _nf4_stream(tmp_path, cache=True, n_layers=_N_LAYERS)+ filled_by_one_layer = nf4_scale_cache_bytes(index.quant_specs, n_layers=1)+ assert filled_by_one_layer > 0+ allocated, _calls = _readings(1_000, 1_000 + 900_000)+ probe = RetainedLayerProbe(allocated=allocated)+ probe.arm(model)+ _step(model, _ids()) # the first visit of every layer+ assert runtime.scale_cache.nbytes == _N_LAYERS * filled_by_one_layer+ assert probe.per_layer_bytes == 900_000 - filled_by_one_layer - _OUTPUT_BYTES++ def test_a_cpu_pool_without_a_reader_cannot_measure(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import RetainedLayerProbe, streamed_decoder_layers++ model, _runtime = _stream(tmp_path, "none")+ probe = RetainedLayerProbe()+ assert probe.arm(model) is None+ assert probe.error == "no CUDA device to measure the retained bytes on"+ assert all(layer.recompute for layer in streamed_decoder_layers(model))++ def test_a_reader_that_raises_is_recorded_not_raised(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import RetainedLayerProbe++ model, _runtime = _stream(tmp_path, "none")++ def broken():+ raise RuntimeError("boom")++ probe = RetainedLayerProbe(allocated=broken)+ probe.arm(model)+ _step(model, _ids()) # training goes on+ assert probe.per_layer_bytes is None+ assert probe.error == "the measurement raised RuntimeError: boom"++ def test_no_streamed_layers(self):+ torch = pytest.importorskip("torch")+ from soup_cli.utils.layer_stream_runtime import RetainedLayerProbe++ probe = RetainedLayerProbe(allocated=lambda: 0)+ assert probe.arm(torch.nn.Linear(2, 2)) is None+ assert probe.error == "the model has no streamed decoder layers"+++def _kept(model):+ from soup_cli.utils.layer_stream_runtime import streamed_decoder_layers++ return {layer.idx for layer in streamed_decoder_layers(model) if not layer.recompute}+++class _FakeMemory:+ """The allocator figures the planner reads, CPU-side. ``readings`` feed the+ probe's ``allocated()`` in order; None makes it raise."""++ def __init__(+ self,+ *,+ readings=(1_000, 21_000),+ peak_allocated=50_000,+ peak_reserved=60_000,+ reserved=60_000,+ free=30_000,+ total=200_000,+ ):+ self._readings = None if readings is None else iter(readings)+ self._figures = dict(+ peak_allocated=peak_allocated,+ peak_reserved=peak_reserved,+ reserved=reserved,+ free=free,+ total=total,+ )+ self.events = []++ def allocated(self):+ self.events.append("allocated")+ if self._readings is None:+ raise RuntimeError("boom")+ return next(self._readings)++ def reset_peak(self):+ self.events.append("reset_peak")++ def synchronize(self):+ self.events.append("synchronize")++ def release(self):+ self.events.append("release")++ def peak_allocated(self):+ return self._figures["peak_allocated"]++ def peak_reserved(self):+ return self._figures["peak_reserved"]++ def reserved(self):+ return self._figures["reserved"]++ def free(self):+ return self._figures["free"]++ def total(self):+ return self._figures["total"]+++#: The configured shape the planner is handed, and its last layer's fp32 output.+_ROWS, _SEQ = 2, 16+_SHAPE_OUTPUT_BYTES = _ROWS * _SEQ * _HIDDEN * 4 # 8_192+++def _plan(model, console, memory, **kwargs):+ from soup_cli.utils.layer_stream_runtime import plan_stream_recompute_auto++ kwargs.setdefault("rows", _ROWS)+ kwargs.setdefault("seq_len", _SEQ)+ kwargs.setdefault("vocab_size", 64)+ return plan_stream_recompute_auto(model, notify=console.print, memory=memory, **kwargs)+++class TestPlanStreamRecomputeAuto:+ """'auto' decides before training, from one synthetic step at the configured+ shape (not from whatever the first real batch pads to)."""++ def test_plans_k_from_the_synthetic_step(self, tmp_path):+ model, _runtime = _stream(tmp_path, "none")+ console, buffer = _console()+ memory = _FakeMemory()+ # per layer = 20_000 - 8_192 = 11_808; charge = ceil(11_808 x 60_000 / 50_000)+ # = 14_170; cap = min(170_000 card, 30_000 + 60_000 free) = 90_000;+ # budget = 90_000 - 60_000 = 30_000 -> K = 2+ assert _plan(model, console, memory) == 2+ assert _kept(model) == {2, 3}+ assert memory.events[0] == "reset_peak"+ assert memory.events[-1] == "release"+ said = _plain(buffer.getvalue())+ assert (+ "training.stream_recompute='auto': the last 2 of 4 streamed layers keep their "+ "activations, the rest recompute" in said+ )+ assert (+ "measured at batch 2 x seq 16: one layer 11808 bytes retained, charged 14170 "+ "bytes at reserved/allocated 1.20" in said+ )+ assert (+ "budget 30000 bytes = free + reserved VRAM 90000 bytes minus the reserved "+ "peak 60000 bytes" in said+ )++ def test_the_step_runs_at_the_configured_shape(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import _synthetic_step++ model, _runtime = _stream(tmp_path, "none")+ console, buffer = _console()+ shapes = []++ def step(model, *, rows, seq_len, vocab_size, device):+ shapes.append((rows, seq_len, vocab_size, str(device)))+ _synthetic_step(model, rows=rows, seq_len=seq_len, vocab_size=vocab_size, device=device)++ _plan(model, console, _FakeMemory(), rows=3, seq_len=24, step=step)+ assert shapes == [(3, 24, 64, "cpu")]+ # The measured layer's output was that shape's: 20_000 - 3 x 24 x 64 x 4.+ assert "one layer 1568 bytes retained" in _plain(buffer.getvalue())++ def test_the_synthetic_step_leaves_no_gradient(self, tmp_path):+ model, _runtime = _stream(tmp_path, "none")+ console, _buffer = _console()+ _plan(model, console, _FakeMemory())+ assert all(param.grad is None for param in model.parameters())++ def test_the_planned_mode_trains_bit_identically(self, tmp_path):+ model, _runtime = _stream(tmp_path, "none")+ console, _buffer = _console()+ ids = _ids()+ shipped = _step(model, ids)+ assert _plan(model, console, _FakeMemory()) == 2+ _assert_same(shipped, _step(model, ids))++ def test_the_override_caps_the_budget(self, tmp_path):+ model, _runtime = _stream(tmp_path, "none")+ console, buffer = _console()+ # free cap 10**6 + 60_000, card 170_000; override 74_170 -> budget 14_170 -> K = 1+ memory = _FakeMemory(free=10**6)+ assert _plan(model, console, memory, override_bytes=74_170) == 1+ assert _kept(model) == {3}+ assert "training.stream_vram_override 74170 bytes minus the reserved peak" in _plain(+ buffer.getvalue()+ )++ def test_the_card_cap_binds(self, tmp_path):+ model, _runtime = _stream(tmp_path, "none")+ console, buffer = _console()+ # card int(0.85 x 100_000) = 85_000 -> budget 25_000 -> K = 1+ memory = _FakeMemory(free=10**6, total=100_000)+ assert _plan(model, console, memory) == 1+ assert "85% of the card's 100000 bytes minus the reserved peak" in _plain(+ buffer.getvalue()+ )+++class TestPlanStreamRecomputeAutoFallbacks:+ """Every path that cannot plan keeps the checkpoint recompute and says why."""++ def _never(self, *_a, **_k):+ raise AssertionError("the synthetic step ran although nothing could be measured")++ def _run(self, tmp_path, memory, **kwargs):+ model, _runtime = _stream(tmp_path, "none")+ console, buffer = _console()+ k = _plan(model, console, memory, **kwargs)+ return k, model, _plain(buffer.getvalue())++ def _assert_kept(self, k, model, said, why):+ from soup_cli.utils.layer_stream_runtime import streamed_decoder_layers++ assert k == 0+ assert _kept(model) == set()+ assert all(layer._retained_probe is None for layer in streamed_decoder_layers(model))+ assert (+ "training.stream_recompute='auto' keeps the recompute on all 4 streamed layers:"+ in said+ )+ assert why in said++ def test_no_cuda_device(self, tmp_path):+ k, model, said = self._run(tmp_path, None, step=self._never)+ self._assert_kept(k, model, said, "no CUDA device to measure VRAM on")++ @staticmethod+ def _out_of_memory_errors():+ import torch++ return [+ # the allocator's own+ torch.cuda.OutOfMemoryError("CUDA out of memory. Tried to allocate 2.00 GiB"),+ # an asynchronous OOM surfacing at a synchronise (#649 / #901)+ RuntimeError("CUDA error: out of memory"),+ ]++ @pytest.mark.parametrize("which", [0, 1])+ def test_an_out_of_memory_step_keeps_the_recompute(self, tmp_path, which):+ error = self._out_of_memory_errors()[which]++ def oom(*_a, **_k):+ raise error++ k, model, said = self._run(tmp_path, _FakeMemory(), step=oom)+ self._assert_kept(+ k,+ model,+ said,+ "the synthetic step at batch 2 x seq 16 ran out of memory, so the configured "+ "shape does not fit with one layer retained; set training.stream_vram_probe: "+ "true to measure whether it fits at all",+ )++ @pytest.mark.parametrize(+ "error",+ [+ RuntimeError("CUDA error: an illegal memory access was encountered"),+ ValueError("a bug in the step"),+ ],+ )+ def test_any_other_step_error_propagates(self, tmp_path, error):+ """Not a fit answer: the context may be unusable, so the setup must get+ the exception (and close the runtime) rather than train on."""+ from soup_cli.utils.layer_stream_runtime import streamed_decoder_layers++ model, _runtime = _stream(tmp_path, "none")+ console, buffer = _console()+ memory = _FakeMemory()++ def broken(*_a, **_k):+ raise error++ with pytest.raises(type(error), match=re.escape(str(error))):+ _plan(model, console, memory, step=broken)+ assert _kept(model) == set()+ assert all(layer._retained_probe is None for layer in streamed_decoder_layers(model))+ assert memory.events[-1] == "release"+ assert "keeps the recompute" not in _plain(buffer.getvalue())++ def test_the_measurement_raised(self, tmp_path):+ k, model, said = self._run(tmp_path, _FakeMemory(readings=None))+ self._assert_kept(k, model, said, "the measurement raised RuntimeError: boom")++ def test_a_layer_measured_at_or_below_zero(self, tmp_path):+ k, model, said = self._run(tmp_path, _FakeMemory(readings=(1_000, 1_000)))+ self._assert_kept(+ k, model, said, f"one layer measured at {-_SHAPE_OUTPUT_BYTES} bytes retained (<= 0)"+ )++ def test_no_vram_headroom(self, tmp_path):+ k, model, said = self._run(tmp_path, _FakeMemory(peak_reserved=95_000))+ self._assert_kept(+ k,+ model,+ said,+ "no VRAM headroom: free + reserved VRAM 90000 bytes minus the reserved peak of "+ "the batch 2 x seq 16 step, 95000 bytes, is -5000 bytes",+ )++ def test_a_budget_smaller_than_one_layer(self, tmp_path):+ # cap 12_000 + 60_000 = 72_000 -> budget 12_000 < one layer's charge 14_170+ k, model, said = self._run(tmp_path, _FakeMemory(free=12_000))+ self._assert_kept(+ k, model, said, "one layer (14170 bytes charged) does not fit the budget of 12000 bytes"+ )++ def test_no_step_shape(self, tmp_path):+ k, model, said = self._run(tmp_path, _FakeMemory(), vocab_size=0, step=self._never)+ self._assert_kept(+ k, model, said, "no step shape to measure (batch 2 x seq 16, vocab_size 0)"+ )++ def test_no_streamed_layers(self):+ torch = pytest.importorskip("torch")+ console, buffer = _console()+ assert _plan(torch.nn.Linear(2, 2), console, _FakeMemory(), step=self._never) == 0+ assert "the model has no streamed decoder layers" in _plain(buffer.getvalue())+++# --------------------------------------------------------------------------+# the setup path, from YAML to the runtime (spied), on CPU+# --------------------------------------------------------------------------+_MAX_LENGTH = 96+++def _drive(+ tmp_path,+ monkeypatch,+ *,+ mode,+ stream_source="ram",+ device="cuda",+ extra_yaml="",+ plan_error=None,+):+ """Real config and pre-flight, spied ``build_streamed_model``, switch, planner+ and VRAM probe.++ Returns ``(wrapper, model, events, console_text)``; ``events`` records each+ spied call in order, and ``("close",)`` when the runtime is closed. With+ ``plan_error`` the planner raises it, and the setup must raise it too.+ """+ pytest.importorskip("torch")+ from rich.console import Console++ import soup_cli.trainer.stream_setup as ss+ import soup_cli.utils.layer_stream as ls+ import soup_cli.utils.layer_stream_runtime as rt+ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.sft import SFTTrainerWrapper+ from tests.test_issue623_stream_pin_allocation import (+ _FREE_BYTES,+ _cfg_yaml,+ _tiny_checkpoint,+ )++ weights = _tiny_checkpoint(tmp_path)+ monkeypatch.chdir(tmp_path)+ monkeypatch.setenv("SOUP_LAYER_STREAM_CACHE_DIR", str(tmp_path / "cache"))+ monkeypatch.setattr(+ "soup_cli.utils.spectrum_scan.resolve_model_weights", lambda *_a, **_k: weights+ )+ monkeypatch.setattr(ls, "free_ram_bytes", lambda: _FREE_BYTES)+ monkeypatch.setattr(+ ls, "classify_disk_kind", lambda *_a, **_k: ls.DiskClassification("nvme")+ )+ import torch++ monkeypatch.setattr(torch.cuda, "mem_get_info", lambda *_a, **_k: (_FREE_BYTES, _FREE_BYTES))+ monkeypatch.setattr(rt, "expandable_segments_status", lambda *_a, **_k: (True, ""))++ built = MagicMock(name="streamed-model")+ events = []++ def spy_build(**kwargs):+ runtime = MagicMock()+ runtime.close.side_effect = lambda: events.append(("close",))+ runtime.tier = kwargs.get("tier", "ram")+ runtime.stats.return_value = {+ "tier": runtime.tier,+ "store_bytes": 1_000_000,+ "disk_bytes": 1_000_000,+ "pinned": False,+ "read_ahead": kwargs.get("read_ahead"),+ "buffers": 2,+ "buffer_bytes": 4_000,+ "n_layers": 3,+ }+ return built, runtime++ monkeypatch.setattr(rt, "build_streamed_model", spy_build)++ def spy_switch(model, enabled, **kwargs):+ events.append(("switch", model, enabled, kwargs))+ return 3++ def spy_plan(model, **kwargs):+ events.append(("plan", model, kwargs))+ if plan_error is not None:+ raise plan_error+ return 2++ def spy_probe(self, model, plan):+ events.append(("probe", model, plan.rows, plan.seq_len))++ monkeypatch.setattr(rt, "set_stream_recompute", spy_switch)+ monkeypatch.setattr(rt, "plan_stream_recompute_auto", spy_plan)+ monkeypatch.setattr(SFTTrainerWrapper, "_run_stream_vram_probe", spy_probe)+ buffer = io.StringIO()+ monkeypatch.setattr(ss, "console", Console(file=buffer, width=200))++ extra = "" if mode is None else f" stream_recompute: '{mode}'\n"+ text = _cfg_yaml(+ weights, "", stream_source=stream_source, extra_training_yaml=extra + extra_yaml+ ).replace(" format: alpaca\n", f" format: alpaca\n max_length: {_MAX_LENGTH}\n")+ cfg = load_config_from_string(text)+ assert cfg.data.max_length == _MAX_LENGTH+ wrapper = SFTTrainerWrapper(cfg)+ wrapper.device = device+ if plan_error is None:+ wrapper._setup_streaming_transformers(cfg, cfg.training)+ else:+ with pytest.raises(type(plan_error), match=re.escape(str(plan_error))):+ wrapper._setup_streaming_transformers(cfg, cfg.training)+ return wrapper, built, events, _plain(buffer.getvalue())+++class _FakeTrainer:+ def __init__(self):+ self.callbacks = []++ def add_callback(self, callback):+ self.callbacks.append(callback)+++def _callbacks(wrapper):+ """What the trainer gets right before training: the save guard only."""+ wrapper.trainer = _FakeTrainer()+ wrapper._attach_streamed_save_guard()+ return [type(callback).__name__ for callback in wrapper.trainer.callbacks]+++class TestSetupWiring:+ @pytest.mark.parametrize("mode", [None, "on"])+ def test_on_changes_nothing(self, tmp_path, monkeypatch, mode):+ wrapper, _model, events, said = _drive(tmp_path, monkeypatch, mode=mode)+ assert events == []+ assert _callbacks(wrapper) == ["StreamedSaveGuardCallback"]+ assert "stream_recompute" not in said++ def test_off_switches_every_layer_and_says_how_many(self, tmp_path, monkeypatch):+ _wrapper, model, events, said = _drive(tmp_path, monkeypatch, mode="off")+ assert events == [("switch", model, False, {})]+ assert (+ "training.stream_recompute='off': 3 streamed layers keep their activations "+ "instead of recomputing them in the backward" in said+ )+ assert "read-bound" not in said++ def test_off_on_the_disk_tier_warns_and_still_switches(self, tmp_path, monkeypatch):+ _wrapper, model, events, said = _drive(+ tmp_path, monkeypatch, mode="off", stream_source="disk"+ )+ assert events == [("switch", model, False, {})]+ assert "training.stream_recompute='off' on the disk tier:" in said+ assert "a disk-tier step is usually read-bound" in said+ assert "the recompute ('on') may be cheaper here" in said++ def test_auto_plans_at_setup_from_the_configured_shape(self, tmp_path, monkeypatch):+ wrapper, model, events, said = _drive(tmp_path, monkeypatch, mode="auto")+ assert events == [+ (+ "plan",+ model,+ {+ "rows": 1, # training.batch_size+ "seq_len": _MAX_LENGTH, # data.max_length+ "vocab_size": 64,+ "notify": events[0][2]["notify"],+ "override_bytes": None,+ },+ )+ ]+ # Decided before training: nothing is left for a training callback.+ assert _callbacks(wrapper) == ["StreamedSaveGuardCallback"]+ assert (+ f"training.stream_recompute='auto': measuring one step at batch 1 x seq "+ f"{_MAX_LENGTH} to choose how many of the last layers keep their activations "+ "within 85% of the card." in said+ )++ def test_auto_passes_the_vram_override(self, tmp_path, monkeypatch):+ _wrapper, _model, events, _said = _drive(+ tmp_path, monkeypatch, mode="auto", extra_yaml=" stream_vram_override: 5000000000\n"+ )+ assert [event[0] for event in events] == ["plan"]+ assert events[0][2]["override_bytes"] == 5_000_000_000++ def test_a_planner_error_closes_the_runtime_and_refuses(self, tmp_path, monkeypatch):+ """A non-OOM error of the synthetic step propagates out of the planner;+ the setup releases the streaming runtime and re-raises it."""+ error = RuntimeError("CUDA error: an illegal memory access was encountered")+ _wrapper, _model, events, _said = _drive(+ tmp_path, monkeypatch, mode="auto", plan_error=error+ )+ assert [event[0] for event in events] == ["plan", "close"]++ @pytest.mark.parametrize("mode", ["off", "auto"])+ def test_the_vram_probe_measures_the_mode_that_will_train(self, tmp_path, monkeypatch, mode):+ """stream_vram_probe runs after the switch (or the plan), so its peak is+ the retained mode's, not the recompute's."""+ _wrapper, model, events, _said = _drive(+ tmp_path, monkeypatch, mode=mode, extra_yaml=" stream_vram_probe: true\n"+ )+ assert [event[0] for event in events] == [+ "switch" if mode == "off" else "plan",+ "probe",+ ]+ assert events[-1] == ("probe", model, 1, _MAX_LENGTH)++ def test_auto_on_the_disk_tier_keeps_the_recompute(self, tmp_path, monkeypatch):+ wrapper, _model, events, said = _drive(+ tmp_path, monkeypatch, mode="auto", stream_source="disk"+ )+ assert events == []+ assert _callbacks(wrapper) == ["StreamedSaveGuardCallback"]+ assert (+ "training.stream_recompute='auto' keeps the recompute on the disk tier" in said+ )++ def test_auto_without_cuda_keeps_the_recompute(self, tmp_path, monkeypatch):+ _wrapper, _model, events, said = _drive(+ tmp_path, monkeypatch, mode="auto", device="cpu"+ )+ assert events == []+ assert (+ "training.stream_recompute='auto' keeps the recompute: no CUDA device to "+ "measure VRAM on" in said+ )++ @pytest.mark.parametrize("mode", ["off", "auto"])+ def test_the_setup_refuses_a_task_other_than_sft_too(self, mode, monkeypatch):+ """Defence in depth behind the config gate: a caller that reaches the+ setup with another task (a stand-in config) is refused before anything+ is switched."""+ from types import SimpleNamespace++ import soup_cli.utils.layer_stream_runtime as rt+ from soup_cli.trainer.stream_setup import StreamingSetupMixin++ def never(*_a, **_k):+ raise AssertionError("switched before the task was checked")++ monkeypatch.setattr(rt, "set_stream_recompute", never)+ monkeypatch.setattr(rt, "plan_stream_recompute_auto", never)+ with pytest.raises(+ ValueError,+ match=rf"training.stream_recompute='{mode}' supports task='sft' only; got task='dpo'",+ ):+ StreamingSetupMixin()._configure_stream_recompute(+ MagicMock(),+ SimpleNamespace(stream_recompute=mode),+ tier="ram",+ on_cuda=True,+ task="dpo",+ step_shape=(1, 8, 64),+ )+++class TestTheProbeHarness:+ """benchmarks/harness/stream_probe.py replaces the pool's ``load_async`` with+ a timed replica; the replica must bump the slot's load generation too, or+ the no-recompute backward's reload check goes blind under the probe."""++ def _harness(self):+ import importlib.util+ from pathlib import Path++ path = Path(__file__).resolve().parents[1] / "benchmarks" / "harness" / "stream_probe.py"+ spec = importlib.util.spec_from_file_location("stream_probe_under_test", path)+ module = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(module)+ return module++ def test_the_replica_load_bumps_the_slot_generation(self, tmp_path):+ harness = self._harness()+ model, runtime = _stream(tmp_path, "none")+ harness.Instruments(runtime, model, "none")+ pool = runtime.pool+ assert "Instruments" in pool.load_async.__qualname__, "the replica is not installed"+ before = list(pool.generation)+ pool.load_async(2, runtime.source)+ assert pool.owner[0] == 2+ assert pool.generation == [before[0] + 1, before[1]]++ def test_off_under_the_replica_is_bit_identical(self, tmp_path):+ from soup_cli.utils.layer_stream_runtime import set_stream_recompute++ harness = self._harness()+ model, runtime = _stream(tmp_path, "none")+ harness.Instruments(runtime, model, "none")+ ids = _ids()+ shipped = _step(model, ids)+ set_stream_recompute(model, False)+ _assert_same(shipped, _step(model, ids))+++class TestTheDefaultPathAddsNothing:+ """With ``stream_recompute`` 'on' (or a stand-in config that never heard of+ it) the setup must not read anything the feature added, call anything, or+ reach for a runtime method. Regression: the setup evaluated ``cfg.task`` and+ the step shape for every mode, so a stub config without ``task`` raised and+ the cleanup then called ``close()`` on a stub runtime that has none (#2152,+ test_issue803 / test_qwen35_streaming on CI)."""++ @pytest.mark.parametrize("spelled", [False, True])+ def test_a_minimal_stub_setup_makes_no_new_calls(self, monkeypatch, tmp_path, spelled):+ import soup_cli.utils.layer_stream_runtime as rt+ from soup_cli.trainer.stream_setup import StreamingSetupMixin+ from soup_cli.utils.layer_stream import DEFAULT_STREAM_READ_AHEAD+ from tests.test_issue803_lora_patterns_all_trainers import _config, _tokenizer++ def never(name):+ def refuse(*_a, **_k):+ raise AssertionError(f"{name} was reached on the default path")++ return refuse++ class _Wrapper(StreamingSetupMixin):+ def __init__(self):+ self.device = "cpu"+ self._trust_remote_code = False++ def _stream_budget_lines(self, *_args, **_kwargs):+ return (), None++ for name in ("_configure_stream_recompute", "_stream_step_shape"):+ monkeypatch.setattr(StreamingSetupMixin, name, never(name))+ for name in ("set_stream_recompute", "plan_stream_recompute_auto"):+ monkeypatch.setattr(rt, name, never(name))++ extra = {"stream_recompute": "on"} if spelled else {}+ tcfg = SimpleNamespace(+ quantization="none",+ double_quant_on=True,+ stream_source="auto",+ stream_buffers=2,+ stream_disk_kind=None,+ stream_pin=None,+ seed=7,+ moe_lora=False,+ batch_size=1,+ gradient_accumulation_steps=1,+ stream_vram_probe=False,+ stream_vram_override=None,+ stream_read_ahead=DEFAULT_STREAM_READ_AHEAD,+ lora=_config("sft").training.lora,+ **extra,+ )+ # No `task`, no `data.max_length`: nothing on the default path may need them.+ cfg = SimpleNamespace(base="Qwen/Qwen3-0.6B", data=SimpleNamespace())+ model_cfg = SimpleNamespace(+ model_type="qwen3", hidden_size=64, num_hidden_layers=2, vocab_size=128+ )+ # Only `stats`: no `close`, so any cleanup call would raise here.+ runtime = SimpleNamespace(+ stats=lambda: {+ "tier": "ram",+ "store_bytes": 0,+ "pinned": False,+ "n_layers": 2,+ "buffers": 2,+ "buffer_bytes": 8,+ }+ )+ built = SimpleNamespace()+ fakes = {+ "transformers.AutoTokenizer.from_pretrained": lambda *_a, **_k: _tokenizer(),+ "transformers.AutoConfig.from_pretrained": lambda *_a, **_k: model_cfg,+ "soup_cli.utils.layer_shard.resolve_shard_dir": lambda *_a, **_k: str(tmp_path / "s"),+ "soup_cli.utils.layer_shard.shard_checkpoint": lambda *_a, **_k: SimpleNamespace(+ n_layers=2, total_params=100, quant="none", quant_specs={}+ ),+ "soup_cli.utils.layer_shard.source_weight_bytes": lambda *_a, **_k: 1024,+ "soup_cli.utils.spectrum_scan.resolve_model_weights": (+ lambda *_a, **_k: str(tmp_path / "w")+ ),+ "soup_cli.utils.layer_stream.free_ram_bytes": lambda: 1_000_000,+ "soup_cli.utils.layer_stream_runtime.build_meta_skeleton": (+ lambda *_a, **_k: SimpleNamespace()+ ),+ "soup_cli.utils.layer_stream_runtime.RamSource.layer_specs_from_paths": (+ lambda *_a, **_k: [+ {"self_attn.q_proj.weight": ((4, 4), "float32")},+ {"self_attn.q_proj.weight": ((4, 4), "float32")},+ ]+ ),+ "soup_cli.utils.layer_stream_runtime.extras_resident_bytes": lambda *_a, **_k: 0,+ "soup_cli.utils.layer_stream_runtime.build_streamed_model": (+ lambda **_k: (built, runtime)+ ),+ "soup_cli.utils.moe.detect_moe_model": lambda *_a, **_k: False,+ "soup_cli.utils.layer_stream.render_stream_panel": lambda *_a, **_k: "panel",+ "soup_cli.utils.layer_stream_runtime.expandable_segments_status": lambda: (True, ""),+ }+ for target, fake in fakes.items():+ monkeypatch.setattr(target, fake)++ wrapper = _Wrapper()+ wrapper._setup_streaming_transformers(cfg, tcfg)+ assert wrapper.model is built+ assert wrapper._stream_runtime is runtimediff --git a/tests/test_stream_recompute_config.py b/tests/test_stream_recompute_config.pynew file mode 100644index 00000000..e9ce6bc1--- /dev/null+++ b/tests/test_stream_recompute_config.py@@ -0,0 +1,170 @@+"""Layer streaming: the ``training.stream_recompute`` config field.++'on' is the shipped checkpoint recompute; 'off' keeps every streamed layer's+activations; 'auto' keeps them for as many of the last layers as fit, decided at+setup from one synthetic step at the configured batch x max_length (RAM tier+only). The runtime and the setup wiring are in+``tests/test_stream_recompute.py``.+"""++import pytest++from soup_cli.config.loader import load_config_from_string+++def _cfg(**training):+ body = "\n".join(f" {k}: {v}" for k, v in training.items())+ return (+ "base: HuggingFaceTB/SmolLM2-135M\n"+ "task: sft\n"+ "data:\n"+ " train: d.jsonl\n"+ " max_length: 512\n"+ "training:\n"+ " batch_size: 1\n"+ " lora:\n"+ " r: 8\n" + body + "\n"+ )+++def _schema(task, **training):+ """A config built from Python, as tests/test_stream_loss_chunk.py does; kto+ refuses a batch of one before anything else is looked at."""+ from soup_cli.config.schema import SoupConfig++ data = {"train": "tests/fixtures/sample_train.jsonl"}+ if task != "sft":+ data["format"] = "dpo"+ return SoupConfig(+ base="test/model", task=task, data=data, training={"batch_size": 2, **training}+ )+++class TestAccepted:+ def test_default_is_on(self):+ cfg = load_config_from_string(_cfg(stream_layers="true"))+ assert cfg.training.stream_recompute == "on"++ def test_default_is_on_without_streaming(self):+ cfg = load_config_from_string(_cfg())+ assert cfg.training.stream_recompute == "on"++ @pytest.mark.parametrize("mode", ["on", "off", "auto"])+ def test_every_mode_parses_quoted(self, mode):+ cfg = load_config_from_string(_cfg(stream_layers="true", stream_recompute=f"'{mode}'"))+ assert cfg.training.stream_recompute == mode++ def test_auto_parses_unquoted(self):+ cfg = load_config_from_string(_cfg(stream_layers="true", stream_recompute="auto"))+ assert cfg.training.stream_recompute == "auto"++ @pytest.mark.parametrize(+ ("spelling", "mode"),+ [("on", "on"), ("off", "off"), ("true", "on"), ("false", "off"), ("no", "off")],+ )+ def test_yaml_booleans_map_to_the_mode_they_spell(self, spelling, mode):+ """YAML 1.1 reads an unquoted on/off (and yes/no/true/false) as a bool.+ For this field the bool reading agrees with the mode ('recompute: off'+ = no recompute), so it is mapped rather than refused."""+ cfg = load_config_from_string(_cfg(stream_layers="true", stream_recompute=spelling))+ assert cfg.training.stream_recompute == mode++ def test_the_explicit_default_is_accepted_without_streaming(self):+ """A default is not a decision: spelling it out must not trip the gate."""+ cfg = load_config_from_string(_cfg(stream_layers="false", stream_recompute="'on'"))+ assert cfg.training.stream_recompute == "on"+++class TestRefused:+ @pytest.mark.parametrize("mode", ["'off'", "'auto'", "off"])+ def test_refused_while_streaming_is_off(self, mode):+ with pytest.raises(Exception) as exc:+ load_config_from_string(_cfg(stream_layers="false", stream_recompute=mode))+ message = str(exc.value)+ assert "training.stream_recompute" in message+ assert "set but stream_layers is false" in message+ assert "set stream_layers=true" in message++ def test_refused_while_streaming_is_unset(self):+ with pytest.raises(Exception) as exc:+ load_config_from_string(_cfg(stream_recompute="auto"))+ assert "training.stream_recompute" in str(exc.value)+ assert "set but stream_layers is false" in str(exc.value)++ @pytest.mark.parametrize("value", ["1", "0", "2"])+ def test_an_int_is_refused(self, value):+ with pytest.raises(Exception) as exc:+ load_config_from_string(_cfg(stream_layers="true", stream_recompute=value))+ message = str(exc.value)+ assert "training.stream_recompute must be one of 'on', 'off', 'auto'" in message+ assert "got int" in message++ @pytest.mark.parametrize(("value", "kind"), [("[off]", "list"), ("1.5", "float")])+ def test_other_non_strings_are_refused(self, value, kind):+ with pytest.raises(Exception) as exc:+ load_config_from_string(_cfg(stream_layers="true", stream_recompute=value))+ message = str(exc.value)+ assert "training.stream_recompute must be one of 'on', 'off', 'auto'" in message+ assert f"got {kind}" in message++ @pytest.mark.parametrize("value", ["'maybe'", "'ON'", "'recompute'", "''"])+ def test_an_unknown_mode_is_refused(self, value):+ with pytest.raises(Exception) as exc:+ load_config_from_string(_cfg(stream_layers="true", stream_recompute=value))+ message = str(exc.value)+ assert "stream_recompute" in message+ assert "'on', 'off' or 'auto'" in message++ @pytest.mark.parametrize("task", ["dpo", "orpo", "simpo", "kto"])+ @pytest.mark.parametrize("mode", ["off", "auto"])+ def test_a_task_other_than_sft_is_refused(self, task, mode):+ """Only the SFT step is measured; a loss with two grad-enabled forwards+ per backward would stop at the slot-owner check mid-step."""+ from pydantic import ValidationError++ with pytest.raises(ValidationError) as exc:+ _schema(task, stream_layers=True, stream_recompute=mode)+ message = exc.value.errors()[0]["msg"]+ assert (+ f"training.stream_recompute='{mode}' supports task='sft' only; got task='{task}'"+ in message+ )+ assert "set it to 'on'" in message++ @pytest.mark.parametrize("task", ["dpo", "kto"])+ def test_on_is_accepted_for_a_task_other_than_sft(self, task):+ cfg = _schema(task, stream_layers=True, stream_recompute="on")+ assert cfg.training.stream_recompute == "on"++ @pytest.mark.parametrize("mode", ["off", "auto"])+ def test_sft_is_accepted(self, mode):+ cfg = _schema("sft", stream_layers=True, stream_recompute=mode)+ assert cfg.training.stream_recompute == mode++ @pytest.mark.parametrize("offload", ["cpu", "disk"])+ @pytest.mark.parametrize("mode", ["off", "auto"])+ def test_activation_offloading_is_refused(self, mode, offload):+ """A retained layer's own saved-tensor hooks are the innermost, so the+ offloading hooks would never see its activations."""+ from pydantic import ValidationError++ with pytest.raises(ValidationError) as exc:+ _schema(+ "sft", stream_layers=True, stream_recompute=mode, activation_offloading=offload+ )+ message = exc.value.errors()[0]["msg"]+ assert (+ f"training.stream_recompute='{mode}' cannot be combined with "+ f"training.activation_offloading='{offload}'" in message+ )+ assert "unset activation_offloading" in message++ def test_on_with_activation_offloading_is_accepted(self):+ cfg = _schema("sft", stream_layers=True, activation_offloading="cpu")+ assert cfg.training.stream_recompute == "on"+ assert cfg.training.activation_offloading == "cpu"++ def test_a_misspelt_key_is_refused(self):+ with pytest.raises(Exception) as exc:+ load_config_from_string(_cfg(stream_layers="true", stream_recomputes="'off'"))+ assert "stream_recomputes" in str(exc.value)