Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
bench: publish the stream_loss_chunk peak-VRAM harness and its sizing section (#2102)
Yang Bai63dbb8cc5065
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 9f6e45c1..e5c1732b 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -83,6 +83,7 @@ only in a scratchpad on a machine that is gone, which is| [`fsdp_sharding_probe.py`](harness/fsdp_sharding_probe.py) | STEP 20 / [#373](https://github.com/MakazhanAlpamys/Soup/issues/373) | Is the base actually **sharded**, or was the flag merely accepted? Per-rank `local == total / world_size` asserted, with the single-GPU control expecting the *full* count so a probe that always divides fails instead of passing everything. Repairs the STEP 20 probe, which raised `TypeError('FullyShardedDataParallel does not support len()')` on all eight ranks at 70B — a `torch.compile` wrapper, not FSDP | Accounting is pure-Python and CPU-tested; a real reading needs the multi-GPU box under test || [`mlx_sft_smoke.py`](harness/mlx_sft_smoke.py) | [#23](https://github.com/MakazhanAlpamys/Soup/issues/23) / [`run-m1-8gb-mlx-sft.md`](run-m1-8gb-mlx-sft.md) | Does `MLXSFTTrainerWrapper.train()` actually run against a real MLX runtime, and what does it peak at? Asserts MLX **dispatch before measuring**, so a transformers-path number cannot be published as an MLX one (#363). Generates its own fixture; prints host memory and swap either side | Apple Silicon + `pip install -e ".[mlx]"`; downloads the chosen model; 20 s at 0.5B, ~6 min at 8B including the Hub fetch || [`issue395_second_stack_vram.py`](harness/issue395_second_stack_vram.py) | #395 / [`gate-395-second-stack-vram.md`](gate-395-second-stack-vram.md) | Does the long-sequence under-prediction reproduce on a second stack, and is the gap the logits multiplier? Answer: no and no — the loss term measures 12.000000 (spread 0), and `14 x E` alone exceeds the whole real peak on every row, so the `+2` retained copy is absent. Three modes: sweep, `--measure`, `--sdpa` | 24 GB card (largest arm peaks 11.85 GB); downloads SmolLM2-135M and Qwen2.5-0.5B; needs `jinja2>=3.1.0`; a few minutes |+| [`stream_loss_chunk_vram.py`](harness/stream_loss_chunk_vram.py) | [#1693](https://github.com/MakazhanAlpamys/Soup/issues/1693) | Peak VRAM of one streamed sft step at 512 / 1024 / 2048 tokens with and without `training.stream_loss_chunk`, beside the pre-flight's own prediction for the same run and the free VRAM it compared against; a row that ran out of memory is a row, and one whose prediction is below its real peak (or that ran out of memory inside the predicted fit) under-predicts and exits 1. `--self-test` runs the same protocol on a tiny local Llama without reading a peak. **Not yet run on a GPU**: there is no record for it | CUDA GPU required; downloads the model given as `--base`; `--self-test` needs no GPU and no download, a few seconds || [`issue361_nf4_throughput.py`](harness/issue361_nf4_throughput.py) | [#361](https://github.com/MakazhanAlpamys/Soup/issues/361) / [`gate-v0.72.2-nf4.md`](gate-v0.72.2-nf4.md) | Re-run the v0.72.2 8B NF4 protocol on repaired code (batch 1, S=512, 50 steps after 10 warm-up, `PagedAdamW8bit`, same-session GEMM ceiling)? Answer: recorded as the post-#331 row in `gate-v0.72.2-nf4.md`: 208.6 tok/s at 1935 MHz, 59% of the same-session ceiling against 68% pre-repair; read it with its clock. | RTX 3050 Laptop 4 GB; Llama-3.1-8B weights; CPU-tested protocol guards only || [`stream_probe.py`](harness/stream_probe.py) | [`probe-rtx5070-what-bounds-streaming.md`](probe-rtx5070-what-bounds-streaming.md), [`gate-971-async-nvme-source.md`](gate-971-async-nvme-source.md) | What bounds the streamed step on THIS card? Builds the model through the shipped path, replicates the two shipped copy paths with CUDA events added (a source guard refuses to run if the shipped body drifts), and runs the ceiling / step / sweep / interleaved-ablation modes of the v0.73.0 probe, writing JSON after every point. `--tier disk` measures the ASYNC source (#971) at the depth `--read-ahead N` sets, and `--control-sync-source` swaps in the shipped synchronous `DiskSource` for a same-session control; every run prints and records the source class it actually built and refuses when that disagrees with the flag, so a control cannot be published as a measurement. `--no-pin` selects pageable host memory on EITHER tier. `--lazy-shard-handles` **was** the sharding workaround for the Windows mapping limit and is no longer needed: [#934](https://github.com/MakazhanAlpamys/Soup/pull/934) (merged 2026-09-14) bounds the shipped sharder's live mappings, and its gate re-sharded the 138 GB 70B-shaped fixture with no workaround at all — 80/80 layers in 221.4 s against 216.9 s with the flag, and 83 output files / 36.39 GB byte-identical to the workaround's cache, `index.json` included. The flag is kept so the 2026-09-12 record can be re-run as it was measured. Since #974 every JSON's `meta` also records `soup_cli_file`, the source tree the run actually imported — an arm that claims to be "the old code" is only evidence if the JSON says where it came from — and `--skip-events` runs only the uninstrumented `step_plain` point (halves a cold 70B block; not used by the #974 record, whose blocks run both points as gate-971 did). Since [`probe-rtx5070-batch-control.md`](probe-rtx5070-batch-control.md) every timed step records its wall-clock start, an out-of-memory error inside a block is written as an `oom` point and the run exits 3 instead of losing the arm, and every step point carries a SHA-256 per row of its token ids and the allocator's counters | CUDA GPU; model weights; a 7B NF4 run is ~10 min all modes; the 70B-shaped cold-disk run is 10-60 min || [`batch_control_probe.py`](harness/batch_control_probe.py) | [`probe-rtx5070-batch-control.md`](probe-rtx5070-batch-control.md) §1-§4 | The batch control's driver: one `stream_probe.py` process per arm (SINGLE and STRIPED at batch 1, 2 and 3, two rounds, the second reversed, after a build arm and a 300 s settle), importing the two-drive gate's stamps, pre-arm checks, sampler and void rule unchanged. Runs the read-free floor (`--ablate --arms B_nocopy`) after every SINGLE arm's step block, requires an empty card before each arm, voids an arm another process read 1 GB during, and treats an out-of-memory error, an arm past a 600 s cap or a spill as a result that skips the larger batches. `--summarize` applies the rule from the committed files. No torch import | Windows only; a CUDA box holding both caches; about 25 min, 29 with a re-shard |diff --git a/benchmarks/harness/stream_loss_chunk_vram.py b/benchmarks/harness/stream_loss_chunk_vram.pynew file mode 100644index 00000000..98712d08--- /dev/null+++ b/benchmarks/harness/stream_loss_chunk_vram.py@@ -0,0 +1,607 @@+#!/usr/bin/env python3+"""Peak VRAM of a streamed sft step with and without ``training.stream_loss_chunk``.++The measurement #1693 asks for: peak VRAM at 512 / 1024 / 2048 tokens, with and+without the option, same model and stack, beside the pre-flight's prediction for+the same run, so that "the pre-flight does not under-predict on those rows" can+be checked rather than assumed.++``--self-test`` runs the same protocol on a tiny local Llama and never reads a+peak: it checks the plumbing (the arms, the realised length, the hook, the+pre-flight's inputs and arithmetic, the verdicts), not memory. It runs on the+device ``TrainingArguments`` picks by itself, so CPU, or MPS on Apple Silicon.++Requirements+------------+- CUDA GPU for the measurement. The unchunked arm is expected to run out of+ memory at the longer lengths on a small card; that is a row, not a crash.+- The base model, by Hub id or local path (``--base``).+- ``--self-test`` needs neither: no GPU, no download, a few seconds.++Usage+-----+ python benchmarks/harness/stream_loss_chunk_vram.py --base <id-or-path>+ python benchmarks/harness/stream_loss_chunk_vram.py --base <id-or-path> \\+ --quantization 4bit --seqs 512,1024,2048 --chunk 128+ python benchmarks/harness/stream_loss_chunk_vram.py --self-test++Protocol+--------+The real ``soup train`` setup (``SFTTrainerWrapper.setup``) and one training+step per arm, batch 1, LoRA r=8 on ``q_proj`` / ``v_proj``, ``stream_buffers: 2``,+the rewind log on (its default). Each arm is built from scratch in the same+process, the off arm first at each length.++- "real peak" is ``torch.cuda.max_memory_allocated()`` over the step, after+ ``reset_peak_memory_stats()``.+- "predicted" is the number the pre-flight itself computed during that setup,+ read from ``estimate_stream_peak_vram`` as it was called, so the two columns+ are about the same run.+- "free" is the ``torch.cuda.mem_get_info()`` reading the pre-flight compared+ against. ``stream_vram_override`` is set high so the pre-flight never refuses+ an arm this harness is trying to measure; "free" says what it would have done.++As in ``issue395_second_stack_vram.py``, ``data.max_length`` truncates, it does+not pad, so the rows overshoot 3x and the realised length is read off the+collated batch. A row whose realised length is not the requested one is void.++Verdicts and exit status+------------------------+- ``ok``: predicted >= real peak.+- ``UNDER-PREDICTS``: predicted < real peak, or the arm ran out of memory+ although the prediction fit in "free" (the pre-flight would have let it run).+- ``OOM, refused``: the arm ran out of memory and the pre-flight would have+ refused it. Expected for the unchunked arm at long lengths on a small card.+- ``VOID``: the realised length differs from the requested one.++Exit 0 when every row is ``ok`` or ``OOM, refused``; 1 when a row+under-predicts; 3 when a row is void; 2 for bad arguments or no CUDA device.+``--self-test`` exits 0 when every check passes and 1 otherwise.++Reading the table+-----------------+``pred/real`` below 1.0 is the direction the pre-flight must not err in. The+chunked rows are the ones nobody has measured: their logits term is the shipped+14 bytes per element charged on ``min(seq, chunk)`` tokens, plus two+``[batch, seq, hidden]`` tensors (``estimate_loss_chunk_bytes``).+"""++from __future__ import annotations++import argparse+import contextlib+import gc+import inspect+import json+import math+import os+import pathlib+import sys+import tempfile+from dataclasses import dataclass+from typing import Any, Iterator, Optional++import yaml++#: Bytes the pre-flight is told are free, so that it measures instead of refusing.+ASSUME_FREE_BYTES = 1 << 40++#: Rows per arm; all identical, so the step does not depend on the shuffle.+ROWS_PER_ARM = 4++#: Overshoot factor of each row over the requested length; see the docstring.+OVERSHOOT = 3++VERDICT_OK = "ok"+VERDICT_UNDER = "UNDER-PREDICTS"+VERDICT_REFUSED = "OOM, refused"+VERDICT_VOID = "VOID"+VERDICT_NOT_MEASURED = "no peak read"++EXIT_OK = 0+EXIT_UNDER = 1+EXIT_USAGE = 2+EXIT_VOID = 3+++@dataclass(frozen=True)+class Arm:+ """One arm of one length: what the pre-flight predicted and what happened."""++ seq: int+ chunk: Optional[int] # None is the unchunked arm+ realised: int+ predicted: int+ preflight: dict # the keyword arguments the pre-flight's estimate was called with+ free: Optional[int] # the free VRAM the pre-flight compared against; None off CUDA+ peak: Optional[int] # None when not measured (off CUDA) or out of memory+ oom: bool+ hooked: bool # the chunked head and loss was on the trainer+ loss: Optional[float] # the step's training loss; None when it ran out of memory+++def rows(seq: int) -> list[dict]:+ """``ROWS_PER_ARM`` sft rows that tokenize past ``seq``, so truncation lands on it."""+ body = "word " * (seq * OVERSHOOT)+ return [+ {"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": body}]}+ for _ in range(ROWS_PER_ARM)+ ]+++def config_doc(+ base: str, seq: int, chunk: Optional[int], quantization: str, out_dir: pathlib.Path+) -> dict:+ """One arm's ``soup.yaml`` as a dict; ``chunk=None`` is the unchunked arm."""+ training: dict[str, Any] = {+ "batch_size": 1,+ "gradient_accumulation_steps": 1,+ "epochs": 1,+ "quantization": quantization,+ "stream_layers": True,+ "stream_buffers": 2,+ "stream_vram_override": ASSUME_FREE_BYTES,+ "logging_steps": 1,+ "save_steps": 100000,+ "lora": {"r": 8, "alpha": 16, "target_modules": ["q_proj", "v_proj"]},+ }+ if chunk is not None:+ training["stream_loss_chunk"] = chunk+ return {+ "base": base,+ "task": "sft",+ "backend": "transformers",+ "modality": "text",+ "data": {+ "train": str(out_dir / "d.jsonl"),+ "max_length": seq,+ "chat_template": "chatml",+ },+ "output": str(out_dir / "out"),+ "training": training,+ }+++def verdict(arm: Arm) -> str:+ """The row's verdict; see the module docstring."""+ if arm.realised != arm.seq:+ return VERDICT_VOID+ if arm.oom:+ if arm.free is not None and arm.predicted <= arm.free:+ return VERDICT_UNDER+ return VERDICT_REFUSED+ if arm.peak is None:+ return VERDICT_NOT_MEASURED+ return VERDICT_OK if arm.predicted >= arm.peak else VERDICT_UNDER+++def exit_status(arms: list[Arm]) -> int:+ """``EXIT_VOID`` beats ``EXIT_UNDER``: a void row says nothing either way."""+ verdicts = [verdict(arm) for arm in arms]+ if VERDICT_VOID in verdicts:+ return EXIT_VOID+ if VERDICT_UNDER in verdicts:+ return EXIT_UNDER+ return EXIT_OK+++HEADER = (+ f"{'seq':>6} {'realised':>8} {'chunk':>6} | {'predicted':>11} {'free':>11} "+ f"{'real peak':>11} {'pred/real':>9} | verdict"+)+++def _gb(value: Optional[int]) -> str:+ return "-" if value is None else f"{value / 1e9:.3f}GB"+++def table_row(arm: Arm) -> str:+ """One printed line of the table."""+ if arm.oom:+ peak, ratio = "OOM", "-"+ elif arm.peak is None:+ peak, ratio = "-", "-"+ else:+ peak, ratio = _gb(arm.peak), f"{arm.predicted / arm.peak:.4f}"+ chunk = "off" if arm.chunk is None else str(arm.chunk)+ return (+ f"{arm.seq:>6} {arm.realised:>8} {chunk:>6} | {_gb(arm.predicted):>11} "+ f"{_gb(arm.free):>11} {peak:>11} {ratio:>9} | {verdict(arm)}"+ )+++def recording_preflight() -> Iterator[dict]:+ """Record what the pre-flight computes, without changing it.++ ``stream_setup`` imports both functions from ``layer_stream`` when the+ pre-flight runs, so replacing the module attributes for the duration of one+ setup is enough; they are put back however the setup ends.+ """+ from soup_cli.utils import layer_stream++ estimate = layer_stream.estimate_stream_peak_vram+ resolve = layer_stream.resolve_available_vram_bytes+ seen: dict = {}++ def recording_estimate(**kwargs):+ seen["preflight"] = dict(kwargs)+ seen["predicted"] = estimate(**kwargs)+ return seen["predicted"]++ def recording_resolve(**kwargs):+ seen["free"] = kwargs["measured_bytes"]+ return resolve(**kwargs)++ layer_stream.estimate_stream_peak_vram = recording_estimate+ layer_stream.resolve_available_vram_bytes = recording_resolve+ try:+ yield seen+ finally:+ layer_stream.estimate_stream_peak_vram = estimate+ layer_stream.resolve_available_vram_bytes = resolve+++def run_one(+ base: str,+ seq: int,+ chunk: Optional[int],+ quantization: str,+ device: str,+ work_dir: pathlib.Path,+ measure: bool = True,+) -> Arm:+ """Build one arm through the real setup and run one training step.++ The peak is read only when ``measure`` is set and ``device`` is CUDA.+ """+ import torch++ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.sft import SFTTrainerWrapper+ from soup_cli.utils.stream_loss_chunk import HOOK_ATTR++ on_cuda = device == "cuda"+ measuring = measure and on_cuda+ data = rows(seq)+ work_dir.mkdir(parents=True, exist_ok=True)+ (work_dir / "d.jsonl").write_text("\n".join(json.dumps(r) for r in data), encoding="utf-8")+ (work_dir / "out").mkdir(exist_ok=True)++ gc.collect()+ if on_cuda:+ torch.cuda.empty_cache()+ wrapper = None+ try:+ with recording_preflight() as seen:+ cfg = load_config_from_string(+ yaml.safe_dump(config_doc(base, seq, chunk, quantization, work_dir))+ )+ wrapper = SFTTrainerWrapper(cfg, device=device)+ wrapper.setup({"train": data})+ if "predicted" not in seen:+ raise RuntimeError(+ "the pre-flight did not run (it could not read the model's shape), "+ "so there is no prediction to compare against"+ )+ trainer = wrapper.trainer+ hooked = getattr(trainer, HOOK_ATTR, None) is not None+ batch = next(iter(trainer.get_train_dataloader()))+ realised = int(batch["input_ids"].shape[-1])+ trainer.args.max_steps = 1+ peak: Optional[int] = None+ loss: Optional[float] = None+ oom = False+ if measuring:+ torch.cuda.empty_cache()+ torch.cuda.reset_peak_memory_stats()+ try:+ loss = float(trainer.train().training_loss)+ peak = int(torch.cuda.max_memory_allocated())+ except torch.cuda.OutOfMemoryError:+ oom = True+ else:+ loss = float(trainer.train().training_loss)+ finally:+ if wrapper is not None:+ # `wrapper.train()` does both on its way out; this harness calls the+ # trainer directly, as issue395_second_stack_vram.py does.+ for teardown in (wrapper._remove_stream_loss_chunk_hook, wrapper._close_stream_runtime):+ try:+ teardown()+ except Exception: # noqa: BLE001 - teardown must not mask the measurement+ pass+ del wrapper+ gc.collect()+ if on_cuda:+ torch.cuda.empty_cache()+ return Arm(+ seq=seq,+ chunk=chunk,+ realised=realised,+ predicted=int(seen["predicted"]),+ preflight=seen["preflight"],+ free=seen.get("free"),+ peak=peak,+ oom=oom,+ hooked=hooked,+ loss=loss,+ )+++def sweep(base: str, seqs: list[int], chunk: int, quantization: str) -> int:+ import torch++ import soup_cli++ print(f"# soup_cli {soup_cli.__version__} from {soup_cli.__file__}")+ print(+ f"# torch {torch.__version__} (CUDA {torch.version.cuda}) | "+ f"{torch.cuda.get_device_name(0)} | "+ f"{torch.cuda.get_device_properties(0).total_memory / 1e9:.3f} GB"+ )+ print(f"# base {base} | quantization {quantization} | chunk {chunk}")+ print(HEADER)+ arms = []+ with tempfile.TemporaryDirectory(prefix="stream_loss_chunk_vram_") as tmp:+ for seq in seqs:+ for arm_chunk in (None, chunk):+ work_dir = pathlib.Path(tmp) / f"s{seq}-{arm_chunk or 'off'}"+ arm = run_one(base, seq, arm_chunk, quantization, "cuda", work_dir)+ print(table_row(arm), flush=True)+ arms.append(arm)+ return exit_status(arms)+++# --- --self-test: the same protocol on a tiny local Llama, no peak read ----++#: Small enough to run in seconds, with a vocabulary large enough that the+#: logits term the option removes dominates the two hidden-sized tensors it adds.+_TINY = {"vocab": 512, "hidden": 64, "layers": 2}+_SELF_TEST_SEQ = 64+++def _tiny_llama(directory: pathlib.Path) -> str:+ """A real (tiny, untied) Llama checkpoint and an offline tokenizer on disk."""+ import torch+ from safetensors.torch import save_file+ from tokenizers import Tokenizer, models, pre_tokenizers+ from transformers import LlamaConfig, LlamaForCausalLM++ torch.manual_seed(7)+ config = LlamaConfig(+ vocab_size=_TINY["vocab"],+ hidden_size=_TINY["hidden"],+ intermediate_size=_TINY["hidden"] * 2,+ num_hidden_layers=_TINY["layers"],+ num_attention_heads=4,+ num_key_value_heads=2,+ tie_word_embeddings=False,+ max_position_embeddings=4 * _SELF_TEST_SEQ,+ )+ model = LlamaForCausalLM(config).to(torch.float32)+ directory.mkdir(parents=True, exist_ok=True)+ state = {name: value.contiguous() for name, value in model.state_dict().items()}+ save_file(state, str(directory / "model.safetensors"))+ config.save_pretrained(str(directory))++ vocab = {"<unk>": 0, "<s>": 1, "</s>": 2, "<pad>": 3, "hi": 4, "word": 5}+ tokenizer = Tokenizer(models.WordLevel(vocab=vocab, unk_token="<unk>"))+ tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()+ tokenizer.save(str(directory / "tokenizer.json"))+ (directory / "tokenizer_config.json").write_text(+ json.dumps(+ {+ "tokenizer_class": "PreTrainedTokenizerFast",+ "unk_token": "<unk>",+ "bos_token": "<s>",+ "eos_token": "</s>",+ "pad_token": "<pad>",+ "model_max_length": 4 * _SELF_TEST_SEQ,+ "clean_up_tokenization_spaces": False,+ }+ ),+ encoding="utf-8",+ )+ return str(directory)+++def trainer_device() -> str:+ """The device ``TrainingArguments`` picks by itself: cuda, else mps, else cpu.++ A streamed model built anywhere else fails its first step on a device+ mismatch, so the self-test builds there too.+ """+ import torch++ if torch.cuda.is_available():+ return "cuda"+ mps = getattr(torch.backends, "mps", None)+ return "mps" if mps is not None and mps.is_available() else "cpu"+++def _environment(**values: str) -> Iterator[None]:+ before = {name: os.environ.get(name) for name in values}+ os.environ.update(values)+ try:+ yield+ finally:+ for name, value in before.items():+ if value is None:+ os.environ.pop(name, None)+ else:+ os.environ[name] = value+++def _self_test_checks(off: Arm, chunked: Arm, chunk: int) -> list[tuple[str, bool]]:+ from soup_cli.utils.layer_stream import (+ estimate_logits_bytes,+ estimate_loss_chunk_bytes,+ estimate_stream_peak_vram,+ )++ seq = off.seq+ kwargs = off.preflight+ # The pre-flight does not pass a dtype; the estimate then charges its default.+ dtype = kwargs.get(+ "dtype", inspect.signature(estimate_stream_peak_vram).parameters["dtype"].default+ )+ full = estimate_logits_bytes(+ vocab_size=kwargs["vocab_size"],+ seq_len=seq,+ batch_size=kwargs["batch_size"],+ bytes_per_element=kwargs["logits_bytes_per_element"],+ )+ one_chunk = estimate_logits_bytes(+ vocab_size=kwargs["vocab_size"],+ seq_len=min(seq, chunk),+ batch_size=kwargs["batch_size"],+ bytes_per_element=kwargs["logits_bytes_per_element"],+ )+ kept = estimate_loss_chunk_bytes(+ hidden_size=kwargs["hidden_size"],+ seq_len=seq,+ batch_size=kwargs["batch_size"],+ dtype=dtype,+ )+ same_loss = (+ off.loss is not None+ and chunked.loss is not None+ and math.isfinite(off.loss)+ and abs(chunked.loss - off.loss) <= 1e-5 * abs(off.loss)+ )+ rest_equal = {k: v for k, v in off.preflight.items() if k != "loss_chunk"} == {+ k: v for k, v in chunked.preflight.items() if k != "loss_chunk"+ }++ def synthetic(**fields: Any) -> Arm:+ values = dict(off.__dict__)+ values.update(fields)+ return Arm(**values)++ return [+ ("both arms realise the requested length", off.realised == chunked.realised == seq),+ ("the pre-flight saw seq_len == the requested length", kwargs["seq_len"] == seq),+ (+ "the pre-flight saw loss_chunk None / the chunk",+ off.preflight["loss_chunk"] is None and chunked.preflight["loss_chunk"] == chunk,+ ),+ ("the arms' pre-flight inputs differ only in loss_chunk", rest_equal),+ ("the chunked head and loss is on the chunked arm only", chunked.hooked and not off.hooked),+ (+ "predicted(off) - predicted(chunk) == full logits - one chunk - two hidden tensors",+ off.predicted - chunked.predicted == full - one_chunk - kept,+ ),+ ("the step losses agree within 1e-5 (LoRA B starts at zero)", same_loss),+ ("no peak is read and no row claims one", off.peak is None and not off.oom),+ (+ "a measured row at or above its peak is ok",+ verdict(synthetic(peak=off.predicted)) == VERDICT_OK,+ ),+ (+ "a row below its peak under-predicts and exits 1",+ exit_status([synthetic(peak=off.predicted + 1)]) == EXIT_UNDER,+ ),+ (+ "an OOM the pre-flight would refuse is a row, exit 0",+ exit_status([synthetic(oom=True, free=off.predicted - 1)]) == EXIT_OK+ and "OOM" in table_row(synthetic(oom=True, free=off.predicted - 1)),+ ),+ (+ "an OOM inside the predicted fit under-predicts",+ verdict(synthetic(oom=True, free=off.predicted)) == VERDICT_UNDER,+ ),+ (+ "a row at another length is void and exits 3",+ exit_status([synthetic(realised=seq - 1), synthetic(peak=off.predicted + 1)])+ == EXIT_VOID,+ ),+ ]+++def run_self_test(chunk: int = 16) -> int:+ """The protocol end to end without a peak: both arms, one length, every check printed."""+ with tempfile.TemporaryDirectory(prefix="stream_loss_chunk_vram_self_test_") as tmp:+ root = pathlib.Path(tmp)+ base = _tiny_llama(root / "model")+ device = trainer_device()+ with _environment(SOUP_LAYER_STREAM_CACHE_DIR=str(root / "cache")):+ arms = [+ run_one(+ base,+ _SELF_TEST_SEQ,+ arm,+ "none",+ device,+ root / f"arm-{arm or 'off'}",+ measure=False,+ )+ for arm in (None, chunk)+ ]+ print(f"# self-test on {device}; no peak is read")+ print(HEADER)+ for arm in arms:+ print(table_row(arm))+ for arm in arms:+ name = "off" if arm.chunk is None else f"chunk {arm.chunk}"+ print(f"# {name}: predicted {arm.predicted} bytes, step loss {arm.loss!r}")+ checks = _self_test_checks(arms[0], arms[1], chunk)+ for name, passed in checks:+ print(f"{'PASS' if passed else 'FAIL'} {name}")+ failed = sum(not passed for _, passed in checks)+ print(f"self-test: {len(checks) - failed} passed, {failed} failed (no peak measured)")+ return EXIT_OK if failed == 0 else EXIT_UNDER+++def parse_args(argv: Optional[list[str]] = None) -> argparse.Namespace:+ parser = argparse.ArgumentParser(+ description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter+ )+ parser.add_argument("--base", help="model id or local path (required unless --self-test)")+ parser.add_argument("--seqs", default="512,1024,2048", help="comma-separated token counts")+ parser.add_argument("--chunk", type=int, default=128, help="training.stream_loss_chunk")+ parser.add_argument("--quantization", default="none", choices=["none", "4bit"])+ parser.add_argument(+ "--self-test",+ action="store_true",+ help="run the protocol on a tiny local Llama, reading no peak (no GPU, no download)",+ )+ return parser.parse_args(argv)+++def main(argv: Optional[list[str]] = None) -> int:+ from soup_cli.utils.config_bounds import MAX_STREAM_LOSS_CHUNK, MIN_STREAM_LOSS_CHUNK++ args = parse_args(argv)+ if args.self_test:+ return run_self_test()+ if not args.base:+ print("ERROR: --base is required unless --self-test is used")+ return EXIT_USAGE+ if not MIN_STREAM_LOSS_CHUNK <= args.chunk <= MAX_STREAM_LOSS_CHUNK:+ print(+ f"ERROR: --chunk must be between {MIN_STREAM_LOSS_CHUNK} and "+ f"{MAX_STREAM_LOSS_CHUNK}, the bounds of training.stream_loss_chunk"+ )+ return EXIT_USAGE+ try:+ seqs = [int(part) for part in args.seqs.split(",")]+ except ValueError:+ print(f"ERROR: --seqs must be comma-separated integers; got {args.seqs!r}")+ return EXIT_USAGE+ if not seqs or any(seq <= 0 for seq in seqs):+ print("ERROR: every --seqs entry must be positive")+ return EXIT_USAGE++ import torch++ if not torch.cuda.is_available():+ print("ERROR: CUDA required: every figure in this table is a device peak (try --self-test)")+ return EXIT_USAGE+ base = os.path.abspath(args.base) if os.path.exists(args.base) else args.base+ return sweep(base, seqs, args.chunk, args.quantization)+++if __name__ == "__main__":+ sys.exit(main())diff --git a/docs/performance-and-quantization.md b/docs/performance-and-quantization.mdindex 48171ec5..0f94a0a9 100644--- a/docs/performance-and-quantization.md+++ b/docs/performance-and-quantization.md@@ -723,6 +723,8 @@ The same term decides what an 8B base needs on a 4 GB card. Llama-3.1-8B NF4 (vofurther 512 tokens per step** — 4.512 GB at 1024, already above the 4.294 GB card([probe v0.73.0, section 9](../benchmarks/probe-v0.73.0-what-bounds-streaming.md#9-peak-vram-and-the-microbatch-sweep)).`data.max_length` defaults to 2048, so the "8B on 4 GB" setup needs `data.max_length: 512`.+`training.stream_loss_chunk` (below) is built to stop the logits term growing with the length; no chunked+run of that setup has been measured.So `soup train` predicts peak VRAM before building the model, and **refuses a run itexpects not to fit**:@@ -752,6 +754,73 @@ because this card alone produced 3.5 and 7.6 TFLOPS in two sessions at the sameclock. A per-card constant compiled into Soup would be a fabrication. Real streamed runs landed at68–100% of their measured ceiling.+**Long contexts: the head and the loss in chunks (`training.stream_loss_chunk`, #1693).**+The logits term is `vocab × seq × batch`, so it is what caps the context length on a small+card. `training.stream_loss_chunk: 128` runs the output head and the loss 128 tokens at a+time: each chunk runs the head, takes its cross-entropy and the gradient on its own slice of+the hidden states, and drops its logits before the next chunk starts. The logits term the+pre-flight charges becomes `vocab × min(seq, chunk) × batch`. By the pre-flight's own+arithmetic (the shipped 14 bytes per element, the 128 256-token vocabulary above, batch 1)+that is an estimated 0.23 GB at a chunk of 128 whatever `data.max_length` is, against+0.92 GB at 512 tokens, 1.84 GB at 1024 and 3.68 GB at 2048 unchunked. These are estimates,+not measurements: no run with the option has been measured on a GPU (see the end of this+section). The option takes an explicit number of tokens from 16 to 8192; there is no default+and no `true`, and unset is off.++**What it costs.** Two `[batch, seq, hidden]` tensors in the stream dtype, which the+pre-flight adds to its estimate: the gradient on the hidden states, filled chunk by chunk in+the forward and handed to the backward, and its product with the incoming gradient. At+2048 tokens, hidden 4096 and bf16 that charge is 34 MB for the two. They are not live at the+same time as a chunk's logits, so charging both beside the logits term errs high. And a loop+over the head, one matmul per chunk instead of one over the sequence; what that does to speed+has not been measured.++**What it gives up.**++- `outputs.logits` on a labelled forward: the output carries the loss and no logits. The+ trainer's `entropy`, `mean_token_accuracy` and `num_tokens` columns and the rewind log's+ per-row losses (`rewind.jsonl`) are written as before, under the same names, from what the+ chunk loop collects. A forward without labels (generation, `predict`) is unchanged; a+ labelled forward without gradients (the eval loss) is chunked too.+- Equality to the last bit with an unchunked step. The per-token terms are the ones+ `ForCausalLMLoss` computes; the order of the final sum and the shape of each matmul are+ not. `tests/test_stream_loss_chunk.py` holds the loss, the gradients, the logged columns+ and the rewind log's losses to a relative `1e-5` of the same run unchunked, on a tiny+ Llama. The bit-exactness the streamed path is published with does not extend to a run with+ this option.+- A trainable output head: the gradient is taken on the hidden states only.++**The pre-flight** charges the logits on `min(seq, chunk)` tokens at the shipped bytes per+element, plus the two tensors above; the panel line ends+`(logits 0.23 GB in chunks of 128 tokens)` at batch 1 with the vocabulary above. That charge+is arithmetic. Of the 14 bytes per element, 12 are the loss arithmetic measured on+unchunked steps and 2 are charged for a retained copy of the logits whose cause is not+identified (#327); whether a chunked step still pays those 2 is not known. The hook is on the model before `stream_vram_probe` runs, so with+`stream_vram_probe: true` the step that decides the fit is the chunked one.++**Refused.** At config load, as listed under "Rejected at config load" below: without+`stream_layers`, for a task other than `sft`, with `data.format: raft`, and beside+`use_cut_ce`. When the model is built, with the reason: a trainable output head, a model+without one, and a model that changes its logits after the head: Gemma 2's+`final_logit_softcapping` is read from its config, and any other such model is caught at its+first labelled forward.++**Not measured yet.** #1693 asks for peak VRAM at 512, 1024 and 2048 tokens with and without+the option, on one model and stack, and for a pre-flight that does not under-predict on those+rows. That table needs a CUDA card and is left for whoever has one. To produce it:++```bash+python benchmarks/harness/stream_loss_chunk_vram.py --base <id-or-path> --quantization 4bit+```++The harness runs the setup `soup train` uses and one step per arm at batch 1 (`--seqs` and+`--chunk` change the lengths and the chunk, 512,1024,2048 and 128 by default) and prints one+row per length and arm: the pre-flight's prediction for that run, the free VRAM it compared+against, and `torch.cuda.max_memory_allocated()` over the step. An arm that runs out of+memory is a row, not a crash. It exits 1 when a row under-predicts, including an arm that ran+out of memory inside the predicted fit. `--self-test` runs the same protocol on a tiny local+Llama with no GPU and no download, and reads no peak.+### Batch size vs gradient accumulationBoth work from v0.72.3, and they are not interchangeable. Measured on Qwen2.5-0.5B bf16,diff --git a/tests/test_stream_loss_chunk_vram_harness.py b/tests/test_stream_loss_chunk_vram_harness.pynew file mode 100644index 00000000..6329a121--- /dev/null+++ b/tests/test_stream_loss_chunk_vram_harness.py@@ -0,0 +1,195 @@+"""benchmarks/harness/stream_loss_chunk_vram.py, the #1693 peak-VRAM harness, without a GPU.++The measurement needs a CUDA card. What can be pinned here: the self-test (the real+setup and one step per arm on a tiny local Llama, no peak read), the configs of both+arms, the arguments the harness refuses, and the verdicts and exit status a row gets.+"""++from __future__ import annotations++import importlib.util+import sys+from pathlib import Path++import pytest++pytest.importorskip("torch")+pytest.importorskip("transformers")++_HARNESS = (+ Path(__file__).resolve().parents[1] / "benchmarks" / "harness" / "stream_loss_chunk_vram.py"+)+++def _load_harness():+ spec = importlib.util.spec_from_file_location("stream_loss_chunk_vram", _HARNESS)+ assert spec is not None and spec.loader is not None+ module = importlib.util.module_from_spec(spec)+ sys.modules[spec.name] = module+ spec.loader.exec_module(module)+ return module+++harness = _load_harness()+++def _arm(**fields):+ values = dict(+ seq=1024,+ chunk=128,+ realised=1024,+ predicted=3_500_000_000,+ preflight={},+ free=4_000_000_000,+ peak=3_463_000_000,+ oom=False,+ hooked=True,+ loss=1.0,+ )+ values.update(fields)+ return harness.Arm(**values)+++def test_the_self_test_passes(capsys):+ assert harness.run_self_test() == harness.EXIT_OK++ out = capsys.readouterr().out+ assert "self-test: 13 passed, 0 failed" in out # ansi-ok: print() output, no Rich console+ assert "FAIL" not in out+++def test_each_arm_is_torn_down(monkeypatch, capsys):+ """On a card a hook or streamed runtime left behind would sit in the next arm's peak."""+ from soup_cli.trainer.sft import SFTTrainerWrapper++ calls = {"hook": 0, "runtime": 0}+ remove_hook = SFTTrainerWrapper._remove_stream_loss_chunk_hook+ close_runtime = SFTTrainerWrapper._close_stream_runtime++ def counting_remove_hook(self):+ calls["hook"] += 1+ return remove_hook(self)++ def counting_close_runtime(self):+ calls["runtime"] += 1+ return close_runtime(self)++ monkeypatch.setattr(SFTTrainerWrapper, "_remove_stream_loss_chunk_hook", counting_remove_hook)+ monkeypatch.setattr(SFTTrainerWrapper, "_close_stream_runtime", counting_close_runtime)++ assert harness.run_self_test() == harness.EXIT_OK+ capsys.readouterr()+ assert calls == {"hook": 2, "runtime": 2}, "both teardowns, once per arm"++[email protected]("chunk", [None, 128])[email protected]("quantization", ["none", "4bit"])+def test_both_arms_are_configs_that_load(tmp_path, chunk, quantization):+ import yaml++ from soup_cli.config.loader import load_config_from_string++ doc = harness.config_doc("test/model", 1024, chunk, quantization, tmp_path)+ cfg = load_config_from_string(yaml.safe_dump(doc))++ assert cfg.task == "sft"+ assert cfg.training.stream_layers is True+ assert cfg.training.stream_loss_chunk == chunk+ assert cfg.data.max_length == 1024 and cfg.training.quantization == quantization+ assert cfg.training.batch_size == 1 and cfg.training.stream_buffers == 2+ assert cfg.training.rewind_log is True # the default trainer, as a user runs it+++def test_the_rows_overshoot_the_length_they_are_cut_to():+ rows = harness.rows(512)++ assert len(rows) == harness.ROWS_PER_ARM+ assert len(rows[0]["messages"][1]["content"].split()) == harness.OVERSHOOT * 512+++def test_a_row_at_or_above_its_peak_is_ok():+ row = harness.table_row(_arm())++ assert "1.0107" in row and row.endswith("| ok")+ assert harness.exit_status([_arm()]) == harness.EXIT_OK+++def test_a_row_below_its_peak_under_predicts_and_exits_1():+ short = _arm(predicted=3_400_000_000)++ assert "0.9818" in harness.table_row(short)+ assert harness.verdict(short) == harness.VERDICT_UNDER+ assert harness.exit_status([_arm(), short]) == harness.EXIT_UNDER+++def test_an_out_of_memory_arm_the_preflight_would_refuse_is_a_row_not_a_failure():+ refused = _arm(+ chunk=None, seq=2048, realised=2048, predicted=6_000_000_000, peak=None, oom=True+ )++ assert harness.table_row(refused).split() == [+ "2048", "2048", "off", "|", "6.000GB", "4.000GB", "OOM", "-", "|", "OOM,", "refused",+ ]+ assert harness.exit_status([refused]) == harness.EXIT_OK+++def test_an_out_of_memory_arm_inside_the_predicted_fit_under_predicts():+ inside = _arm(predicted=3_900_000_000, peak=None, oom=True)++ assert harness.verdict(inside) == harness.VERDICT_UNDER+ assert harness.exit_status([inside]) == harness.EXIT_UNDER+++def test_a_row_at_another_length_is_void_and_outranks_an_under_prediction():+ void = _arm(realised=1000)++ assert harness.verdict(void) == harness.VERDICT_VOID+ assert harness.exit_status([void, _arm(predicted=1)]) == harness.EXIT_VOID+++def test_the_free_vram_recorded_is_the_measured_reading_not_the_override():+ from soup_cli.utils import layer_stream++ with harness.recording_preflight() as seen:+ # What the pre-flight calls on CUDA, with the harness's override set.+ available = layer_stream.resolve_available_vram_bytes(+ measured_bytes=4_000_000_000, override_bytes=harness.ASSUME_FREE_BYTES+ )++ assert available == harness.ASSUME_FREE_BYTES # the pre-flight itself is unchanged+ assert seen["free"] == 4_000_000_000+++def test_the_preflight_functions_are_put_back_when_the_setup_raises():+ from soup_cli.utils import layer_stream++ estimate = layer_stream.estimate_stream_peak_vram+ resolve = layer_stream.resolve_available_vram_bytes+ with pytest.raises(RuntimeError):+ with harness.recording_preflight():+ assert layer_stream.estimate_stream_peak_vram is not estimate+ raise RuntimeError("setup failed")++ assert layer_stream.estimate_stream_peak_vram is estimate+ assert layer_stream.resolve_available_vram_bytes is resolve+++ ("argv", "message"),+ [+ ([], "--base is required"),+ (["--base", "m", "--chunk", "15"], "between 16 and 8192"),+ (["--base", "m", "--chunk", "8193"], "between 16 and 8192"),+ (["--base", "m", "--seqs", "512,x"], "comma-separated integers"),+ (["--base", "m", "--seqs", "512,0"], "must be positive"),+ (["--base", "m", "--chunk", "16"], "CUDA required"),+ ],+)+def test_refused_arguments_exit_2_before_anything_is_built(monkeypatch, capsys, argv, message):+ import torch++ monkeypatch.setattr(torch.cuda, "is_available", lambda: False)+ monkeypatch.setattr(harness, "sweep", lambda *args: pytest.fail("sweep must not run"))++ assert harness.main(argv) == harness.EXIT_USAGE+ assert message in capsys.readouterr().out