Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
bench: publish layer count reconstruction validator (#1289)
Samarth Mukhijae23db1b27399
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex e375d5a4..79a5b372 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -117,6 +117,7 @@ only in a scratchpad on a machine that is gone, which is| [`determinism.py`](harness/determinism.py) | STEP 2b / Measurements 1-2 | Historical reconstruction that measures forward, backward repetitions 1 and 3, and trained loss-curve reproducibility in-process against a resident NF4 reference; optional JSON replay remains separate from measurement | CUDA measurement requires the recorded H100-class stack and checkpoint; CPU tests cover the real measurement and invalid-measurement guards; empty overlap, non-finite values, missing measurements, or absent asymmetry fail || [`repeat_backward.py`](harness/repeat_backward.py) | STEP 2b / Measurement 3 | In-process repeated streamed backwards against one persistent resident NF4 reference; reports the first-pass count and requires later corruption while every loss remains finite and equal to the resident loss | CUDA GPU + real checkpoint files required for live NF4 measurement; seq 128, batch 1, two buffers, five repeats; historical stack: H100 80 GB, torch 2.13.0+cu130, bitsandbytes 0.50.0; CPU tests cover healthy, later-corruption, changed-loss, and invalid-measurement controls || [`depth_vs_bytes.py`](harness/depth_vs_bytes.py) | STEP 6 / NF4 bytes-per-layer bracket | Historical reconstruction of the fixed 48-layer synthetic-Llama sweep: all three NF4 reads at or below 163.8 MiB/layer must be exact, later reads at or above 171.5 MiB/layer must be wrong, and the bf16 control above the bracket must remain exact | `--measure` creates local synthetic checkpoints and needs CUDA plus the historical torch/bitsandbytes stack; JSON replay and CPU verdict tests do not require CUDA; no CUDA measurement is claimed by the index |+| [`layercount.py`](harness/layercount.py) | STEP 6 / rejected layer-count hypothesis | Historical reconstruction: small NF4 and bf16 layers remain exact across the 48-to-64 depth boundary, while a separate 48-layer NF4 control above 171.5 MiB/layer must turn wrong; each row carries three backwards and exactly four LoRA gradients per layer | `--measure` creates local synthetic checkpoints and needs CUDA plus the historical torch/bitsandbytes stack; JSON replay and CPU verdict tests do not require CUDA; no CUDA measurement is claimed by the index || [`ab_nig_sweep.py`](harness/ab_nig_sweep.py) | [#1265](https://github.com/MakazhanAlpamys/Soup/issues/1265) / [`gate-1265-ab-nig.md`](gate-1265-ab-nig.md) | Type-I and power of `soup ab`'s NIG mixture under a peek after every pair, side by side with the #1227 burn-in design, by effect_size / sigma, held-out rows, alpha and horizon; checked against `_nig_log_bayes_factor` and `msprt_step` first; picks `PRIOR_SCALE_ROWS` | CPU + numpy; ~350 s per ratio at 100,000 H0 runs over 1000 rows; `run --mode h0/h1` writes JSON parts, `report` merges them || [`ab_cs_accept_sweep.py`](harness/ab_cs_accept_sweep.py) | [#1418](https://github.com/MakazhanAlpamys/Soup/issues/1418) / [`gate-1418-ab-cs-accept.md`](gate-1418-ab-cs-accept.md) | Time to `accept_h0` under H0 and wrong `accept_h0` under H1 for `soup ab`'s confidence-sequence accept rule against #1265's, at level beta or alpha and with or without the accept hold; replays #1265's seeded runs and checks its counts against that record; checked against `_nig_confidence_half_width` and `msprt_step` first | CPU + numpy; ~430-600 s per ratio at 100,000 H0 runs over 1000 rows; `run --mode h0/h1` writes JSON parts, `report` merges them, `check` re-runs only the self-check || [`fp8_block_quantize.py`](harness/fp8_block_quantize.py) | [`gate-fp8-shard.md`](gate-fp8-shard.md) G1-G3 | Converts a bf16 DeepSeek-V3-family checkpoint to the FP8 block format it is published in (`float8_e4m3fn` + `weight_scale_inv` per 128 x 128 block, ragged edges as DeepSeek has them), one tensor at a time, and reports each tensor's dequant error against its bf16 original. Also the quantiser the FP8 tests use | CPU; Moonlight-16B-A3B: 76 s, 2.5 GB peak RSS, 16.6 GB written |diff --git a/benchmarks/harness/layercount.py b/benchmarks/harness/layercount.pynew file mode 100644index 00000000..133e9c3f--- /dev/null+++ b/benchmarks/harness/layercount.py@@ -0,0 +1,358 @@+#!/usr/bin/env python3+"""Reconstruct STEP 6's layer-count control sweep.++The record reports exact streamed/resident gradients across the 48-to-64 layer+boundary while layers remain small, plus a wrong 48-layer NF4 control above the+per-layer-size bracket. The live mode creates those synthetic checkpoints.+"""++from __future__ import annotations++import argparse+import json+import math+import sys+from pathlib import Path+from typing import Any+++class MeasurementInvalidError(RuntimeError):+ """The layer-count sweep is invalid."""+++def layercount_verdict(samples: list[dict[str, Any]]) -> bool:+ if len(samples) < 4:+ return False+ nf4_depths = set()+ sweep_sizes: dict[str, set[float]] = {"nf4": set(), "bf16": set()}+ bf16_sweep = False+ positive_control = False+ for sample in samples:+ layers = sample.get("layers")+ quant = sample.get("quant")+ role = sample.get("role")+ exact_counts = sample.get("exact_counts")+ total = sample.get("total")+ if quant not in {"nf4", "bf16"}:+ raise MeasurementInvalidError("quant must be nf4 or bf16")+ if role not in {"sweep", "control"}:+ raise MeasurementInvalidError("role must be sweep or control")+ if not isinstance(layers, int) or layers <= 0:+ raise MeasurementInvalidError("layer counts must be positive integers")+ if total != 4 * layers:+ raise MeasurementInvalidError("total must equal four LoRA gradients per layer")+ if not isinstance(exact_counts, list) or len(exact_counts) != 3:+ raise MeasurementInvalidError("exact_counts must contain three backwards")+ if any(not isinstance(exact, int) or exact < 0 or exact > total for exact in exact_counts):+ raise MeasurementInvalidError("invalid exact gradient count")+ value = float(sample.get("layer_mib", 0.0))+ if not math.isfinite(value) or value <= 0:+ raise MeasurementInvalidError("layer_mib must be finite and positive")+ all_exact = all(exact == total for exact in exact_counts)++ if role == "sweep":+ if value > 163.8 or not all_exact:+ return False+ sweep_sizes[quant].add(round(value, 6))+ if quant == "nf4":+ nf4_depths.add(layers)+ else:+ bf16_sweep = True+ else:+ if quant != "nf4" or layers != 48 or value < 171.5:+ return False+ if not any(exact < total for exact in exact_counts[1:]):+ return False+ positive_control = True++ spans_boundary = any(depth <= 48 for depth in nf4_depths) and any(+ depth >= 64 for depth in nf4_depths+ )+ constant_sizes = all(len(sizes) <= 1 for sizes in sweep_sizes.values())+ return spans_boundary and bf16_sweep and positive_control and constant_sizes+++def measure_layer_sweep(*, layer_counts: list[int], measure) -> list[dict[str, Any]]:+ """Measure both small-layer sweeps and the above-bracket NF4 control."""+ if len(set(layer_counts)) < 2:+ raise MeasurementInvalidError("layer sweep must contain distinct counts")+ rows = []+ for layers in layer_counts:+ for quant in ("nf4", "bf16"):+ row = dict(+ measure(+ layers=layers,+ hidden=64,+ intermediate=128,+ quant=quant,+ role="sweep",+ )+ )+ row.update(layers=layers, quant=quant, role="sweep")+ rows.append(row)++ control = dict(+ measure(+ layers=48,+ hidden=5120,+ intermediate=20480,+ quant="nf4",+ role="control",+ )+ )+ control.update(layers=48, quant="nf4", role="control")+ rows.append(control)+ return rows+++def _measure_checkpoint(weights: str, shards: str, quant: str) -> dict[str, Any]:+ """Run three streamed backwards against one resident reference."""+ import bitexact+ import torch+ from mechanism_cost import (+ MeasurementInvalidError as MechanismMeasurementInvalidError,+ )+ from mechanism_cost import (+ build_streamed_arm,+ gradient_snapshot,+ make_input,+ prepare_shards,+ run_backward,+ )++ from soup_cli.utils.layer_shard import shard_checkpoint+ from soup_cli.utils.layer_stream_runtime import (+ assert_canonical_parameters_intersect,+ build_meta_skeleton,+ build_streamed_model,+ )++ def restore() -> None:+ return None++ if quant == "nf4":+ index, arch = prepare_shards(weights, shards)+ streamed, runtime, restore = build_streamed_arm(weights, shards, index, arch, "control", 2)+ else:+ runtime_quant = "none"+ probe = build_meta_skeleton(weights, dtype="bfloat16", quant=runtime_quant)+ arch = bitexact.model_arch_name(probe)+ del probe+ index = shard_checkpoint(+ weights,+ shards,+ dtype="bfloat16",+ arch=arch,+ quant=runtime_quant,+ quant_suffixes=(),+ double_quant=True,+ quant_device="cuda",+ )+ streamed, runtime = build_streamed_model(+ model_id=weights,+ shard_dir=shards,+ index=index,+ lora_config=bitexact.lora_config(),+ device="cuda",+ dtype="bfloat16",+ buffers=2,+ pin=True,+ seed=3,+ quant=runtime_quant,+ double_quant=True,+ tier="ram",+ )++ reference = None+ try:+ reference = bitexact.load_resident_reference(weights, quant)+ bitexact.make_non_vacuous_lora(streamed)+ bitexact.copy_lora(streamed, reference)+ assert_canonical_parameters_intersect(streamed, reference)+ input_ids = make_input(reference, seq=128, batch=1)+ reference_loss = run_backward(reference, input_ids)+ try:+ reference_grads = gradient_snapshot(reference)+ except MechanismMeasurementInvalidError as exc:+ raise MeasurementInvalidError(str(exc)) from exc+ if (+ not math.isfinite(reference_loss)+ or not reference_grads+ or any(not torch.isfinite(value).all() for value in reference_grads.values())+ ):+ raise MeasurementInvalidError("resident measurement is empty or non-finite")++ exact_counts = []+ for _ in range(3):+ loss = run_backward(streamed, input_ids)+ try:+ current = gradient_snapshot(streamed)+ except MechanismMeasurementInvalidError as exc:+ raise MeasurementInvalidError(str(exc)) from exc+ if not math.isfinite(loss) or loss != reference_loss:+ raise MeasurementInvalidError("streamed loss differs from resident")+ if set(current) != set(reference_grads):+ raise MeasurementInvalidError("gradient parameter sets differ")+ if any(not torch.isfinite(value).all() for value in current.values()):+ raise MeasurementInvalidError("streamed gradients are non-finite")+ exact_counts.append(+ sum(torch.equal(current[name], reference_grads[name]) for name in current)+ )++ store_bytes = getattr(getattr(runtime, "source", None), "nbytes", 0)+ layers = getattr(index, "n_layers", 0)+ if store_bytes <= 0 or layers <= 0:+ raise MeasurementInvalidError("streaming store size is unavailable")+ return {+ "layer_mib": store_bytes / layers / (1024 * 1024),+ "exact_counts": exact_counts,+ "total": len(reference_grads),+ }+ finally:+ if reference is not None:+ del reference+ try:+ runtime.close()+ finally:+ restore()+ del streamed+ torch.cuda.empty_cache()+++def run_live_sweep(work_dir: Path) -> list[dict[str, Any]]:+ """Create and measure the recorded depth sweep and positive control."""+ import subprocess++ script = Path(__file__).with_name("synth_checkpoint.py")++ def measure(*, layers: int, hidden: int, intermediate: int, quant: str, role: str):+ checkpoint = work_dir / f"llama-{role}-{layers}-{hidden}-{intermediate}"+ shards = work_dir / f"shards-{role}-{layers}-{hidden}-{intermediate}-{quant}"+ if not checkpoint.exists():+ subprocess.run(+ [+ sys.executable,+ str(script),+ "--out",+ str(checkpoint),+ "--shape",+ "qwen2.5-14b",+ "--arch",+ "llama",+ "--layers",+ str(layers),+ "--hidden",+ str(hidden),+ "--intermediate",+ str(intermediate),+ "--heads",+ "4" if hidden == 64 else "40",+ "--kv-heads",+ "2" if hidden == 64 else "10",+ "--vocab",+ "128" if hidden == 64 else "1024",+ ],+ check=True,+ )+ shards.mkdir(parents=True, exist_ok=True)+ return _measure_checkpoint(str(checkpoint), str(shards), quant)++ return measure_layer_sweep(+ layer_counts=[32, 48, 56, 60, 64, 80],+ measure=measure,+ )+++def parse_args() -> argparse.Namespace:+ parser = argparse.ArgumentParser(description="Check the STEP 6 layer-count sweep.")+ parser.add_argument("--records", type=Path, help="JSON array of measured sweep rows")+ parser.add_argument("--measure", action="store_true", help="run the synthetic CUDA sweep")+ parser.add_argument("--work-dir", type=Path, default=Path("./layercount-work"))+ parser.add_argument("--self-test", action="store_true")+ return parser.parse_args()+++def run_self_test() -> int:+ assert layercount_verdict(+ [+ {+ "layers": 48,+ "quant": "nf4",+ "role": "sweep",+ "layer_mib": 0.01,+ "exact_counts": [192] * 3,+ "total": 192,+ },+ {+ "layers": 64,+ "quant": "nf4",+ "role": "sweep",+ "layer_mib": 0.01,+ "exact_counts": [256] * 3,+ "total": 256,+ },+ {+ "layers": 64,+ "quant": "bf16",+ "role": "sweep",+ "layer_mib": 0.05,+ "exact_counts": [256] * 3,+ "total": 256,+ },+ {+ "layers": 48,+ "quant": "nf4",+ "role": "control",+ "layer_mib": 187.0,+ "exact_counts": [192, 8, 8],+ "total": 192,+ },+ ]+ )+ print("PASS: layer-count verdict requires the sweep and positive control")+ return 0+++def main() -> int:+ args = parse_args()+ if args.self_test:+ return run_self_test()+ if args.measure:+ import torch++ if not torch.cuda.is_available():+ print("SKIP: CUDA is required for the synthetic measurement")+ return 0+ try:+ samples = run_live_sweep(args.work_dir.expanduser().resolve())+ print(json.dumps(samples, indent=2, sort_keys=True))+ if layercount_verdict(samples):+ print("RESULT: layer-count control relationship reproduced")+ return 0+ print("ERROR: layer-count control relationship not reproduced")+ return 1+ except (MeasurementInvalidError, RuntimeError, ValueError) as exc:+ print(f"ERROR: invalid measurement: {exc}")+ return 3+ if args.records is None:+ print("ERROR: --records is required; no CUDA measurement was performed")+ return 2+ try:+ samples = json.loads(args.records.read_text(encoding="utf-8"))+ if not isinstance(samples, list):+ raise MeasurementInvalidError("records must be a JSON array")+ if not layercount_verdict(samples):+ print("ERROR: layer-count control relationship not reproduced")+ return 1+ print("RESULT: layer-count control relationship reproduced")+ return 0+ except MeasurementInvalidError as exc:+ print(f"ERROR: invalid measurement: {exc}")+ return 3+ except (OSError, json.JSONDecodeError, TypeError) as exc:+ print(f"ERROR: invalid records: {exc}")+ return 2+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tests/test_issue379_layercount.py b/tests/test_issue379_layercount.pynew file mode 100644index 00000000..612c23db--- /dev/null+++ b/tests/test_issue379_layercount.py@@ -0,0 +1,272 @@+from argparse import Namespace+from copy import deepcopy+from pathlib import Path++import pytest++from benchmarks.harness.layercount import (+ MeasurementInvalidError,+ layercount_verdict,+ measure_layer_sweep,+ run_self_test,+)+++def recorded_rows():+ return [+ {+ "layers": 48,+ "quant": "nf4",+ "role": "sweep",+ "layer_mib": 0.01,+ "exact_counts": [192] * 3,+ "total": 192,+ },+ {+ "layers": 64,+ "quant": "nf4",+ "role": "sweep",+ "layer_mib": 0.01,+ "exact_counts": [256] * 3,+ "total": 256,+ },+ {+ "layers": 48,+ "quant": "bf16",+ "role": "sweep",+ "layer_mib": 0.05,+ "exact_counts": [192] * 3,+ "total": 192,+ },+ {+ "layers": 64,+ "quant": "bf16",+ "role": "sweep",+ "layer_mib": 0.05,+ "exact_counts": [256] * 3,+ "total": 256,+ },+ {+ "layers": 48,+ "quant": "nf4",+ "role": "control",+ "layer_mib": 187.0,+ "exact_counts": [192, 8, 8],+ "total": 192,+ },+ ]+++def test_recorded_layercount_relationship_passes():+ assert layercount_verdict(recorded_rows())+++def test_nf4_sweep_must_span_the_48_to_64_boundary():+ rows = recorded_rows()+ rows[1]["layers"] = 32+ rows[1]["total"] = 128+ rows[1]["exact_counts"] = [128] * 3+ assert not layercount_verdict(rows)+++def test_small_sweep_rows_must_be_exact_and_constant_size():+ rows = recorded_rows()+ rows[0]["exact_counts"] = [192, 8, 8]+ assert not layercount_verdict(rows)++ rows = recorded_rows()+ rows[1]["layer_mib"] = 0.02+ assert not layercount_verdict(rows)++ rows = recorded_rows()+ rows[0]["layer_mib"] = 187.0+ rows[1]["layer_mib"] = 187.0+ assert not layercount_verdict(rows)+++def test_bf16_sweep_is_required_and_must_be_exact():+ rows = [row for row in recorded_rows() if row["quant"] != "bf16"]+ assert not layercount_verdict(rows)++ rows = recorded_rows()+ rows[2]["exact_counts"] = [192, 8, 8]+ assert not layercount_verdict(rows)+++def test_positive_control_is_required_and_must_turn_wrong_later():+ rows = [row for row in recorded_rows() if row["role"] != "control"]+ assert not layercount_verdict(rows)++ rows = recorded_rows()+ rows[-1]["exact_counts"] = [192] * 3+ assert not layercount_verdict(rows)++ rows = recorded_rows()+ rows[-1]["layer_mib"] = 1.0+ assert not layercount_verdict(rows)+++ ("field", "value", "message"),+ [+ ("total", 128, "four LoRA"),+ ("exact_counts", [300, 300, 300], "invalid exact"),+ ("layer_mib", float("nan"), "finite"),+ ],+)+def test_malformed_measurements_fail_closed(field, value, message):+ rows = deepcopy(recorded_rows())+ rows[0][field] = value+ with pytest.raises(MeasurementInvalidError, match=message):+ layercount_verdict(rows)+++def test_nonpositive_layer_count_is_rejected_independently():+ rows = recorded_rows()+ rows[0].update(layers=0, total=0, exact_counts=[0] * 3)+ with pytest.raises(MeasurementInvalidError, match="positive integers"):+ layercount_verdict(rows)+++def test_measurement_api_rejects_degenerate_sweep():+ with pytest.raises(MeasurementInvalidError, match="distinct counts"):+ measure_layer_sweep(layer_counts=[32, 32], measure=lambda **_: {})+++def test_measurement_api_covers_both_quantizations_and_control():+ calls = []++ def measure(**kwargs):+ calls.append(kwargs)+ return {"layer_mib": 0.01, "exact_counts": [1] * 3, "total": 1}++ rows = measure_layer_sweep(layer_counts=[48, 64], measure=measure)++ assert len(rows) == 5+ assert [call["quant"] for call in calls] == ["nf4", "bf16", "nf4", "bf16", "nf4"]+ assert all(call["hidden"] == 64 for call in calls[:4])+ assert calls[-1] == {+ "layers": 48,+ "hidden": 5120,+ "intermediate": 20480,+ "quant": "nf4",+ "role": "control",+ }+++def test_live_sweep_builds_recorded_shapes(tmp_path, monkeypatch):+ from benchmarks.harness import layercount++ generated = []+ measured = []++ def fake_run(command, *, check):+ assert check+ checkpoint = Path(command[command.index("--out") + 1])+ generated.append(command)+ checkpoint.mkdir(parents=True)++ def fake_measure(weights, shards, quant):+ measured.append((weights, shards, quant))+ return {"layer_mib": 0.01, "exact_counts": [1] * 3, "total": 1}++ monkeypatch.setattr("subprocess.run", fake_run)+ monkeypatch.setattr(layercount, "_measure_checkpoint", fake_measure)++ rows = layercount.run_live_sweep(tmp_path)++ assert len(generated) == 7+ assert len(measured) == 13+ assert len(rows) == 13+ assert [row["layers"] for row in rows if row["quant"] == "nf4"] == [+ 32,+ 48,+ 56,+ 60,+ 64,+ 80,+ 48,+ ]+ assert rows[-1]["role"] == "control"+ assert any("20480" in command for command in generated)+ control_command = next(command for command in generated if "20480" in command)+ assert control_command[control_command.index("--kv-heads") + 1] == "10"+++def test_self_test_exercises_the_full_verdict():+ assert run_self_test() == 0+++def test_nf4_sweep_needs_a_point_at_or_below_48_layers():+ rows = recorded_rows()+ rows[0].update(layers=56, total=224, exact_counts=[224] * 3)+ assert not layercount_verdict(rows)+++def test_bf16_sweep_row_is_required_not_just_accepted():+ rows = [row for row in recorded_rows() if row["quant"] != "bf16"]+ rows.insert(+ 1,+ {+ "layers": 56,+ "quant": "nf4",+ "role": "sweep",+ "layer_mib": 0.01,+ "exact_counts": [224] * 3,+ "total": 224,+ },+ )+ assert len(rows) >= 4+ assert not layercount_verdict(rows)+++ "change",+ [+ {"quant": "bf16"},+ {"layers": 64, "total": 256, "exact_counts": [256, 8, 8]},+ ],+)+def test_control_must_be_the_48_layer_nf4_row(change):+ rows = recorded_rows()+ rows[-1].update(change)+ assert not layercount_verdict(rows)+++def test_control_must_go_wrong_after_the_first_read():+ rows = recorded_rows()+ rows[-1]["exact_counts"] = [8, 192, 192]+ assert not layercount_verdict(rows)+++ ("field", "value", "message"),+ [+ ("quant", "fp8", "nf4 or bf16"),+ ("role", "probe", "sweep or control"),+ ("exact_counts", [192, 192], "three backwards"),+ ],+)+def test_unknown_labels_and_short_reads_fail_closed(field, value, message):+ rows = recorded_rows()+ rows[0][field] = value+ with pytest.raises(MeasurementInvalidError, match=message):+ layercount_verdict(rows)++[email protected]("error", [RuntimeError("runtime"), ValueError("setup")])+def test_live_measurement_setup_errors_are_invalid(monkeypatch, tmp_path, error):+ from benchmarks.harness import layercount++ monkeypatch.setattr(+ layercount,+ "parse_args",+ lambda: Namespace(self_test=False, measure=True, work_dir=tmp_path, records=None),+ )+ monkeypatch.setattr("torch.cuda.is_available", lambda: True)+ monkeypatch.setattr(+ layercount,+ "run_live_sweep",+ lambda work_dir: (_ for _ in ()).throw(error),+ )+ assert layercount.main() == 3diff --git a/tests/test_issue379_layercount_measure.py b/tests/test_issue379_layercount_measure.pynew file mode 100644index 00000000..43563974--- /dev/null+++ b/tests/test_issue379_layercount_measure.py@@ -0,0 +1,196 @@+"""CPU tests of layercount._measure_checkpoint() with toy LoRA models.++Only the CUDA and checkpoint boundary is replaced. The production copy,+backward, gradient snapshot, and measurement assertions run on CPU.+"""++import sys+from pathlib import Path+from types import SimpleNamespace++import pytest++torch = pytest.importorskip("torch")+from torch import nn # noqa: E402++HARNESS = Path(__file__).parents[1] / "benchmarks" / "harness"+sys.path.insert(0, str(HARNESS))+import bitexact # noqa: E402+import mechanism_cost # noqa: E402++from benchmarks.harness import layercount as module # noqa: E402++LAYERS = 8+HIDDEN = 16+VOCAB = 64+STORE_BYTES = LAYERS * 123_456+++class _GradScale(torch.autograd.Function):+ """Identity forward, scaled backward: same loss, wrong gradients."""++ @staticmethod+ def forward(ctx, x, factor):+ ctx.factor = factor+ return x.view_as(x)++ @staticmethod+ def backward(ctx, grad):+ return grad * ctx.factor, None+++class _Layer(nn.Module):+ def __init__(self):+ super().__init__()+ self.base = nn.Linear(HIDDEN, HIDDEN, bias=False).requires_grad_(False)+ self.lora_A = nn.Linear(HIDDEN, 4, bias=False)+ self.lora_B = nn.Linear(4, HIDDEN, bias=False)++ def forward(self, x, factor=None):+ delta = self.lora_B(self.lora_A(x))+ if factor is not None:+ delta = _GradScale.apply(delta, factor)+ return torch.tanh(self.base(x) + delta)+++class _Out:+ def __init__(self, loss):+ self.loss = loss+++class _Toy(nn.Module):+ """wrong_from is the first 1-based backward with corrupted gradients."""++ def __init__(self, wrong_from=None, loss_shift=0.0, grad_factor=3.0):+ super().__init__()+ torch.manual_seed(0)+ self.config = SimpleNamespace(vocab_size=VOCAB)+ self.embed = nn.Embedding(VOCAB, HIDDEN).requires_grad_(False)+ self.layers = nn.ModuleList(_Layer() for _ in range(LAYERS))+ self.head = nn.Linear(HIDDEN, VOCAB, bias=False).requires_grad_(False)+ self.wrong_from = wrong_from+ self.loss_shift = loss_shift+ self.grad_factor = grad_factor+ self.calls = 0++ def forward(self, input_ids, labels):+ self.calls += 1+ wrong = self.wrong_from is not None and self.calls >= self.wrong_from+ x = self.embed(input_ids)+ for index, layer in enumerate(self.layers):+ factor = self.grad_factor if wrong and index < LAYERS - 2 else None+ x = layer(x, factor)+ logits = self.head(x)+ loss = nn.functional.cross_entropy(logits.view(-1, VOCAB), labels.view(-1))+ return _Out(loss + self.loss_shift)+++class _Runtime:+ def __init__(self, nbytes):+ self.source = SimpleNamespace(nbytes=nbytes)+ self.closed = 0++ def close(self):+ self.closed += 1+++def _measure(monkeypatch, streamed, store_bytes=STORE_BYTES):+ runtime = _Runtime(store_bytes)+ restored = []+ arms = []+ monkeypatch.setattr(mechanism_cost, "DEVICE", "cpu")+ monkeypatch.setattr(+ bitexact,+ "make_non_vacuous_lora",+ mechanism_cost.make_non_vacuous_lora,+ )+ monkeypatch.setattr(+ mechanism_cost,+ "prepare_shards",+ lambda weights, shards: (SimpleNamespace(n_layers=LAYERS), "llama"),+ )+ monkeypatch.setattr(+ mechanism_cost,+ "build_streamed_arm",+ lambda weights, shards, index, arch, arm, buffers: (+ arms.append(arm),+ streamed,+ runtime,+ lambda: restored.append(True),+ )[1:],+ )+ monkeypatch.setattr(bitexact, "load_resident_reference", lambda weights, quant: _Toy())+ row = module._measure_checkpoint("weights", "shards", "nf4")+ assert runtime.closed == 1 and restored == [True]+ assert arms == ["control"]+ return row+++def test_later_corruption_is_counted_per_read(monkeypatch):+ row = _measure(monkeypatch, _Toy(wrong_from=2))+ total = row["total"]+ assert total > 0+ assert row["exact_counts"][0] == total+ assert all(count < total for count in row["exact_counts"][1:])+++def test_healthy_streaming_reads_exact_three_times(monkeypatch):+ row = _measure(monkeypatch, _Toy())+ assert row["exact_counts"] == [row["total"]] * 3+++def test_layer_mib_is_store_bytes_per_layer(monkeypatch):+ row = _measure(monkeypatch, _Toy())+ assert row["layer_mib"] == pytest.approx(STORE_BYTES / LAYERS / (1024 * 1024))+++ ("kwargs", "message"),+ [+ ({"wrong_from": 2, "loss_shift": 1.0e-3}, "loss differs"),+ ({"wrong_from": 2, "loss_shift": float("nan")}, "loss differs"),+ ({"wrong_from": 2, "grad_factor": float("inf")}, "non-finite"),+ ],+)+def test_changed_or_nonfinite_measurements_are_invalid(monkeypatch, kwargs, message):+ with pytest.raises(module.MeasurementInvalidError, match=message):+ _measure(monkeypatch, _Toy(**kwargs))+++def test_missing_store_size_is_invalid(monkeypatch):+ with pytest.raises(module.MeasurementInvalidError, match="store size"):+ _measure(monkeypatch, _Toy(wrong_from=2), store_bytes=0)+++def test_bf16_control_only_asks_the_runtime_for_supported_quants(monkeypatch):+ from soup_cli.utils import layer_shard, layer_stream_runtime++ seen = {}++ class _StopError(Exception):+ pass++ def skeleton(model_id, *, dtype, quant="none", **kwargs):+ seen["build_meta_skeleton"] = quant+ return SimpleNamespace(config=SimpleNamespace(model_type="llama"))++ def shard(weights, shards, *, quant="none", **kwargs):+ seen["shard_checkpoint"] = quant+ return SimpleNamespace(n_layers=LAYERS)++ def build(*, quant="none", **kwargs):+ seen["build_streamed_model"] = quant+ raise _StopError++ monkeypatch.setattr(layer_stream_runtime, "build_meta_skeleton", skeleton)+ monkeypatch.setattr(layer_shard, "shard_checkpoint", shard)+ monkeypatch.setattr(layer_stream_runtime, "build_streamed_model", build)+ with pytest.raises(_StopError):+ module._measure_checkpoint("weights", "shards", "bf16")+ assert set(seen) == {+ "build_meta_skeleton",+ "shard_checkpoint",+ "build_streamed_model",+ }+ for call, quant in seen.items():+ assert quant in layer_shard.SUPPORTED_STREAM_QUANTS, (call, quant)