Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

bench: publish layer count reconstruction validator (#1289)

Samarth Mukhijae23db1b27399
diff --git a/benchmarks/README.md b/benchmarks/README.md
index e375d5a4..79a5b372 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -117,6 +117,7 @@ only in a scratchpad on a machine that is gone, which is
| [`determinism.py`](harness/determinism.py) | STEP 2b / Measurements 1-2 | Historical reconstruction that measures forward, backward repetitions 1 and 3, and trained loss-curve reproducibility in-process against a resident NF4 reference; optional JSON replay remains separate from measurement | CUDA measurement requires the recorded H100-class stack and checkpoint; CPU tests cover the real measurement and invalid-measurement guards; empty overlap, non-finite values, missing measurements, or absent asymmetry fail |
| [`repeat_backward.py`](harness/repeat_backward.py) | STEP 2b / Measurement 3 | In-process repeated streamed backwards against one persistent resident NF4 reference; reports the first-pass count and requires later corruption while every loss remains finite and equal to the resident loss | CUDA GPU + real checkpoint files required for live NF4 measurement; seq 128, batch 1, two buffers, five repeats; historical stack: H100 80 GB, torch 2.13.0+cu130, bitsandbytes 0.50.0; CPU tests cover healthy, later-corruption, changed-loss, and invalid-measurement controls |
| [`depth_vs_bytes.py`](harness/depth_vs_bytes.py) | STEP 6 / NF4 bytes-per-layer bracket | Historical reconstruction of the fixed 48-layer synthetic-Llama sweep: all three NF4 reads at or below 163.8 MiB/layer must be exact, later reads at or above 171.5 MiB/layer must be wrong, and the bf16 control above the bracket must remain exact | `--measure` creates local synthetic checkpoints and needs CUDA plus the historical torch/bitsandbytes stack; JSON replay and CPU verdict tests do not require CUDA; no CUDA measurement is claimed by the index |
+| [`layercount.py`](harness/layercount.py) | STEP 6 / rejected layer-count hypothesis | Historical reconstruction: small NF4 and bf16 layers remain exact across the 48-to-64 depth boundary, while a separate 48-layer NF4 control above 171.5 MiB/layer must turn wrong; each row carries three backwards and exactly four LoRA gradients per layer | `--measure` creates local synthetic checkpoints and needs CUDA plus the historical torch/bitsandbytes stack; JSON replay and CPU verdict tests do not require CUDA; no CUDA measurement is claimed by the index |
| [`ab_nig_sweep.py`](harness/ab_nig_sweep.py) | [#1265](https://github.com/MakazhanAlpamys/Soup/issues/1265) / [`gate-1265-ab-nig.md`](gate-1265-ab-nig.md) | Type-I and power of `soup ab`'s NIG mixture under a peek after every pair, side by side with the #1227 burn-in design, by effect_size / sigma, held-out rows, alpha and horizon; checked against `_nig_log_bayes_factor` and `msprt_step` first; picks `PRIOR_SCALE_ROWS` | CPU + numpy; ~350 s per ratio at 100,000 H0 runs over 1000 rows; `run --mode h0/h1` writes JSON parts, `report` merges them |
| [`ab_cs_accept_sweep.py`](harness/ab_cs_accept_sweep.py) | [#1418](https://github.com/MakazhanAlpamys/Soup/issues/1418) / [`gate-1418-ab-cs-accept.md`](gate-1418-ab-cs-accept.md) | Time to `accept_h0` under H0 and wrong `accept_h0` under H1 for `soup ab`'s confidence-sequence accept rule against #1265's, at level beta or alpha and with or without the accept hold; replays #1265's seeded runs and checks its counts against that record; checked against `_nig_confidence_half_width` and `msprt_step` first | CPU + numpy; ~430-600 s per ratio at 100,000 H0 runs over 1000 rows; `run --mode h0/h1` writes JSON parts, `report` merges them, `check` re-runs only the self-check |
| [`fp8_block_quantize.py`](harness/fp8_block_quantize.py) | [`gate-fp8-shard.md`](gate-fp8-shard.md) G1-G3 | Converts a bf16 DeepSeek-V3-family checkpoint to the FP8 block format it is published in (`float8_e4m3fn` + `weight_scale_inv` per 128 x 128 block, ragged edges as DeepSeek has them), one tensor at a time, and reports each tensor's dequant error against its bf16 original. Also the quantiser the FP8 tests use | CPU; Moonlight-16B-A3B: 76 s, 2.5 GB peak RSS, 16.6 GB written |
diff --git a/benchmarks/harness/layercount.py b/benchmarks/harness/layercount.py
new file mode 100644
index 00000000..133e9c3f
--- /dev/null
+++ b/benchmarks/harness/layercount.py
@@ -0,0 +1,358 @@
+#!/usr/bin/env python3
+"""Reconstruct STEP 6's layer-count control sweep.
+
+The record reports exact streamed/resident gradients across the 48-to-64 layer
+boundary while layers remain small, plus a wrong 48-layer NF4 control above the
+per-layer-size bracket. The live mode creates those synthetic checkpoints.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import sys
+from pathlib import Path
+from typing import Any
+
+
+class MeasurementInvalidError(RuntimeError):
+ """The layer-count sweep is invalid."""
+
+
+def layercount_verdict(samples: list[dict[str, Any]]) -> bool:
+ if len(samples) < 4:
+ return False
+ nf4_depths = set()
+ sweep_sizes: dict[str, set[float]] = {"nf4": set(), "bf16": set()}
+ bf16_sweep = False
+ positive_control = False
+ for sample in samples:
+ layers = sample.get("layers")
+ quant = sample.get("quant")
+ role = sample.get("role")
+ exact_counts = sample.get("exact_counts")
+ total = sample.get("total")
+ if quant not in {"nf4", "bf16"}:
+ raise MeasurementInvalidError("quant must be nf4 or bf16")
+ if role not in {"sweep", "control"}:
+ raise MeasurementInvalidError("role must be sweep or control")
+ if not isinstance(layers, int) or layers <= 0:
+ raise MeasurementInvalidError("layer counts must be positive integers")
+ if total != 4 * layers:
+ raise MeasurementInvalidError("total must equal four LoRA gradients per layer")
+ if not isinstance(exact_counts, list) or len(exact_counts) != 3:
+ raise MeasurementInvalidError("exact_counts must contain three backwards")
+ if any(not isinstance(exact, int) or exact < 0 or exact > total for exact in exact_counts):
+ raise MeasurementInvalidError("invalid exact gradient count")
+ value = float(sample.get("layer_mib", 0.0))
+ if not math.isfinite(value) or value <= 0:
+ raise MeasurementInvalidError("layer_mib must be finite and positive")
+ all_exact = all(exact == total for exact in exact_counts)
+
+ if role == "sweep":
+ if value > 163.8 or not all_exact:
+ return False
+ sweep_sizes[quant].add(round(value, 6))
+ if quant == "nf4":
+ nf4_depths.add(layers)
+ else:
+ bf16_sweep = True
+ else:
+ if quant != "nf4" or layers != 48 or value < 171.5:
+ return False
+ if not any(exact < total for exact in exact_counts[1:]):
+ return False
+ positive_control = True
+
+ spans_boundary = any(depth <= 48 for depth in nf4_depths) and any(
+ depth >= 64 for depth in nf4_depths
+ )
+ constant_sizes = all(len(sizes) <= 1 for sizes in sweep_sizes.values())
+ return spans_boundary and bf16_sweep and positive_control and constant_sizes
+
+
+def measure_layer_sweep(*, layer_counts: list[int], measure) -> list[dict[str, Any]]:
+ """Measure both small-layer sweeps and the above-bracket NF4 control."""
+ if len(set(layer_counts)) < 2:
+ raise MeasurementInvalidError("layer sweep must contain distinct counts")
+ rows = []
+ for layers in layer_counts:
+ for quant in ("nf4", "bf16"):
+ row = dict(
+ measure(
+ layers=layers,
+ hidden=64,
+ intermediate=128,
+ quant=quant,
+ role="sweep",
+ )
+ )
+ row.update(layers=layers, quant=quant, role="sweep")
+ rows.append(row)
+
+ control = dict(
+ measure(
+ layers=48,
+ hidden=5120,
+ intermediate=20480,
+ quant="nf4",
+ role="control",
+ )
+ )
+ control.update(layers=48, quant="nf4", role="control")
+ rows.append(control)
+ return rows
+
+
+def _measure_checkpoint(weights: str, shards: str, quant: str) -> dict[str, Any]:
+ """Run three streamed backwards against one resident reference."""
+ import bitexact
+ import torch
+ from mechanism_cost import (
+ MeasurementInvalidError as MechanismMeasurementInvalidError,
+ )
+ from mechanism_cost import (
+ build_streamed_arm,
+ gradient_snapshot,
+ make_input,
+ prepare_shards,
+ run_backward,
+ )
+
+ from soup_cli.utils.layer_shard import shard_checkpoint
+ from soup_cli.utils.layer_stream_runtime import (
+ assert_canonical_parameters_intersect,
+ build_meta_skeleton,
+ build_streamed_model,
+ )
+
+ def restore() -> None:
+ return None
+
+ if quant == "nf4":
+ index, arch = prepare_shards(weights, shards)
+ streamed, runtime, restore = build_streamed_arm(weights, shards, index, arch, "control", 2)
+ else:
+ runtime_quant = "none"
+ probe = build_meta_skeleton(weights, dtype="bfloat16", quant=runtime_quant)
+ arch = bitexact.model_arch_name(probe)
+ del probe
+ index = shard_checkpoint(
+ weights,
+ shards,
+ dtype="bfloat16",
+ arch=arch,
+ quant=runtime_quant,
+ quant_suffixes=(),
+ double_quant=True,
+ quant_device="cuda",
+ )
+ streamed, runtime = build_streamed_model(
+ model_id=weights,
+ shard_dir=shards,
+ index=index,
+ lora_config=bitexact.lora_config(),
+ device="cuda",
+ dtype="bfloat16",
+ buffers=2,
+ pin=True,
+ seed=3,
+ quant=runtime_quant,
+ double_quant=True,
+ tier="ram",
+ )
+
+ reference = None
+ try:
+ reference = bitexact.load_resident_reference(weights, quant)
+ bitexact.make_non_vacuous_lora(streamed)
+ bitexact.copy_lora(streamed, reference)
+ assert_canonical_parameters_intersect(streamed, reference)
+ input_ids = make_input(reference, seq=128, batch=1)
+ reference_loss = run_backward(reference, input_ids)
+ try:
+ reference_grads = gradient_snapshot(reference)
+ except MechanismMeasurementInvalidError as exc:
+ raise MeasurementInvalidError(str(exc)) from exc
+ if (
+ not math.isfinite(reference_loss)
+ or not reference_grads
+ or any(not torch.isfinite(value).all() for value in reference_grads.values())
+ ):
+ raise MeasurementInvalidError("resident measurement is empty or non-finite")
+
+ exact_counts = []
+ for _ in range(3):
+ loss = run_backward(streamed, input_ids)
+ try:
+ current = gradient_snapshot(streamed)
+ except MechanismMeasurementInvalidError as exc:
+ raise MeasurementInvalidError(str(exc)) from exc
+ if not math.isfinite(loss) or loss != reference_loss:
+ raise MeasurementInvalidError("streamed loss differs from resident")
+ if set(current) != set(reference_grads):
+ raise MeasurementInvalidError("gradient parameter sets differ")
+ if any(not torch.isfinite(value).all() for value in current.values()):
+ raise MeasurementInvalidError("streamed gradients are non-finite")
+ exact_counts.append(
+ sum(torch.equal(current[name], reference_grads[name]) for name in current)
+ )
+
+ store_bytes = getattr(getattr(runtime, "source", None), "nbytes", 0)
+ layers = getattr(index, "n_layers", 0)
+ if store_bytes <= 0 or layers <= 0:
+ raise MeasurementInvalidError("streaming store size is unavailable")
+ return {
+ "layer_mib": store_bytes / layers / (1024 * 1024),
+ "exact_counts": exact_counts,
+ "total": len(reference_grads),
+ }
+ finally:
+ if reference is not None:
+ del reference
+ try:
+ runtime.close()
+ finally:
+ restore()
+ del streamed
+ torch.cuda.empty_cache()
+
+
+def run_live_sweep(work_dir: Path) -> list[dict[str, Any]]:
+ """Create and measure the recorded depth sweep and positive control."""
+ import subprocess
+
+ script = Path(__file__).with_name("synth_checkpoint.py")
+
+ def measure(*, layers: int, hidden: int, intermediate: int, quant: str, role: str):
+ checkpoint = work_dir / f"llama-{role}-{layers}-{hidden}-{intermediate}"
+ shards = work_dir / f"shards-{role}-{layers}-{hidden}-{intermediate}-{quant}"
+ if not checkpoint.exists():
+ subprocess.run(
+ [
+ sys.executable,
+ str(script),
+ "--out",
+ str(checkpoint),
+ "--shape",
+ "qwen2.5-14b",
+ "--arch",
+ "llama",
+ "--layers",
+ str(layers),
+ "--hidden",
+ str(hidden),
+ "--intermediate",
+ str(intermediate),
+ "--heads",
+ "4" if hidden == 64 else "40",
+ "--kv-heads",
+ "2" if hidden == 64 else "10",
+ "--vocab",
+ "128" if hidden == 64 else "1024",
+ ],
+ check=True,
+ )
+ shards.mkdir(parents=True, exist_ok=True)
+ return _measure_checkpoint(str(checkpoint), str(shards), quant)
+
+ return measure_layer_sweep(
+ layer_counts=[32, 48, 56, 60, 64, 80],
+ measure=measure,
+ )
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description="Check the STEP 6 layer-count sweep.")
+ parser.add_argument("--records", type=Path, help="JSON array of measured sweep rows")
+ parser.add_argument("--measure", action="store_true", help="run the synthetic CUDA sweep")
+ parser.add_argument("--work-dir", type=Path, default=Path("./layercount-work"))
+ parser.add_argument("--self-test", action="store_true")
+ return parser.parse_args()
+
+
+def run_self_test() -> int:
+ assert layercount_verdict(
+ [
+ {
+ "layers": 48,
+ "quant": "nf4",
+ "role": "sweep",
+ "layer_mib": 0.01,
+ "exact_counts": [192] * 3,
+ "total": 192,
+ },
+ {
+ "layers": 64,
+ "quant": "nf4",
+ "role": "sweep",
+ "layer_mib": 0.01,
+ "exact_counts": [256] * 3,
+ "total": 256,
+ },
+ {
+ "layers": 64,
+ "quant": "bf16",
+ "role": "sweep",
+ "layer_mib": 0.05,
+ "exact_counts": [256] * 3,
+ "total": 256,
+ },
+ {
+ "layers": 48,
+ "quant": "nf4",
+ "role": "control",
+ "layer_mib": 187.0,
+ "exact_counts": [192, 8, 8],
+ "total": 192,
+ },
+ ]
+ )
+ print("PASS: layer-count verdict requires the sweep and positive control")
+ return 0
+
+
+def main() -> int:
+ args = parse_args()
+ if args.self_test:
+ return run_self_test()
+ if args.measure:
+ import torch
+
+ if not torch.cuda.is_available():
+ print("SKIP: CUDA is required for the synthetic measurement")
+ return 0
+ try:
+ samples = run_live_sweep(args.work_dir.expanduser().resolve())
+ print(json.dumps(samples, indent=2, sort_keys=True))
+ if layercount_verdict(samples):
+ print("RESULT: layer-count control relationship reproduced")
+ return 0
+ print("ERROR: layer-count control relationship not reproduced")
+ return 1
+ except (MeasurementInvalidError, RuntimeError, ValueError) as exc:
+ print(f"ERROR: invalid measurement: {exc}")
+ return 3
+ if args.records is None:
+ print("ERROR: --records is required; no CUDA measurement was performed")
+ return 2
+ try:
+ samples = json.loads(args.records.read_text(encoding="utf-8"))
+ if not isinstance(samples, list):
+ raise MeasurementInvalidError("records must be a JSON array")
+ if not layercount_verdict(samples):
+ print("ERROR: layer-count control relationship not reproduced")
+ return 1
+ print("RESULT: layer-count control relationship reproduced")
+ return 0
+ except MeasurementInvalidError as exc:
+ print(f"ERROR: invalid measurement: {exc}")
+ return 3
+ except (OSError, json.JSONDecodeError, TypeError) as exc:
+ print(f"ERROR: invalid records: {exc}")
+ return 2
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/test_issue379_layercount.py b/tests/test_issue379_layercount.py
new file mode 100644
index 00000000..612c23db
--- /dev/null
+++ b/tests/test_issue379_layercount.py
@@ -0,0 +1,272 @@
+from argparse import Namespace
+from copy import deepcopy
+from pathlib import Path
+
+import pytest
+
+from benchmarks.harness.layercount import (
+ MeasurementInvalidError,
+ layercount_verdict,
+ measure_layer_sweep,
+ run_self_test,
+)
+
+
+def recorded_rows():
+ return [
+ {
+ "layers": 48,
+ "quant": "nf4",
+ "role": "sweep",
+ "layer_mib": 0.01,
+ "exact_counts": [192] * 3,
+ "total": 192,
+ },
+ {
+ "layers": 64,
+ "quant": "nf4",
+ "role": "sweep",
+ "layer_mib": 0.01,
+ "exact_counts": [256] * 3,
+ "total": 256,
+ },
+ {
+ "layers": 48,
+ "quant": "bf16",
+ "role": "sweep",
+ "layer_mib": 0.05,
+ "exact_counts": [192] * 3,
+ "total": 192,
+ },
+ {
+ "layers": 64,
+ "quant": "bf16",
+ "role": "sweep",
+ "layer_mib": 0.05,
+ "exact_counts": [256] * 3,
+ "total": 256,
+ },
+ {
+ "layers": 48,
+ "quant": "nf4",
+ "role": "control",
+ "layer_mib": 187.0,
+ "exact_counts": [192, 8, 8],
+ "total": 192,
+ },
+ ]
+
+
+def test_recorded_layercount_relationship_passes():
+ assert layercount_verdict(recorded_rows())
+
+
+def test_nf4_sweep_must_span_the_48_to_64_boundary():
+ rows = recorded_rows()
+ rows[1]["layers"] = 32
+ rows[1]["total"] = 128
+ rows[1]["exact_counts"] = [128] * 3
+ assert not layercount_verdict(rows)
+
+
+def test_small_sweep_rows_must_be_exact_and_constant_size():
+ rows = recorded_rows()
+ rows[0]["exact_counts"] = [192, 8, 8]
+ assert not layercount_verdict(rows)
+
+ rows = recorded_rows()
+ rows[1]["layer_mib"] = 0.02
+ assert not layercount_verdict(rows)
+
+ rows = recorded_rows()
+ rows[0]["layer_mib"] = 187.0
+ rows[1]["layer_mib"] = 187.0
+ assert not layercount_verdict(rows)
+
+
+def test_bf16_sweep_is_required_and_must_be_exact():
+ rows = [row for row in recorded_rows() if row["quant"] != "bf16"]
+ assert not layercount_verdict(rows)
+
+ rows = recorded_rows()
+ rows[2]["exact_counts"] = [192, 8, 8]
+ assert not layercount_verdict(rows)
+
+
+def test_positive_control_is_required_and_must_turn_wrong_later():
+ rows = [row for row in recorded_rows() if row["role"] != "control"]
+ assert not layercount_verdict(rows)
+
+ rows = recorded_rows()
+ rows[-1]["exact_counts"] = [192] * 3
+ assert not layercount_verdict(rows)
+
+ rows = recorded_rows()
+ rows[-1]["layer_mib"] = 1.0
+ assert not layercount_verdict(rows)
+
+
+ ("field", "value", "message"),
+ [
+ ("total", 128, "four LoRA"),
+ ("exact_counts", [300, 300, 300], "invalid exact"),
+ ("layer_mib", float("nan"), "finite"),
+ ],
+)
+def test_malformed_measurements_fail_closed(field, value, message):
+ rows = deepcopy(recorded_rows())
+ rows[0][field] = value
+ with pytest.raises(MeasurementInvalidError, match=message):
+ layercount_verdict(rows)
+
+
+def test_nonpositive_layer_count_is_rejected_independently():
+ rows = recorded_rows()
+ rows[0].update(layers=0, total=0, exact_counts=[0] * 3)
+ with pytest.raises(MeasurementInvalidError, match="positive integers"):
+ layercount_verdict(rows)
+
+
+def test_measurement_api_rejects_degenerate_sweep():
+ with pytest.raises(MeasurementInvalidError, match="distinct counts"):
+ measure_layer_sweep(layer_counts=[32, 32], measure=lambda **_: {})
+
+
+def test_measurement_api_covers_both_quantizations_and_control():
+ calls = []
+
+ def measure(**kwargs):
+ calls.append(kwargs)
+ return {"layer_mib": 0.01, "exact_counts": [1] * 3, "total": 1}
+
+ rows = measure_layer_sweep(layer_counts=[48, 64], measure=measure)
+
+ assert len(rows) == 5
+ assert [call["quant"] for call in calls] == ["nf4", "bf16", "nf4", "bf16", "nf4"]
+ assert all(call["hidden"] == 64 for call in calls[:4])
+ assert calls[-1] == {
+ "layers": 48,
+ "hidden": 5120,
+ "intermediate": 20480,
+ "quant": "nf4",
+ "role": "control",
+ }
+
+
+def test_live_sweep_builds_recorded_shapes(tmp_path, monkeypatch):
+ from benchmarks.harness import layercount
+
+ generated = []
+ measured = []
+
+ def fake_run(command, *, check):
+ assert check
+ checkpoint = Path(command[command.index("--out") + 1])
+ generated.append(command)
+ checkpoint.mkdir(parents=True)
+
+ def fake_measure(weights, shards, quant):
+ measured.append((weights, shards, quant))
+ return {"layer_mib": 0.01, "exact_counts": [1] * 3, "total": 1}
+
+ monkeypatch.setattr("subprocess.run", fake_run)
+ monkeypatch.setattr(layercount, "_measure_checkpoint", fake_measure)
+
+ rows = layercount.run_live_sweep(tmp_path)
+
+ assert len(generated) == 7
+ assert len(measured) == 13
+ assert len(rows) == 13
+ assert [row["layers"] for row in rows if row["quant"] == "nf4"] == [
+ 32,
+ 48,
+ 56,
+ 60,
+ 64,
+ 80,
+ 48,
+ ]
+ assert rows[-1]["role"] == "control"
+ assert any("20480" in command for command in generated)
+ control_command = next(command for command in generated if "20480" in command)
+ assert control_command[control_command.index("--kv-heads") + 1] == "10"
+
+
+def test_self_test_exercises_the_full_verdict():
+ assert run_self_test() == 0
+
+
+def test_nf4_sweep_needs_a_point_at_or_below_48_layers():
+ rows = recorded_rows()
+ rows[0].update(layers=56, total=224, exact_counts=[224] * 3)
+ assert not layercount_verdict(rows)
+
+
+def test_bf16_sweep_row_is_required_not_just_accepted():
+ rows = [row for row in recorded_rows() if row["quant"] != "bf16"]
+ rows.insert(
+ 1,
+ {
+ "layers": 56,
+ "quant": "nf4",
+ "role": "sweep",
+ "layer_mib": 0.01,
+ "exact_counts": [224] * 3,
+ "total": 224,
+ },
+ )
+ assert len(rows) >= 4
+ assert not layercount_verdict(rows)
+
+
+ "change",
+ [
+ {"quant": "bf16"},
+ {"layers": 64, "total": 256, "exact_counts": [256, 8, 8]},
+ ],
+)
+def test_control_must_be_the_48_layer_nf4_row(change):
+ rows = recorded_rows()
+ rows[-1].update(change)
+ assert not layercount_verdict(rows)
+
+
+def test_control_must_go_wrong_after_the_first_read():
+ rows = recorded_rows()
+ rows[-1]["exact_counts"] = [8, 192, 192]
+ assert not layercount_verdict(rows)
+
+
+ ("field", "value", "message"),
+ [
+ ("quant", "fp8", "nf4 or bf16"),
+ ("role", "probe", "sweep or control"),
+ ("exact_counts", [192, 192], "three backwards"),
+ ],
+)
+def test_unknown_labels_and_short_reads_fail_closed(field, value, message):
+ rows = recorded_rows()
+ rows[0][field] = value
+ with pytest.raises(MeasurementInvalidError, match=message):
+ layercount_verdict(rows)
+
+
[email protected]("error", [RuntimeError("runtime"), ValueError("setup")])
+def test_live_measurement_setup_errors_are_invalid(monkeypatch, tmp_path, error):
+ from benchmarks.harness import layercount
+
+ monkeypatch.setattr(
+ layercount,
+ "parse_args",
+ lambda: Namespace(self_test=False, measure=True, work_dir=tmp_path, records=None),
+ )
+ monkeypatch.setattr("torch.cuda.is_available", lambda: True)
+ monkeypatch.setattr(
+ layercount,
+ "run_live_sweep",
+ lambda work_dir: (_ for _ in ()).throw(error),
+ )
+ assert layercount.main() == 3
diff --git a/tests/test_issue379_layercount_measure.py b/tests/test_issue379_layercount_measure.py
new file mode 100644
index 00000000..43563974
--- /dev/null
+++ b/tests/test_issue379_layercount_measure.py
@@ -0,0 +1,196 @@
+"""CPU tests of layercount._measure_checkpoint() with toy LoRA models.
+
+Only the CUDA and checkpoint boundary is replaced. The production copy,
+backward, gradient snapshot, and measurement assertions run on CPU.
+"""
+
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+
+torch = pytest.importorskip("torch")
+from torch import nn # noqa: E402
+
+HARNESS = Path(__file__).parents[1] / "benchmarks" / "harness"
+sys.path.insert(0, str(HARNESS))
+import bitexact # noqa: E402
+import mechanism_cost # noqa: E402
+
+from benchmarks.harness import layercount as module # noqa: E402
+
+LAYERS = 8
+HIDDEN = 16
+VOCAB = 64
+STORE_BYTES = LAYERS * 123_456
+
+
+class _GradScale(torch.autograd.Function):
+ """Identity forward, scaled backward: same loss, wrong gradients."""
+
+ @staticmethod
+ def forward(ctx, x, factor):
+ ctx.factor = factor
+ return x.view_as(x)
+
+ @staticmethod
+ def backward(ctx, grad):
+ return grad * ctx.factor, None
+
+
+class _Layer(nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.base = nn.Linear(HIDDEN, HIDDEN, bias=False).requires_grad_(False)
+ self.lora_A = nn.Linear(HIDDEN, 4, bias=False)
+ self.lora_B = nn.Linear(4, HIDDEN, bias=False)
+
+ def forward(self, x, factor=None):
+ delta = self.lora_B(self.lora_A(x))
+ if factor is not None:
+ delta = _GradScale.apply(delta, factor)
+ return torch.tanh(self.base(x) + delta)
+
+
+class _Out:
+ def __init__(self, loss):
+ self.loss = loss
+
+
+class _Toy(nn.Module):
+ """wrong_from is the first 1-based backward with corrupted gradients."""
+
+ def __init__(self, wrong_from=None, loss_shift=0.0, grad_factor=3.0):
+ super().__init__()
+ torch.manual_seed(0)
+ self.config = SimpleNamespace(vocab_size=VOCAB)
+ self.embed = nn.Embedding(VOCAB, HIDDEN).requires_grad_(False)
+ self.layers = nn.ModuleList(_Layer() for _ in range(LAYERS))
+ self.head = nn.Linear(HIDDEN, VOCAB, bias=False).requires_grad_(False)
+ self.wrong_from = wrong_from
+ self.loss_shift = loss_shift
+ self.grad_factor = grad_factor
+ self.calls = 0
+
+ def forward(self, input_ids, labels):
+ self.calls += 1
+ wrong = self.wrong_from is not None and self.calls >= self.wrong_from
+ x = self.embed(input_ids)
+ for index, layer in enumerate(self.layers):
+ factor = self.grad_factor if wrong and index < LAYERS - 2 else None
+ x = layer(x, factor)
+ logits = self.head(x)
+ loss = nn.functional.cross_entropy(logits.view(-1, VOCAB), labels.view(-1))
+ return _Out(loss + self.loss_shift)
+
+
+class _Runtime:
+ def __init__(self, nbytes):
+ self.source = SimpleNamespace(nbytes=nbytes)
+ self.closed = 0
+
+ def close(self):
+ self.closed += 1
+
+
+def _measure(monkeypatch, streamed, store_bytes=STORE_BYTES):
+ runtime = _Runtime(store_bytes)
+ restored = []
+ arms = []
+ monkeypatch.setattr(mechanism_cost, "DEVICE", "cpu")
+ monkeypatch.setattr(
+ bitexact,
+ "make_non_vacuous_lora",
+ mechanism_cost.make_non_vacuous_lora,
+ )
+ monkeypatch.setattr(
+ mechanism_cost,
+ "prepare_shards",
+ lambda weights, shards: (SimpleNamespace(n_layers=LAYERS), "llama"),
+ )
+ monkeypatch.setattr(
+ mechanism_cost,
+ "build_streamed_arm",
+ lambda weights, shards, index, arch, arm, buffers: (
+ arms.append(arm),
+ streamed,
+ runtime,
+ lambda: restored.append(True),
+ )[1:],
+ )
+ monkeypatch.setattr(bitexact, "load_resident_reference", lambda weights, quant: _Toy())
+ row = module._measure_checkpoint("weights", "shards", "nf4")
+ assert runtime.closed == 1 and restored == [True]
+ assert arms == ["control"]
+ return row
+
+
+def test_later_corruption_is_counted_per_read(monkeypatch):
+ row = _measure(monkeypatch, _Toy(wrong_from=2))
+ total = row["total"]
+ assert total > 0
+ assert row["exact_counts"][0] == total
+ assert all(count < total for count in row["exact_counts"][1:])
+
+
+def test_healthy_streaming_reads_exact_three_times(monkeypatch):
+ row = _measure(monkeypatch, _Toy())
+ assert row["exact_counts"] == [row["total"]] * 3
+
+
+def test_layer_mib_is_store_bytes_per_layer(monkeypatch):
+ row = _measure(monkeypatch, _Toy())
+ assert row["layer_mib"] == pytest.approx(STORE_BYTES / LAYERS / (1024 * 1024))
+
+
+ ("kwargs", "message"),
+ [
+ ({"wrong_from": 2, "loss_shift": 1.0e-3}, "loss differs"),
+ ({"wrong_from": 2, "loss_shift": float("nan")}, "loss differs"),
+ ({"wrong_from": 2, "grad_factor": float("inf")}, "non-finite"),
+ ],
+)
+def test_changed_or_nonfinite_measurements_are_invalid(monkeypatch, kwargs, message):
+ with pytest.raises(module.MeasurementInvalidError, match=message):
+ _measure(monkeypatch, _Toy(**kwargs))
+
+
+def test_missing_store_size_is_invalid(monkeypatch):
+ with pytest.raises(module.MeasurementInvalidError, match="store size"):
+ _measure(monkeypatch, _Toy(wrong_from=2), store_bytes=0)
+
+
+def test_bf16_control_only_asks_the_runtime_for_supported_quants(monkeypatch):
+ from soup_cli.utils import layer_shard, layer_stream_runtime
+
+ seen = {}
+
+ class _StopError(Exception):
+ pass
+
+ def skeleton(model_id, *, dtype, quant="none", **kwargs):
+ seen["build_meta_skeleton"] = quant
+ return SimpleNamespace(config=SimpleNamespace(model_type="llama"))
+
+ def shard(weights, shards, *, quant="none", **kwargs):
+ seen["shard_checkpoint"] = quant
+ return SimpleNamespace(n_layers=LAYERS)
+
+ def build(*, quant="none", **kwargs):
+ seen["build_streamed_model"] = quant
+ raise _StopError
+
+ monkeypatch.setattr(layer_stream_runtime, "build_meta_skeleton", skeleton)
+ monkeypatch.setattr(layer_shard, "shard_checkpoint", shard)
+ monkeypatch.setattr(layer_stream_runtime, "build_streamed_model", build)
+ with pytest.raises(_StopError):
+ module._measure_checkpoint("weights", "shards", "bf16")
+ assert set(seen) == {
+ "build_meta_skeleton",
+ "shard_checkpoint",
+ "build_streamed_model",
+ }
+ for call, quant in seen.items():
+ assert quant in layer_shard.SUPPORTED_STREAM_QUANTS, (call, quant)