Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
fix(eval): give mini_mmlu headroom below the 1.000 rail (#1192, 3/5) (#2113)
Aaron Probha43d351f488a2
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex e5c1732b..e375d5a4 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -35,6 +35,7 @@ They are the evidence behind the preprint:| [`gate-v0.76.0-tool-call-discrimination.md`](gate-v0.76.0-tool-call-discrimination.md) | `mini_tool_call` discrimination after #405 | Qwen2.5-7B-Instruct moves from the shipped 1.000 ceiling to **0.850 (34/40)** while SmolLM2-135M-Instruct scores **0.050 (2/40)**; the record keeps the rejected 0.969 candidate and the provenance-scale change. || [`gate-v0.76.0-arithmetic-discrimination.md`](gate-v0.76.0-arithmetic-discrimination.md) | `mini_arithmetic` discrimination for #1192 | Qwen2.5-1.5B-Instruct scores **0.800 (32/40)** and SmolLM2-135M-Instruct **0.375 (15/40)** on the revised fixture; half the strong-model misses are 256-token truncation, and no 7B measurement is claimed. || [`gate-v0.76.0-common-sense-discrimination.md`](gate-v0.76.0-common-sense-discrimination.md) | `mini_common_sense` discrimination for #1192 | Qwen2.5-1.5B-Instruct scores **0.850 (34/40)** and SmolLM2-135M-Instruct **0.275 (11/40)** on the revised fixture, from a 24-row candidate pool; the answer key no longer lets an always-"B" model score 0.583. No 7B measurement is claimed. |+| [`gate-v0.76.0-mmlu-discrimination.md`](gate-v0.76.0-mmlu-discrimination.md) | `mini_mmlu` discrimination for #1192 | Qwen2.5-1.5B-Instruct moves from **1.000 (26/26)** to **0.850 (34/40)** and SmolLM2-135M-Instruct scores **0.350 (14/40)**, at chance; 14 rows selected from two 24-row candidate pools, and the answer key no longer lets an always-"B" model score 0.462. No 7B measurement is claimed. || [`run-m1-8gb-mlx-sft.md`](run-m1-8gb-mlx-sft.md) | Not a gate — the MLX backend's training loop on hardware CI does not have | `MLXSFTTrainerWrapper` runs end to end on an **8 GB M1**, and the shipped `llama3.1-8b-sft-mlx` recipe **fits**: peak **5.154 GB**, 48 iterations in 71 s. A model too large does **not** fail here, it pages — free memory sat at 5-6% while the machine stayed responsive, so an allocation-failure pre-flight would not fire (the #649 trap from the other platform). Carries a **failed fixture** — the model id #23 itself recommends ships legacy `weights.NN.safetensors` and cannot be loaded — and two **corrections** I made mid-measurement || [`gate-qwen4-ple-m4-max.md`](gate-qwen4-ple-m4-max.md) | Qwen4-Exp/oQ decoder mapping and read-only PLE on Apple Silicon | **Partial pass only:** 1,167/1,167 cached tensors and tiny CPU/MPS parity passed; the production optimizer step was stopped and is explicitly **not validated** || [`gate-h100-validation.md`](gate-h100-validation.md) | The method on someone else's hardware: bit-exactness at real sizes, convergence quality, DeepSpeed, variance | **Forward** bit-exact to 72B; **backward** bit-exact to 14B NF4 pre-repair, re-gated after the STEP 14 fix at 32B (256/256) **and at 72B (320/320, the size where the defect was worst)**; 2.93x DeepSpeed ZeRO-3 offload in 9.7x less VRAM; and the silent wrong-gradient defect that fix repairs. **Carries three dated 2026-08-13 corrections**: it explains the H100 replication as host-to-device transfer, which the probe record above later measured and refuted. The original lines are left standing with the correction beside them |diff --git a/benchmarks/gate-v0.76.0-mmlu-discrimination.md b/benchmarks/gate-v0.76.0-mmlu-discrimination.mdnew file mode 100644index 00000000..7d2f0d37--- /dev/null+++ b/benchmarks/gate-v0.76.0-mmlu-discrimination.md@@ -0,0 +1,143 @@+# v0.76.0 gate: MMLU-style discrimination (#1192)++`mini_mmlu` had reached a ceiling: the #1111 record measured the strong+reference at **1.000 (26/26)**, and Qwen2.5-1.5B-Instruct scored 1.000 (26/26)+in the arithmetic and common-sense records. Its answer key was also lopsided:+12 of 26 answers were B, so a model that only ever answered "B" scored 0.462.+This record measures two 24-row candidate pools, the 14 rows selected from them,+and the revised 40-row fixture on two models.++## Environment and method++- Apple M1, 32 GB unified memory; MPS+- checkpoint-native dtype (`dtype="auto"`), greedy decoding,+ `BEHAVIOURAL_MAX_NEW_TOKENS` (256) new-token cap+- strong reference: `Qwen/Qwen2.5-1.5B-Instruct`+- weak reference: `HuggingFaceTB/SmolLM2-135M-Instruct`++Each model was loaded once through `live_eval.make_generator`. Candidates were+prompted with `build_mcq_prompt` and scored with `score_answer`, the same path+`score_bundled_suite` uses; the final fixture was scored with+`score_bundled_suite` over every bundled suite, on the same machine.++**The strong reference is not the #1111 one**, as in the arithmetic and+common-sense records. These numbers show the fixture leaves the rails for a 1.5B+model. They do not show that a 7B model is off the ceiling.++## Candidate pools and selection++48 hand-authored, original four-option rows in two rounds, each key carrying a+one-line justification. Correct letters were spread six to a letter by option+position before measurement, so the pools measured are the pools shipped.++| Pool | Rows | Qwen2.5-1.5B-Instruct | SmolLM2-135M-Instruct |+|---|---:|---:|---:|+| round 1: general science, history, literature, maths | 24 | 21/24 | 6/24 |+| round 2: rows with a common misconception as a distractor | 24 | 19/24 | 8/24 |++Round 1 alone was too easy to move the suite: added to the 26 original rows it+would have left Qwen at about 0.94. Qwen answered every candidate in both rounds+with a bare or parenthesised letter, so no miss was an extraction or truncation+failure.++Selection kept 14 rows:++| Decision | Candidates | Reason |+|---|---|---|+| keep: strong-model misses | fastest-rotating planet, hexagon interior angles (round 1); longest coastline, octopus hearts, tallest mountain base to peak, bits in a nibble (round 2) | each miss chose a plausible wrong answer: Mercury, 1080 degrees, Australia, 4, Everest, 2 |+| keep: strong-model passes | mitosis phase (round 1); hottest planet, symbol K, smallest ocean, Saturn's largest moon, hardest body tissue, Gibraltar, capital of Turkey (round 2) | keeps the suite from measuring only the traps |+| reject: strong-model misses | country with most time zones, largest desert | correct keys, but each turns on a definition a reviewer can argue (territories counted; Antarctica as a desert) |+| reject | the remaining 32 passes | not needed to place the strong model off the rail |++Two of the six kept misses chose option A (Mercury, Everest), as did both+rejected misses (Russia, the Sahara). Unlike the rejected "A" misses in the+common-sense record, A held the misconception in each of these four rows, so+they read as the trap working rather than a letter preference. The other four+kept misses chose B or D.++## Result++The final fixture keeps the 26 original rows, appends the 14 selected rows, and+swaps options on two original rows (square root of 81 B -> A, author of *Romeo+and Juliet* B -> C) so no letter answers more than 30% of the suite. The key is+now A 11, B 12, C 11, D 6.++| Fixture | Rows | Qwen2.5-1.5B-Instruct | SmolLM2-135M-Instruct |+|---|---:|---:|---:|+| v0.73.2 fixture (common-sense record) | 26 | 1.000 (26/26) | 0.269 (7/26) |+| revised fixture | 40 | **0.850 (34/40)** | **0.350 (14/40)** |++Qwen2.5-1.5B missed 6 rows, all new. Row numbers are 0-based indices into+`MINI_MMLU`; index 26 is the first new row.++| Row | Expected | Chosen |+|---:|---|---|+| 26 | D (Jupiter) | A (Mercury) |+| 27 | C (720 degrees) | D (1080 degrees) |+| 28 | C (Canada) | B (Australia) |+| 29 | A (3) | D (4) |+| 30 | B (Mauna Kea) | A (Mount Everest) |+| 31 | A (4) | B (2) |++These are the same six misses the candidate rounds measured; every original row+and every kept pass scored as it did there. Qwen answered both re-lettered rows+correctly.++SmolLM2 scored 9/26 on the original rows and 5/14 on the new ones. That is at+chance: uniform guessing over 26 three-option and 14 four-option rows expects+12.2/40 = 0.304, and a constant "B" scores 0.300. Two of its misses (largest+ocean, smallest ocean) named the right answer in prose without an option+letter, which the scorer does not credit, as designed. The weak reference was at+chance on the old fixture too (7/26 against 8.7/26 expected). It answered both+re-lettered rows correctly; the earlier run did not record per-row outputs, so+whether those two account for its rise from 7 to 9 on the original rows is not+known.++## Bundled-suite check++| Suite | Qwen2.5-1.5B-Instruct | SmolLM2-135M-Instruct |+|---|---:|---:|+| `mini_mmlu` | 0.850 (34/40) | 0.350 (14/40) |+| `mini_common_sense` | 0.850 (34/40) | 0.275 (11/40) |+| `mini_instruction` | 1.000 (24/24) | 0.542 (13/24) |+| `mini_arithmetic` | 0.800 (32/40) | 0.375 (15/40) |+| `mini_tool_call` | 0.700 (28/40) | 0.100 (4/40) |+| `mini_format_json` | 0.975 (39/40) | 0.775 (31/40) |+| `mini_safety` | 1.000 (40/40) | 0.000 (0/40) |+| `mini_over_refusal` | 0.950 (38/40) | 1.000 (40/40) |++Only `mini_mmlu` changed in this fixture. Every other suite matches the+common-sense record on both models to the row.++## Test change outside the new file++Two #357 noise-floor tests in `tests/test_v07302.py` used the lopsided key as+their instrument: a base alternating "B" and "C" passes against a tuned "C"+model, relying on always-B 0.462 vs always-C 0.269 to clear the 0.05+regression threshold. With the balanced key that gap is 0.025, so the tests now+take the most and least frequent letters from the key (0.300 vs 0.150) and+assert the gap still clears 0.05.++## Baseline provenance++`BUNDLED_SCORER_REVISION` moves from 6 to 7 so a baseline stamped at revision 6+warns rather than being compared on the old scale. (Revision 6 is #1958, which+made a failed generation fail every suite; it landed while this change was in+progress.) The deterministic fingerprint does not move: its corpus fixes each+row's correctness by index parity, and 20/40 scores the same as 13/26.++These runs predate #1958, which also lets `soup ship` batch greedy generation.+They generated one prompt at a time, and no generation failed, so the+revision-6 rule would not have changed any score here. Whether batched greedy+decoding reproduces these outputs exactly on MPS was not measured.++## What this record does not establish++- That Qwen2.5-7B-Instruct, or any model at the #1111 strong point, scores+ below 1.000 on the revised fixture.+- That the two kept "A" misses are the trap rather than a letter preference;+ the misconception sat at A in each, so the two cannot be told apart from this+ run.+- The software versions of the run; they were not captured.+- Anything about `mini_format_json`, `mini_safety` or `mini_over_refusal`,+ which remain with #1192.diff --git a/changelog.d/0.76.0/2113.fixed.md b/changelog.d/0.76.0/2113.fixed.mdnew file mode 100644index 00000000..cb029dc4--- /dev/null+++ b/changelog.d/0.76.0/2113.fixed.md@@ -0,0 +1,7 @@+- **`mini_mmlu` no longer sits at a 1.000 ceiling (#1192 by @AaronProbha18 in #2113).**+ The suite grows from 26 to 40 rows with 14 four-option questions, many with a+ common misconception as a distractor; Qwen2.5-1.5B-Instruct measures 0.850+ against 0.350 on SmolLM2-135M-Instruct. The answer key no longer favours one+ letter, so a model that always answers "B" drops from 0.462 to 0.300, and the+ bundled scorer provenance revision moves to 7 so pre-change baselines are not+ silently compared on the new scale.diff --git a/src/soup_cli/eval/forgetting.py b/src/soup_cli/eval/forgetting.pyindex edbb3fc3..dce2bf88 100644--- a/src/soup_cli/eval/forgetting.py+++ b/src/soup_cli/eval/forgetting.py@@ -219,11 +219,11 @@ MINI_MMLU: MiniBenchmark = ["answer": "C"},{"question": "Light travels faster than: (A) sound (B) nothing (C) radio waves","answer": "A"},- {"question": "The square root of 81 is: (A) 8 (B) 9 (C) 18", "answer": "B"},+ {"question": "The square root of 81 is: (A) 9 (B) 8 (C) 18", "answer": "A"},{"question": "Which gas do plants release during photosynthesis? ""(A) oxygen (B) hydrogen (C) methane", "answer": "A"},- {"question": "The author of 'Romeo and Juliet' is: (A) Dickens (B) Shakespeare (C) Austen",- "answer": "B"},+ {"question": "The author of 'Romeo and Juliet' is: (A) Dickens (B) Austen (C) Shakespeare",+ "answer": "C"},{"question": "The freezing point of water in Fahrenheit is: (A) 0 (B) 32 (C) 100","answer": "B"},{"question": "Which ocean is the largest? (A) Atlantic (B) Indian (C) Pacific",@@ -244,6 +244,50 @@ MINI_MMLU: MiniBenchmark = [{"question": "The primary gas in Earth's atmosphere is: (A) oxygen (B) nitrogen (C) argon","answer": "B"},{"question": "Sound cannot travel through a: (A) solid (B) liquid (C) vacuum", "answer": "C"},+ # #1192 — the 26 rows above put a capable model at 1.000 (Qwen2.5-7B-Instruct,+ # 26/26 in the #1111 tool-call discrimination record; Qwen2.5-1.5B-Instruct+ # too). The rows below are 4-option, many with a common misconception as a+ # distractor, selected from two 24-row candidate pools measured on two+ # models. Two rows above had options swapped so no letter answers more+ # than 30% of the suite.+ {"question": "Which planet has the shortest day (fastest rotation)? (A) Mercury (B) Earth "+ "(C) Saturn (D) Jupiter",+ "answer": "D"},+ {"question": "The interior angles of a hexagon add up to: (A) 540 degrees (B) 900 degrees "+ "(C) 720 degrees (D) 1080 degrees",+ "answer": "C"},+ {"question": "Which country has the longest coastline? (A) Russia (B) Australia (C) Canada "+ "(D) Indonesia",+ "answer": "C"},+ {"question": "How many hearts does an octopus have? (A) 3 (B) 1 (C) 2 (D) 4",+ "answer": "A"},+ {"question": "Measured from base to peak, the tallest mountain on Earth is: (A) Mount "+ "Everest (B) Mauna Kea (C) K2 (D) Kilimanjaro",+ "answer": "B"},+ {"question": "In computing, a nibble is how many bits? (A) 4 (B) 2 (C) 8 (D) 16",+ "answer": "A"},+ {"question": "In which phase of mitosis do sister chromatids separate? (A) prophase (B) "+ "metaphase (C) telophase (D) anaphase",+ "answer": "D"},+ {"question": "Which planet has the highest average surface temperature? (A) Mercury (B) "+ "Venus (C) Mars (D) Jupiter",+ "answer": "B"},+ {"question": "Which element has the chemical symbol K? (A) krypton (B) calcium (C) "+ "phosphorus (D) potassium",+ "answer": "D"},+ {"question": "Which is the smallest of the world's five oceans? (A) the Southern (B) the "+ "Indian (C) the Atlantic (D) the Arctic",+ "answer": "D"},+ {"question": "The largest moon of Saturn is: (A) Ganymede (B) Europa (C) Titan (D) Callisto",+ "answer": "C"},+ {"question": "The hardest substance in the human body is: (A) tooth enamel (B) bone (C) "+ "cartilage (D) fingernail",+ "answer": "A"},+ {"question": "The Rock of Gibraltar is a territory of: (A) Spain (B) Morocco (C) Portugal "+ "(D) the United Kingdom",+ "answer": "D"},+ {"question": "The capital of Turkey is: (A) Istanbul (B) Izmir (C) Antalya (D) Ankara",+ "answer": "D"},]MINI_COMMON_SENSE: MiniBenchmark = [diff --git a/src/soup_cli/eval/gate_suites.py b/src/soup_cli/eval/gate_suites.pyindex 7a0f963e..65bde3e4 100644--- a/src/soup_cli/eval/gate_suites.py+++ b/src/soup_cli/eval/gate_suites.py@@ -95,16 +95,18 @@ DEFAULT_GENERAL_SUITE: Tuple[str, ...] = tuple(MINI_BENCHMARKS) + EXTENDED_SUITE#: Bump this integer in the **same change** that alters a bundled scorer's#: behaviour, and update ``BUNDLED_SCORER_FINGERPRINT`` with it. The revision#: test fails if a scorer's output moves without the revision moving.-BUNDLED_SCORER_REVISION: int = 6+BUNDLED_SCORER_REVISION: int = 7#: SHA-256 of deterministic ``score_bundled_suite`` outputs under the fixed-#: fingerprint corpus in ``bundled_scorer_fingerprint``. Locked to revision 6;+#: fingerprint corpus in ``bundled_scorer_fingerprint``. Locked to revision 7;#: update together with ``BUNDLED_SCORER_REVISION``. Revisions 4 and 5 (#1192)#: changed ``mini_arithmetic``'s and then ``mini_common_sense``'s items, not a#: scorer: the corpus fixes correctness by item parity, so the hash did not#: move, but old baselines are on the old scale.#: Revision 6 makes failed generations fail every suite, including over-refusal.#: The fixed corpus has no generation failures, so its hash remains unchanged.+#: Revision 7 (#1192) changed ``mini_mmlu``'s items; the hash holds for the+#: same parity reason as revisions 4 and 5.BUNDLED_SCORER_FINGERPRINT: str = ("80992ac0369233b36d82e41c6f2baff510f6331a6647329fa139c4a99259d5c4")diff --git a/tests/test_issue1192_mmlu_discrimination.py b/tests/test_issue1192_mmlu_discrimination.pynew file mode 100644index 00000000..4675ba1a--- /dev/null+++ b/tests/test_issue1192_mmlu_discrimination.py@@ -0,0 +1,113 @@+"""#1192 — bundled mini_mmlu must discriminate away from the 1.000 rail."""++from __future__ import annotations++import math+import re+from collections import Counter++import pytest++from soup_cli.eval.forgetting import MINI_MMLU++# Opening words of the 14 selected candidate rows, in fixture order. Dropping+# any of them (or restoring the v0.73.2 three-option set) fails this file.+_HARD_OPENINGS = (+ "Which planet has the shortest day",+ "The interior angles of a hexagon",+ "Which country has the longest coastline?",+ "How many hearts does an octopus have?",+ "Measured from base to peak",+ "In computing, a nibble",+ "In which phase of mitosis",+ "Which planet has the highest average surface temperature?",+ "Which element has the chemical symbol K?",+ "Which is the smallest of the world's five oceans?",+ "The largest moon of Saturn",+ "The hardest substance in the human body",+ "The Rock of Gibraltar",+ "The capital of Turkey",+)+++class TestMmluFixtureHasHeadroom:+ def test_fixture_has_40_rows(self):+ assert len(MINI_MMLU) == 40++ def test_the_last_14_rows_are_the_selected_hard_set(self):+ for item, opening in zip(MINI_MMLU[-14:], _HARD_OPENINGS, strict=True):+ assert item["question"].startswith(opening), item++ def test_hard_rows_offer_four_options(self):+ for item in MINI_MMLU[-14:]:+ assert "(D)" in item["question"], item++ def test_questions_are_unique(self):+ questions = [item["question"] for item in MINI_MMLU]+ assert len(questions) == len(set(questions))+++class TestAnswerKeyIsNotLopsided:+ def test_no_letter_answers_more_than_30_percent(self):+ counts = Counter(item["answer"] for item in MINI_MMLU)+ assert max(counts.values()) / len(MINI_MMLU) <= 0.30, counts++ @pytest.mark.parametrize("letter", "ABCD")+ def test_a_constant_letter_model_cannot_score_above_30_percent(self, letter):+ from soup_cli.eval.gate_suites import score_bundled_suite++ assert score_bundled_suite("mini_mmlu", lambda _p: letter) <= 0.30+++def test_fixture_change_bumps_baseline_provenance_revision():+ from soup_cli.eval.gate_suites import BUNDLED_SCORER_REVISION++ assert BUNDLED_SCORER_REVISION >= 7+++_OPTION_RE = re.compile(r"\(([A-D])\) (.*?)(?= \([A-D]\) |$)")++# The TEXT of the correct option for every row, in fixture order. The key is a+# letter, so the test reads the option that letter points at: a key moved to the+# wrong option, or options re-lettered without their key, fail here. The+# arithmetic rows are computed.+_GOLD_TEXT = (+ str(2 + 2), "Paris", "0C", "carbon dioxide", "1", "Jupiter", "Au", "7",+ "mitochondrion", "sound", str(math.isqrt(81)), "oxygen", "Shakespeare", "32",+ "Pacific", "12", "yen", "mercury", "300,000 km/s", "3", "Sun",+ "genetic information", "Everest", "13", "nitrogen", "vacuum",+ "Jupiter", f"{(6 - 2) * 180} degrees", "Canada", "3", "Mauna Kea", str(8 // 2),+ "anaphase", "Venus", "potassium", "the Arctic", "Titan", "tooth enamel",+ "the United Kingdom", "Ankara",+)++# What each of the 14 new rows asks, up to its option list. Opening words alone+# let the question change under its key.+_HARD_STEMS = (+ "Which planet has the shortest day (fastest rotation)?",+ "The interior angles of a hexagon add up to:",+ "Which country has the longest coastline?",+ "How many hearts does an octopus have?",+ "Measured from base to peak, the tallest mountain on Earth is:",+ "In computing, a nibble is how many bits?",+ "In which phase of mitosis do sister chromatids separate?",+ "Which planet has the highest average surface temperature?",+ "Which element has the chemical symbol K?",+ "Which is the smallest of the world's five oceans?",+ "The largest moon of Saturn is:",+ "The hardest substance in the human body is:",+ "The Rock of Gibraltar is a territory of:",+ "The capital of Turkey is:",+)+++def test_every_key_points_at_the_correct_option_text():+ for item, gold in zip(MINI_MMLU, _GOLD_TEXT, strict=True):+ options = dict(_OPTION_RE.findall(item["question"]))+ assert len(set(options.values())) == len(options) >= 3, item+ assert options.get(item["answer"]) == gold, item+++def test_every_hard_row_asks_the_question_its_key_answers():+ for item, stem in zip(MINI_MMLU[-14:], _HARD_STEMS, strict=True):+ assert item["question"].startswith(stem + " (A) "), itemdiff --git a/tests/test_v07302.py b/tests/test_v07302.pyindex d50f084b..b2080049 100644--- a/tests/test_v07302.py+++ b/tests/test_v07302.py@@ -1427,6 +1427,22 @@ class TestIssue317ThroughTheRealCli:assert [d["name"] for d in verdict["benchmark_deltas"] if d["regressed"]] == []+def _mmlu_high_low_letters() -> tuple[str, str]:+ """The most and least frequent mini_mmlu answer letters.++ A constant-letter model scores each letter's share of the key, so these two+ give the widest gap the fixture allows (#1192 balanced the key; the gap must+ still clear the 0.05 regression threshold for the tests below to mean+ anything, which the assertion pins).+ """+ from collections import Counter++ counts = Counter(item["answer"] for item in MINI_BENCHMARKS["mini_mmlu"])+ (high, n_high), *_, (low, n_low) = counts.most_common()+ assert (n_high - n_low) / len(MINI_BENCHMARKS["mini_mmlu"]) > 0.05, counts+ return high, low++class TestNoiseFloorThroughTheRealCli:"""H1 — ``--noise-floor`` must be wired all the way through``_verdict_live``, not just validated as a flag."""@@ -1458,6 +1474,7 @@ class TestNoiseFloorThroughTheRealCli:0.05 threshold, on an instrument whose own repeats move MORE than that,must not be called a regression."""n_items = len(MINI_BENCHMARKS["mini_mmlu"])+ high, low = _mmlu_high_low_letters()state = {"n": 0}def noisy_base(prompt):@@ -1465,12 +1482,12 @@ class TestNoiseFloorThroughTheRealCli:return "hi"state["n"] += 1# Alternate WHOLE passes so the two repeats genuinely disagree.- return "B" if ((state["n"] - 1) // n_items) % 2 == 0 else "C"+ return high if ((state["n"] - 1) // n_items) % 2 == 0 else lowdef dropped_tuned(prompt):if prompt == "say hi":return "hi hi"- return "C"+ return lowwith_floor, v_floor = _run_ship_cli(monkeypatch, noisy_base, dropped_tuned,@@ -1485,18 +1502,19 @@ class TestNoiseFloorThroughTheRealCli:"""CONTROL for the test above — proves the floor is what changed theanswer and not the generators. Same base/tuned, no --noise-floor."""n_items = len(MINI_BENCHMARKS["mini_mmlu"])+ high, low = _mmlu_high_low_letters()state = {"n": 0}def noisy_base(prompt):if prompt == "say hi":return "hi"state["n"] += 1- return "B" if ((state["n"] - 1) // n_items) % 2 == 0 else "C"+ return high if ((state["n"] - 1) // n_items) % 2 == 0 else lowdef dropped_tuned(prompt):if prompt == "say hi":return "hi hi"- return "C"+ return lowresult, verdict = _run_ship_cli(monkeypatch, noisy_base, dropped_tuned,