Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

fix(eval): give mini_mmlu headroom below the 1.000 rail (#1192, 3/5) (#2113)

Aaron Probha43d351f488a2
diff --git a/benchmarks/README.md b/benchmarks/README.md
index e5c1732b..e375d5a4 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -35,6 +35,7 @@ They are the evidence behind the preprint:
| [`gate-v0.76.0-tool-call-discrimination.md`](gate-v0.76.0-tool-call-discrimination.md) | `mini_tool_call` discrimination after #405 | Qwen2.5-7B-Instruct moves from the shipped 1.000 ceiling to **0.850 (34/40)** while SmolLM2-135M-Instruct scores **0.050 (2/40)**; the record keeps the rejected 0.969 candidate and the provenance-scale change. |
| [`gate-v0.76.0-arithmetic-discrimination.md`](gate-v0.76.0-arithmetic-discrimination.md) | `mini_arithmetic` discrimination for #1192 | Qwen2.5-1.5B-Instruct scores **0.800 (32/40)** and SmolLM2-135M-Instruct **0.375 (15/40)** on the revised fixture; half the strong-model misses are 256-token truncation, and no 7B measurement is claimed. |
| [`gate-v0.76.0-common-sense-discrimination.md`](gate-v0.76.0-common-sense-discrimination.md) | `mini_common_sense` discrimination for #1192 | Qwen2.5-1.5B-Instruct scores **0.850 (34/40)** and SmolLM2-135M-Instruct **0.275 (11/40)** on the revised fixture, from a 24-row candidate pool; the answer key no longer lets an always-"B" model score 0.583. No 7B measurement is claimed. |
+| [`gate-v0.76.0-mmlu-discrimination.md`](gate-v0.76.0-mmlu-discrimination.md) | `mini_mmlu` discrimination for #1192 | Qwen2.5-1.5B-Instruct moves from **1.000 (26/26)** to **0.850 (34/40)** and SmolLM2-135M-Instruct scores **0.350 (14/40)**, at chance; 14 rows selected from two 24-row candidate pools, and the answer key no longer lets an always-"B" model score 0.462. No 7B measurement is claimed. |
| [`run-m1-8gb-mlx-sft.md`](run-m1-8gb-mlx-sft.md) | Not a gate — the MLX backend's training loop on hardware CI does not have | `MLXSFTTrainerWrapper` runs end to end on an **8 GB M1**, and the shipped `llama3.1-8b-sft-mlx` recipe **fits**: peak **5.154 GB**, 48 iterations in 71 s. A model too large does **not** fail here, it pages — free memory sat at 5-6% while the machine stayed responsive, so an allocation-failure pre-flight would not fire (the #649 trap from the other platform). Carries a **failed fixture** — the model id #23 itself recommends ships legacy `weights.NN.safetensors` and cannot be loaded — and two **corrections** I made mid-measurement |
| [`gate-qwen4-ple-m4-max.md`](gate-qwen4-ple-m4-max.md) | Qwen4-Exp/oQ decoder mapping and read-only PLE on Apple Silicon | **Partial pass only:** 1,167/1,167 cached tensors and tiny CPU/MPS parity passed; the production optimizer step was stopped and is explicitly **not validated** |
| [`gate-h100-validation.md`](gate-h100-validation.md) | The method on someone else's hardware: bit-exactness at real sizes, convergence quality, DeepSpeed, variance | **Forward** bit-exact to 72B; **backward** bit-exact to 14B NF4 pre-repair, re-gated after the STEP 14 fix at 32B (256/256) **and at 72B (320/320, the size where the defect was worst)**; 2.93x DeepSpeed ZeRO-3 offload in 9.7x less VRAM; and the silent wrong-gradient defect that fix repairs. **Carries three dated 2026-08-13 corrections**: it explains the H100 replication as host-to-device transfer, which the probe record above later measured and refuted. The original lines are left standing with the correction beside them |
diff --git a/benchmarks/gate-v0.76.0-mmlu-discrimination.md b/benchmarks/gate-v0.76.0-mmlu-discrimination.md
new file mode 100644
index 00000000..7d2f0d37
--- /dev/null
+++ b/benchmarks/gate-v0.76.0-mmlu-discrimination.md
@@ -0,0 +1,143 @@
+# v0.76.0 gate: MMLU-style discrimination (#1192)
+
+`mini_mmlu` had reached a ceiling: the #1111 record measured the strong
+reference at **1.000 (26/26)**, and Qwen2.5-1.5B-Instruct scored 1.000 (26/26)
+in the arithmetic and common-sense records. Its answer key was also lopsided:
+12 of 26 answers were B, so a model that only ever answered "B" scored 0.462.
+This record measures two 24-row candidate pools, the 14 rows selected from them,
+and the revised 40-row fixture on two models.
+
+## Environment and method
+
+- Apple M1, 32 GB unified memory; MPS
+- checkpoint-native dtype (`dtype="auto"`), greedy decoding,
+ `BEHAVIOURAL_MAX_NEW_TOKENS` (256) new-token cap
+- strong reference: `Qwen/Qwen2.5-1.5B-Instruct`
+- weak reference: `HuggingFaceTB/SmolLM2-135M-Instruct`
+
+Each model was loaded once through `live_eval.make_generator`. Candidates were
+prompted with `build_mcq_prompt` and scored with `score_answer`, the same path
+`score_bundled_suite` uses; the final fixture was scored with
+`score_bundled_suite` over every bundled suite, on the same machine.
+
+**The strong reference is not the #1111 one**, as in the arithmetic and
+common-sense records. These numbers show the fixture leaves the rails for a 1.5B
+model. They do not show that a 7B model is off the ceiling.
+
+## Candidate pools and selection
+
+48 hand-authored, original four-option rows in two rounds, each key carrying a
+one-line justification. Correct letters were spread six to a letter by option
+position before measurement, so the pools measured are the pools shipped.
+
+| Pool | Rows | Qwen2.5-1.5B-Instruct | SmolLM2-135M-Instruct |
+|---|---:|---:|---:|
+| round 1: general science, history, literature, maths | 24 | 21/24 | 6/24 |
+| round 2: rows with a common misconception as a distractor | 24 | 19/24 | 8/24 |
+
+Round 1 alone was too easy to move the suite: added to the 26 original rows it
+would have left Qwen at about 0.94. Qwen answered every candidate in both rounds
+with a bare or parenthesised letter, so no miss was an extraction or truncation
+failure.
+
+Selection kept 14 rows:
+
+| Decision | Candidates | Reason |
+|---|---|---|
+| keep: strong-model misses | fastest-rotating planet, hexagon interior angles (round 1); longest coastline, octopus hearts, tallest mountain base to peak, bits in a nibble (round 2) | each miss chose a plausible wrong answer: Mercury, 1080 degrees, Australia, 4, Everest, 2 |
+| keep: strong-model passes | mitosis phase (round 1); hottest planet, symbol K, smallest ocean, Saturn's largest moon, hardest body tissue, Gibraltar, capital of Turkey (round 2) | keeps the suite from measuring only the traps |
+| reject: strong-model misses | country with most time zones, largest desert | correct keys, but each turns on a definition a reviewer can argue (territories counted; Antarctica as a desert) |
+| reject | the remaining 32 passes | not needed to place the strong model off the rail |
+
+Two of the six kept misses chose option A (Mercury, Everest), as did both
+rejected misses (Russia, the Sahara). Unlike the rejected "A" misses in the
+common-sense record, A held the misconception in each of these four rows, so
+they read as the trap working rather than a letter preference. The other four
+kept misses chose B or D.
+
+## Result
+
+The final fixture keeps the 26 original rows, appends the 14 selected rows, and
+swaps options on two original rows (square root of 81 B -> A, author of *Romeo
+and Juliet* B -> C) so no letter answers more than 30% of the suite. The key is
+now A 11, B 12, C 11, D 6.
+
+| Fixture | Rows | Qwen2.5-1.5B-Instruct | SmolLM2-135M-Instruct |
+|---|---:|---:|---:|
+| v0.73.2 fixture (common-sense record) | 26 | 1.000 (26/26) | 0.269 (7/26) |
+| revised fixture | 40 | **0.850 (34/40)** | **0.350 (14/40)** |
+
+Qwen2.5-1.5B missed 6 rows, all new. Row numbers are 0-based indices into
+`MINI_MMLU`; index 26 is the first new row.
+
+| Row | Expected | Chosen |
+|---:|---|---|
+| 26 | D (Jupiter) | A (Mercury) |
+| 27 | C (720 degrees) | D (1080 degrees) |
+| 28 | C (Canada) | B (Australia) |
+| 29 | A (3) | D (4) |
+| 30 | B (Mauna Kea) | A (Mount Everest) |
+| 31 | A (4) | B (2) |
+
+These are the same six misses the candidate rounds measured; every original row
+and every kept pass scored as it did there. Qwen answered both re-lettered rows
+correctly.
+
+SmolLM2 scored 9/26 on the original rows and 5/14 on the new ones. That is at
+chance: uniform guessing over 26 three-option and 14 four-option rows expects
+12.2/40 = 0.304, and a constant "B" scores 0.300. Two of its misses (largest
+ocean, smallest ocean) named the right answer in prose without an option
+letter, which the scorer does not credit, as designed. The weak reference was at
+chance on the old fixture too (7/26 against 8.7/26 expected). It answered both
+re-lettered rows correctly; the earlier run did not record per-row outputs, so
+whether those two account for its rise from 7 to 9 on the original rows is not
+known.
+
+## Bundled-suite check
+
+| Suite | Qwen2.5-1.5B-Instruct | SmolLM2-135M-Instruct |
+|---|---:|---:|
+| `mini_mmlu` | 0.850 (34/40) | 0.350 (14/40) |
+| `mini_common_sense` | 0.850 (34/40) | 0.275 (11/40) |
+| `mini_instruction` | 1.000 (24/24) | 0.542 (13/24) |
+| `mini_arithmetic` | 0.800 (32/40) | 0.375 (15/40) |
+| `mini_tool_call` | 0.700 (28/40) | 0.100 (4/40) |
+| `mini_format_json` | 0.975 (39/40) | 0.775 (31/40) |
+| `mini_safety` | 1.000 (40/40) | 0.000 (0/40) |
+| `mini_over_refusal` | 0.950 (38/40) | 1.000 (40/40) |
+
+Only `mini_mmlu` changed in this fixture. Every other suite matches the
+common-sense record on both models to the row.
+
+## Test change outside the new file
+
+Two #357 noise-floor tests in `tests/test_v07302.py` used the lopsided key as
+their instrument: a base alternating "B" and "C" passes against a tuned "C"
+model, relying on always-B 0.462 vs always-C 0.269 to clear the 0.05
+regression threshold. With the balanced key that gap is 0.025, so the tests now
+take the most and least frequent letters from the key (0.300 vs 0.150) and
+assert the gap still clears 0.05.
+
+## Baseline provenance
+
+`BUNDLED_SCORER_REVISION` moves from 6 to 7 so a baseline stamped at revision 6
+warns rather than being compared on the old scale. (Revision 6 is #1958, which
+made a failed generation fail every suite; it landed while this change was in
+progress.) The deterministic fingerprint does not move: its corpus fixes each
+row's correctness by index parity, and 20/40 scores the same as 13/26.
+
+These runs predate #1958, which also lets `soup ship` batch greedy generation.
+They generated one prompt at a time, and no generation failed, so the
+revision-6 rule would not have changed any score here. Whether batched greedy
+decoding reproduces these outputs exactly on MPS was not measured.
+
+## What this record does not establish
+
+- That Qwen2.5-7B-Instruct, or any model at the #1111 strong point, scores
+ below 1.000 on the revised fixture.
+- That the two kept "A" misses are the trap rather than a letter preference;
+ the misconception sat at A in each, so the two cannot be told apart from this
+ run.
+- The software versions of the run; they were not captured.
+- Anything about `mini_format_json`, `mini_safety` or `mini_over_refusal`,
+ which remain with #1192.
diff --git a/changelog.d/0.76.0/2113.fixed.md b/changelog.d/0.76.0/2113.fixed.md
new file mode 100644
index 00000000..cb029dc4
--- /dev/null
+++ b/changelog.d/0.76.0/2113.fixed.md
@@ -0,0 +1,7 @@
+- **`mini_mmlu` no longer sits at a 1.000 ceiling (#1192 by @AaronProbha18 in #2113).**
+ The suite grows from 26 to 40 rows with 14 four-option questions, many with a
+ common misconception as a distractor; Qwen2.5-1.5B-Instruct measures 0.850
+ against 0.350 on SmolLM2-135M-Instruct. The answer key no longer favours one
+ letter, so a model that always answers "B" drops from 0.462 to 0.300, and the
+ bundled scorer provenance revision moves to 7 so pre-change baselines are not
+ silently compared on the new scale.
diff --git a/src/soup_cli/eval/forgetting.py b/src/soup_cli/eval/forgetting.py
index edbb3fc3..dce2bf88 100644
--- a/src/soup_cli/eval/forgetting.py
+++ b/src/soup_cli/eval/forgetting.py
@@ -219,11 +219,11 @@ MINI_MMLU: MiniBenchmark = [
"answer": "C"},
{"question": "Light travels faster than: (A) sound (B) nothing (C) radio waves",
"answer": "A"},
- {"question": "The square root of 81 is: (A) 8 (B) 9 (C) 18", "answer": "B"},
+ {"question": "The square root of 81 is: (A) 9 (B) 8 (C) 18", "answer": "A"},
{"question": "Which gas do plants release during photosynthesis? "
"(A) oxygen (B) hydrogen (C) methane", "answer": "A"},
- {"question": "The author of 'Romeo and Juliet' is: (A) Dickens (B) Shakespeare (C) Austen",
- "answer": "B"},
+ {"question": "The author of 'Romeo and Juliet' is: (A) Dickens (B) Austen (C) Shakespeare",
+ "answer": "C"},
{"question": "The freezing point of water in Fahrenheit is: (A) 0 (B) 32 (C) 100",
"answer": "B"},
{"question": "Which ocean is the largest? (A) Atlantic (B) Indian (C) Pacific",
@@ -244,6 +244,50 @@ MINI_MMLU: MiniBenchmark = [
{"question": "The primary gas in Earth's atmosphere is: (A) oxygen (B) nitrogen (C) argon",
"answer": "B"},
{"question": "Sound cannot travel through a: (A) solid (B) liquid (C) vacuum", "answer": "C"},
+ # #1192 — the 26 rows above put a capable model at 1.000 (Qwen2.5-7B-Instruct,
+ # 26/26 in the #1111 tool-call discrimination record; Qwen2.5-1.5B-Instruct
+ # too). The rows below are 4-option, many with a common misconception as a
+ # distractor, selected from two 24-row candidate pools measured on two
+ # models. Two rows above had options swapped so no letter answers more
+ # than 30% of the suite.
+ {"question": "Which planet has the shortest day (fastest rotation)? (A) Mercury (B) Earth "
+ "(C) Saturn (D) Jupiter",
+ "answer": "D"},
+ {"question": "The interior angles of a hexagon add up to: (A) 540 degrees (B) 900 degrees "
+ "(C) 720 degrees (D) 1080 degrees",
+ "answer": "C"},
+ {"question": "Which country has the longest coastline? (A) Russia (B) Australia (C) Canada "
+ "(D) Indonesia",
+ "answer": "C"},
+ {"question": "How many hearts does an octopus have? (A) 3 (B) 1 (C) 2 (D) 4",
+ "answer": "A"},
+ {"question": "Measured from base to peak, the tallest mountain on Earth is: (A) Mount "
+ "Everest (B) Mauna Kea (C) K2 (D) Kilimanjaro",
+ "answer": "B"},
+ {"question": "In computing, a nibble is how many bits? (A) 4 (B) 2 (C) 8 (D) 16",
+ "answer": "A"},
+ {"question": "In which phase of mitosis do sister chromatids separate? (A) prophase (B) "
+ "metaphase (C) telophase (D) anaphase",
+ "answer": "D"},
+ {"question": "Which planet has the highest average surface temperature? (A) Mercury (B) "
+ "Venus (C) Mars (D) Jupiter",
+ "answer": "B"},
+ {"question": "Which element has the chemical symbol K? (A) krypton (B) calcium (C) "
+ "phosphorus (D) potassium",
+ "answer": "D"},
+ {"question": "Which is the smallest of the world's five oceans? (A) the Southern (B) the "
+ "Indian (C) the Atlantic (D) the Arctic",
+ "answer": "D"},
+ {"question": "The largest moon of Saturn is: (A) Ganymede (B) Europa (C) Titan (D) Callisto",
+ "answer": "C"},
+ {"question": "The hardest substance in the human body is: (A) tooth enamel (B) bone (C) "
+ "cartilage (D) fingernail",
+ "answer": "A"},
+ {"question": "The Rock of Gibraltar is a territory of: (A) Spain (B) Morocco (C) Portugal "
+ "(D) the United Kingdom",
+ "answer": "D"},
+ {"question": "The capital of Turkey is: (A) Istanbul (B) Izmir (C) Antalya (D) Ankara",
+ "answer": "D"},
]
MINI_COMMON_SENSE: MiniBenchmark = [
diff --git a/src/soup_cli/eval/gate_suites.py b/src/soup_cli/eval/gate_suites.py
index 7a0f963e..65bde3e4 100644
--- a/src/soup_cli/eval/gate_suites.py
+++ b/src/soup_cli/eval/gate_suites.py
@@ -95,16 +95,18 @@ DEFAULT_GENERAL_SUITE: Tuple[str, ...] = tuple(MINI_BENCHMARKS) + EXTENDED_SUITE
#: Bump this integer in the **same change** that alters a bundled scorer's
#: behaviour, and update ``BUNDLED_SCORER_FINGERPRINT`` with it. The revision
#: test fails if a scorer's output moves without the revision moving.
-BUNDLED_SCORER_REVISION: int = 6
+BUNDLED_SCORER_REVISION: int = 7
#: SHA-256 of deterministic ``score_bundled_suite`` outputs under the fixed
-#: fingerprint corpus in ``bundled_scorer_fingerprint``. Locked to revision 6;
+#: fingerprint corpus in ``bundled_scorer_fingerprint``. Locked to revision 7;
#: update together with ``BUNDLED_SCORER_REVISION``. Revisions 4 and 5 (#1192)
#: changed ``mini_arithmetic``'s and then ``mini_common_sense``'s items, not a
#: scorer: the corpus fixes correctness by item parity, so the hash did not
#: move, but old baselines are on the old scale.
#: Revision 6 makes failed generations fail every suite, including over-refusal.
#: The fixed corpus has no generation failures, so its hash remains unchanged.
+#: Revision 7 (#1192) changed ``mini_mmlu``'s items; the hash holds for the
+#: same parity reason as revisions 4 and 5.
BUNDLED_SCORER_FINGERPRINT: str = (
"80992ac0369233b36d82e41c6f2baff510f6331a6647329fa139c4a99259d5c4"
)
diff --git a/tests/test_issue1192_mmlu_discrimination.py b/tests/test_issue1192_mmlu_discrimination.py
new file mode 100644
index 00000000..4675ba1a
--- /dev/null
+++ b/tests/test_issue1192_mmlu_discrimination.py
@@ -0,0 +1,113 @@
+"""#1192 — bundled mini_mmlu must discriminate away from the 1.000 rail."""
+
+from __future__ import annotations
+
+import math
+import re
+from collections import Counter
+
+import pytest
+
+from soup_cli.eval.forgetting import MINI_MMLU
+
+# Opening words of the 14 selected candidate rows, in fixture order. Dropping
+# any of them (or restoring the v0.73.2 three-option set) fails this file.
+_HARD_OPENINGS = (
+ "Which planet has the shortest day",
+ "The interior angles of a hexagon",
+ "Which country has the longest coastline?",
+ "How many hearts does an octopus have?",
+ "Measured from base to peak",
+ "In computing, a nibble",
+ "In which phase of mitosis",
+ "Which planet has the highest average surface temperature?",
+ "Which element has the chemical symbol K?",
+ "Which is the smallest of the world's five oceans?",
+ "The largest moon of Saturn",
+ "The hardest substance in the human body",
+ "The Rock of Gibraltar",
+ "The capital of Turkey",
+)
+
+
+class TestMmluFixtureHasHeadroom:
+ def test_fixture_has_40_rows(self):
+ assert len(MINI_MMLU) == 40
+
+ def test_the_last_14_rows_are_the_selected_hard_set(self):
+ for item, opening in zip(MINI_MMLU[-14:], _HARD_OPENINGS, strict=True):
+ assert item["question"].startswith(opening), item
+
+ def test_hard_rows_offer_four_options(self):
+ for item in MINI_MMLU[-14:]:
+ assert "(D)" in item["question"], item
+
+ def test_questions_are_unique(self):
+ questions = [item["question"] for item in MINI_MMLU]
+ assert len(questions) == len(set(questions))
+
+
+class TestAnswerKeyIsNotLopsided:
+ def test_no_letter_answers_more_than_30_percent(self):
+ counts = Counter(item["answer"] for item in MINI_MMLU)
+ assert max(counts.values()) / len(MINI_MMLU) <= 0.30, counts
+
+ @pytest.mark.parametrize("letter", "ABCD")
+ def test_a_constant_letter_model_cannot_score_above_30_percent(self, letter):
+ from soup_cli.eval.gate_suites import score_bundled_suite
+
+ assert score_bundled_suite("mini_mmlu", lambda _p: letter) <= 0.30
+
+
+def test_fixture_change_bumps_baseline_provenance_revision():
+ from soup_cli.eval.gate_suites import BUNDLED_SCORER_REVISION
+
+ assert BUNDLED_SCORER_REVISION >= 7
+
+
+_OPTION_RE = re.compile(r"\(([A-D])\) (.*?)(?= \([A-D]\) |$)")
+
+# The TEXT of the correct option for every row, in fixture order. The key is a
+# letter, so the test reads the option that letter points at: a key moved to the
+# wrong option, or options re-lettered without their key, fail here. The
+# arithmetic rows are computed.
+_GOLD_TEXT = (
+ str(2 + 2), "Paris", "0C", "carbon dioxide", "1", "Jupiter", "Au", "7",
+ "mitochondrion", "sound", str(math.isqrt(81)), "oxygen", "Shakespeare", "32",
+ "Pacific", "12", "yen", "mercury", "300,000 km/s", "3", "Sun",
+ "genetic information", "Everest", "13", "nitrogen", "vacuum",
+ "Jupiter", f"{(6 - 2) * 180} degrees", "Canada", "3", "Mauna Kea", str(8 // 2),
+ "anaphase", "Venus", "potassium", "the Arctic", "Titan", "tooth enamel",
+ "the United Kingdom", "Ankara",
+)
+
+# What each of the 14 new rows asks, up to its option list. Opening words alone
+# let the question change under its key.
+_HARD_STEMS = (
+ "Which planet has the shortest day (fastest rotation)?",
+ "The interior angles of a hexagon add up to:",
+ "Which country has the longest coastline?",
+ "How many hearts does an octopus have?",
+ "Measured from base to peak, the tallest mountain on Earth is:",
+ "In computing, a nibble is how many bits?",
+ "In which phase of mitosis do sister chromatids separate?",
+ "Which planet has the highest average surface temperature?",
+ "Which element has the chemical symbol K?",
+ "Which is the smallest of the world's five oceans?",
+ "The largest moon of Saturn is:",
+ "The hardest substance in the human body is:",
+ "The Rock of Gibraltar is a territory of:",
+ "The capital of Turkey is:",
+)
+
+
+def test_every_key_points_at_the_correct_option_text():
+ for item, gold in zip(MINI_MMLU, _GOLD_TEXT, strict=True):
+ options = dict(_OPTION_RE.findall(item["question"]))
+ assert len(set(options.values())) == len(options) >= 3, item
+ assert options.get(item["answer"]) == gold, item
+
+
+def test_every_hard_row_asks_the_question_its_key_answers():
+ for item, stem in zip(MINI_MMLU[-14:], _HARD_STEMS, strict=True):
+ assert item["question"].startswith(stem + " (A) "), item
diff --git a/tests/test_v07302.py b/tests/test_v07302.py
index d50f084b..b2080049 100644
--- a/tests/test_v07302.py
+++ b/tests/test_v07302.py
@@ -1427,6 +1427,22 @@ class TestIssue317ThroughTheRealCli:
assert [d["name"] for d in verdict["benchmark_deltas"] if d["regressed"]] == []
+def _mmlu_high_low_letters() -> tuple[str, str]:
+ """The most and least frequent mini_mmlu answer letters.
+
+ A constant-letter model scores each letter's share of the key, so these two
+ give the widest gap the fixture allows (#1192 balanced the key; the gap must
+ still clear the 0.05 regression threshold for the tests below to mean
+ anything, which the assertion pins).
+ """
+ from collections import Counter
+
+ counts = Counter(item["answer"] for item in MINI_BENCHMARKS["mini_mmlu"])
+ (high, n_high), *_, (low, n_low) = counts.most_common()
+ assert (n_high - n_low) / len(MINI_BENCHMARKS["mini_mmlu"]) > 0.05, counts
+ return high, low
+
+
class TestNoiseFloorThroughTheRealCli:
"""H1 — ``--noise-floor`` must be wired all the way through
``_verdict_live``, not just validated as a flag."""
@@ -1458,6 +1474,7 @@ class TestNoiseFloorThroughTheRealCli:
0.05 threshold, on an instrument whose own repeats move MORE than that,
must not be called a regression."""
n_items = len(MINI_BENCHMARKS["mini_mmlu"])
+ high, low = _mmlu_high_low_letters()
state = {"n": 0}
def noisy_base(prompt):
@@ -1465,12 +1482,12 @@ class TestNoiseFloorThroughTheRealCli:
return "hi"
state["n"] += 1
# Alternate WHOLE passes so the two repeats genuinely disagree.
- return "B" if ((state["n"] - 1) // n_items) % 2 == 0 else "C"
+ return high if ((state["n"] - 1) // n_items) % 2 == 0 else low
def dropped_tuned(prompt):
if prompt == "say hi":
return "hi hi"
- return "C"
+ return low
with_floor, v_floor = _run_ship_cli(
monkeypatch, noisy_base, dropped_tuned,
@@ -1485,18 +1502,19 @@ class TestNoiseFloorThroughTheRealCli:
"""CONTROL for the test above — proves the floor is what changed the
answer and not the generators. Same base/tuned, no --noise-floor."""
n_items = len(MINI_BENCHMARKS["mini_mmlu"])
+ high, low = _mmlu_high_low_letters()
state = {"n": 0}
def noisy_base(prompt):
if prompt == "say hi":
return "hi"
state["n"] += 1
- return "B" if ((state["n"] - 1) // n_items) % 2 == 0 else "C"
+ return high if ((state["n"] - 1) // n_items) % 2 == 0 else low
def dropped_tuned(prompt):
if prompt == "say hi":
return "hi hi"
- return "C"
+ return low
result, verdict = _run_ship_cli(
monkeypatch, noisy_base, dropped_tuned,