Skip to content

Official Codebase of 3DCodeVerse

Python

bench: the stats and launcher prose points at EVAL §8 instead of restating it

gaoypengccdf609ef370
diff --git a/eval/bench/_ab_report.py b/eval/bench/_ab_report.py
index 025bb580..20fe3cf0 100644
--- a/eval/bench/_ab_report.py
+++ b/eval/bench/_ab_report.py
@@ -37,19 +37,10 @@ KEEP_DELTA = 0.02
REVERT_DELTA = -0.02
#: this many regressing prompts is a revert even when the mean looks fine
REVERT_REGRESSIONS = 2
-# The keep/revert rule above is the protocol (docs/EVAL.md §8) and a report must not be
-# able to argue itself out of it — but the rule alone is dangerously confident. Measured
-# 2026-08-24 with two A/A runs — arms identical by construction — on the SAME prompt
-# (`ctrl_med_dining_chair`, rounds 1):
-#
-# 0.591 vs 0.934 -> delta +0.344 -> verdict **keep**
-# 0.700 vs 0.600 -> delta -0.100 -> verdict **revert**
-#
-# Nothing was under test either time, and the rule returned opposite decisions. The
-# generator is stochastic; a paired delta carries the spread of two independent
-# generations, not the judge's ±0.02 sampling noise. So every summary states the paired
-# 95 % t-interval beside the verdict (``bench/stats.py``), reported, never enforced, and
-# says plainly when the threshold is inside it.
+# The rule above is the protocol and no report can argue itself out of it, but alone it is
+# dangerously confident: two A/A runs of one prompt printed "keep" (+0.344) and "revert"
+# (-0.100) from identical code (docs/EVAL.md §8). So every summary states the paired 95 %
+# t-interval and the sign test beside it (bench/stats.py) — reported, never enforced.
class PairOutcome(BaseModel):
@@ -140,9 +131,8 @@ def verdict_of(pairs: list[PairOutcome]) -> Verdict:
return Verdict(decision="inconclusive", n_pairs=0, mean_delta=None, median_delta=None,
reason="no prompt has both arms scored")
mean, median = round(statistics.fmean(deltas), 4), round(statistics.median(deltas), 4)
- # the noise block: reported, never enforced. The SIGN of each delta is far cheaper to
- # move than the mean — 7 of 8 in one direction is p = 0.07, a bar an eight-prompt
- # battery can clear where a ±0.02 mean at a paired sd near 0.23 never will.
+ # the SIGN is far cheaper to move than the mean: 7 of 8 one way is p = 0.07, a bar an
+ # eight-prompt battery clears where a ±0.02 mean at a paired sd near 0.23 never will
ci = mean_ci(deltas)
up, down, sign_p = sign_test(deltas)
v = Verdict(decision="inconclusive", n_pairs=len(deltas), mean_delta=mean, median_delta=median,
diff --git a/eval/bench/run_bench.py b/eval/bench/run_bench.py
index b0749bd3..ea3ceba4 100644
--- a/eval/bench/run_bench.py
+++ b/eval/bench/run_bench.py
@@ -1,13 +1,9 @@
"""Run a prompt battery through a track with N parallel workers; resumable.
- cd eval && python -m bench.run_bench bench/prompts/static_objects_v1.yaml \
- --generator gemini-cli:gemini-3.7-flash --judge gemini:gemini-3.1-pro-preview [--rounds 2] \
- [--parallel 4] [--tier easy] [--id furn_easy_stool] [--limit 6] [--out bench/out/x] \
- [--redo-status infra_failed] [--no-report]
-
-(``3dcode bench run`` until 2026-09-22: the launcher lives with the evaluation now, and like
-every bench script it imports THIS tree's ``codeverse3d`` — a console script resolved the
-editable install's, so a battery launched from a worktree ran the main checkout's code.)
+ cd eval && python -m bench.run_bench bench/prompts/<battery>.yaml --generator ... --judge ...
+
+(``3dcode bench run`` until 2026-09-22. Like every bench script it imports THIS tree's
+``codeverse3d``; the console script resolved the editable install's, main's from a worktree.)
Layout of ``out_dir``::
diff --git a/eval/bench/stats.py b/eval/bench/stats.py
index 536dac2b..84444b32 100644
--- a/eval/bench/stats.py
+++ b/eval/bench/stats.py
@@ -1,18 +1,10 @@
-"""The statistics the bench reports state — one rule each, so two reports never disagree
-about the same numbers.
-
-* **The interval** is the Student-t one: ``mean ± t(0.975, n−1) · sd/√n``. A battery is
- 3–40 pairs, and there the normal ``1.96`` or a round ``2`` is too narrow: at n = 8
- (t = 2.365) they understate the 95 % half-width by 17 % / 15 %, at n = 4 (t = 3.182) by
- 38 % / 37 %. t tends to 1.96 as n grows, so it is the one rule that is right at both
- ends. A change is *separated* from noise when its interval excludes zero.
-* **The sign test** is exact and two-sided over the non-zero deltas: how CONSISTENT a
- change is, which an 8-prompt battery can show when its mean cannot.
-* **Correlation** is ``statistics.correlation`` (Pearson; Spearman with ``ranked``), and
- ``None`` below three points or on a constant input, where an r means nothing.
-
-The binomial rate interval (Wilson) and Fisher's exact test stay in
-``plan_stage_report.py``, the one report that states a rate.
+"""The statistics every bench report states, one rule each (docs/EVAL.md §8).
+
+The interval is Student-t, ``mean ± t(0.975, n−1)·sd/√n``: a battery is 3–40 pairs, where
+1.96 or 2 understate the 95 % half-width by 15–17 % at n = 8 and 37–38 % at n = 4, and t
+tends to 1.96 as n grows. *Separated* means the interval excludes zero. The sign test is
+exact and two-sided; correlation is ``statistics.correlation``. (Wilson and Fisher stay in
+``plan_stage_report.py``, the one report that states a rate.)
"""
from __future__ import annotations
@@ -96,6 +88,3 @@ def correlation(xs: Sequence[float], ys: Sequence[float], *, ranked: bool = Fals
if len(xs) < 3 or len(set(xs)) < 2 or len(set(ys)) < 2:
return None
return statistics.correlation(xs, ys, method="ranked" if ranked else "linear")
-
-
-__all__ = ["MeanCI", "correlation", "mean_ci", "n_to_resolve", "sign_test", "t975"]