Official Codebase of 3DCodeVerse
Python
bench: the stats and launcher prose points at EVAL §8 instead of restating it
gaoypengccdf609ef370
diff --git a/eval/bench/_ab_report.py b/eval/bench/_ab_report.pyindex 025bb580..20fe3cf0 100644--- a/eval/bench/_ab_report.py+++ b/eval/bench/_ab_report.py@@ -37,19 +37,10 @@ KEEP_DELTA = 0.02REVERT_DELTA = -0.02#: this many regressing prompts is a revert even when the mean looks fineREVERT_REGRESSIONS = 2-# The keep/revert rule above is the protocol (docs/EVAL.md §8) and a report must not be-# able to argue itself out of it — but the rule alone is dangerously confident. Measured-# 2026-08-24 with two A/A runs — arms identical by construction — on the SAME prompt-# (`ctrl_med_dining_chair`, rounds 1):-#-# 0.591 vs 0.934 -> delta +0.344 -> verdict **keep**-# 0.700 vs 0.600 -> delta -0.100 -> verdict **revert**-#-# Nothing was under test either time, and the rule returned opposite decisions. The-# generator is stochastic; a paired delta carries the spread of two independent-# generations, not the judge's ±0.02 sampling noise. So every summary states the paired-# 95 % t-interval beside the verdict (``bench/stats.py``), reported, never enforced, and-# says plainly when the threshold is inside it.+# The rule above is the protocol and no report can argue itself out of it, but alone it is+# dangerously confident: two A/A runs of one prompt printed "keep" (+0.344) and "revert"+# (-0.100) from identical code (docs/EVAL.md §8). So every summary states the paired 95 %+# t-interval and the sign test beside it (bench/stats.py) — reported, never enforced.class PairOutcome(BaseModel):@@ -140,9 +131,8 @@ def verdict_of(pairs: list[PairOutcome]) -> Verdict:return Verdict(decision="inconclusive", n_pairs=0, mean_delta=None, median_delta=None,reason="no prompt has both arms scored")mean, median = round(statistics.fmean(deltas), 4), round(statistics.median(deltas), 4)- # the noise block: reported, never enforced. The SIGN of each delta is far cheaper to- # move than the mean — 7 of 8 in one direction is p = 0.07, a bar an eight-prompt- # battery can clear where a ±0.02 mean at a paired sd near 0.23 never will.+ # the SIGN is far cheaper to move than the mean: 7 of 8 one way is p = 0.07, a bar an+ # eight-prompt battery clears where a ±0.02 mean at a paired sd near 0.23 never willci = mean_ci(deltas)up, down, sign_p = sign_test(deltas)v = Verdict(decision="inconclusive", n_pairs=len(deltas), mean_delta=mean, median_delta=median,diff --git a/eval/bench/run_bench.py b/eval/bench/run_bench.pyindex b0749bd3..ea3ceba4 100644--- a/eval/bench/run_bench.py+++ b/eval/bench/run_bench.py@@ -1,13 +1,9 @@"""Run a prompt battery through a track with N parallel workers; resumable.- cd eval && python -m bench.run_bench bench/prompts/static_objects_v1.yaml \- --generator gemini-cli:gemini-3.7-flash --judge gemini:gemini-3.1-pro-preview [--rounds 2] \- [--parallel 4] [--tier easy] [--id furn_easy_stool] [--limit 6] [--out bench/out/x] \- [--redo-status infra_failed] [--no-report]--(``3dcode bench run`` until 2026-09-22: the launcher lives with the evaluation now, and like-every bench script it imports THIS tree's ``codeverse3d`` — a console script resolved the-editable install's, so a battery launched from a worktree ran the main checkout's code.)+ cd eval && python -m bench.run_bench bench/prompts/<battery>.yaml --generator ... --judge ...++(``3dcode bench run`` until 2026-09-22. Like every bench script it imports THIS tree's+``codeverse3d``; the console script resolved the editable install's, main's from a worktree.)Layout of ``out_dir``::diff --git a/eval/bench/stats.py b/eval/bench/stats.pyindex 536dac2b..84444b32 100644--- a/eval/bench/stats.py+++ b/eval/bench/stats.py@@ -1,18 +1,10 @@-"""The statistics the bench reports state — one rule each, so two reports never disagree-about the same numbers.--* **The interval** is the Student-t one: ``mean ± t(0.975, n−1) · sd/√n``. A battery is- 3–40 pairs, and there the normal ``1.96`` or a round ``2`` is too narrow: at n = 8- (t = 2.365) they understate the 95 % half-width by 17 % / 15 %, at n = 4 (t = 3.182) by- 38 % / 37 %. t tends to 1.96 as n grows, so it is the one rule that is right at both- ends. A change is *separated* from noise when its interval excludes zero.-* **The sign test** is exact and two-sided over the non-zero deltas: how CONSISTENT a- change is, which an 8-prompt battery can show when its mean cannot.-* **Correlation** is ``statistics.correlation`` (Pearson; Spearman with ``ranked``), and- ``None`` below three points or on a constant input, where an r means nothing.--The binomial rate interval (Wilson) and Fisher's exact test stay in-``plan_stage_report.py``, the one report that states a rate.+"""The statistics every bench report states, one rule each (docs/EVAL.md §8).++The interval is Student-t, ``mean ± t(0.975, n−1)·sd/√n``: a battery is 3–40 pairs, where+1.96 or 2 understate the 95 % half-width by 15–17 % at n = 8 and 37–38 % at n = 4, and t+tends to 1.96 as n grows. *Separated* means the interval excludes zero. The sign test is+exact and two-sided; correlation is ``statistics.correlation``. (Wilson and Fisher stay in+``plan_stage_report.py``, the one report that states a rate.)"""from __future__ import annotations@@ -96,6 +88,3 @@ def correlation(xs: Sequence[float], ys: Sequence[float], *, ranked: bool = Falsif len(xs) < 3 or len(set(xs)) < 2 or len(set(ys)) < 2:return Nonereturn statistics.correlation(xs, ys, method="ranked" if ranked else "linear")---__all__ = ["MeanCI", "correlation", "mean_ci", "n_to_resolve", "sign_test", "t975"]