28 eval-informed mental models and critical-thinking skills for Claude Code, GitHub Copilot, Codex, Cursor, and other Agent Skills-compatible tools
JavaScript agent-skillsai-toolsanthropicclaudeclaude-codecritical-thinkingdecision-makingfirst-principlesmental-modelsproblem-solvingsystems-thinking
fix(m5-scorecard): reconcile scientific-method nested replication evidence with top-level DIRECTIONAL-NOT-REPLICATED verdict
Travis Boudreauxe8dee18935fb
diff --git a/analysis/ELEVATE-OR-KILL-SCORECARD.json b/analysis/ELEVATE-OR-KILL-SCORECARD.jsonindex 2ef7bd8..004cbd8 100644--- a/analysis/ELEVATE-OR-KILL-SCORECARD.json+++ b/analysis/ELEVATE-OR-KILL-SCORECARD.json@@ -1221,13 +1221,13 @@"delta_pp": 8,"p": 0.001,"significant": true,- "verdict": "ELEVATE",+ "verdict": "SIGNIFICANT-REPLICATION-SIGNAL","provenance": "OBJ-powered-significant","temporal": "post-edit","significance": "significant",- "replicated": true,+ "replicated": false,"source": "evals/results/m5-repl/swe-scientific-method.json",- "note": "M5 FRESH replication on the ORIGINAL frozen 150-item SWE-bench set. Replicates M5 fresh primary (+5.4pp) in same direction: 90% vs 82% placebo, +8.0pp, p=0.001, significant. Concordant with primary — ELEVATE confirmed on TWO genuinely post-edit samples."+ "note": "M5 FRESH replication on the ORIGINAL frozen 150-item SWE-bench set. Successfully replicates the M5 fresh primary (+5.3pp) in the same positive direction: 90% vs 82% placebo, +8.0pp, p=0.001, significant. This is a strong replication SIGNAL confirming the directional effect is real. However, the overall skill verdict remains DIRECTIONAL-NOT-REPLICATED because the full-n=150 PRIMARY (+5.3pp, p=0.061) failed the p<0.05 decision gate — per architecture rules, BOTH primary AND replication must pass for ELEVATE."}],"judge_evidence": [@@ -1243,7 +1243,7 @@"n": 9,"source": "evals/results/run1/tier3-behavioral.json","superseded": true,- "note": "Mid-document ELEVATE-OR-KILL.md pooled 63% ELEVATE claim superseded; this skill is the only one with objective ELEVATE"+ "note": "Mid-document ELEVATE-OR-KILL.md pooled 63% ELEVATE claim superseded; this skill was the closest to objective ELEVATE but downgraded to DIRECTIONAL-NOT-REPLICATED (primary +5.3pp p=0.061 fails gate)"}],"verdict": "directional-not-replicated",diff --git a/analysis/ELEVATE-OR-KILL-SCORECARD.md b/analysis/ELEVATE-OR-KILL-SCORECARD.mdindex 889ef24..f7b487f 100644--- a/analysis/ELEVATE-OR-KILL-SCORECARD.md+++ b/analysis/ELEVATE-OR-KILL-SCORECARD.md@@ -227,7 +227,7 @@| `red-team` | 67% | +1 | proven | superseded (powered directional) || `regret-minimization` | 0% | −3 | regresses | superseded || `reversibility` | 67% | +1 | proven | superseded (ceiling) |-| `scientific-method` | 67% | +1 | proven | superseded (robust ELEVATE) |+| `scientific-method` | 67% | +1 | proven | superseded (directional-not-replicated) || `second-order` | 67% | +1 | proven | superseded (powered null) || `socratic` | 67% | +1 | proven | superseded (ceiling, Tier 1 flagged) || `steel-manning` | 67% | +1 | proven | superseded |