28 eval-informed mental models and critical-thinking skills for Claude Code, GitHub Copilot, Codex, Cursor, and other Agent Skills-compatible tools
JavaScript agent-skillsai-toolsanthropicclaudeclaude-codecritical-thinkingdecision-makingfirst-principlesmental-modelsproblem-solvingsystems-thinking
feat(evals): four-tier + objective eval runners
Travis Boudreauxf2a7bbb094aa
diff --git a/evals/README.md b/evals/README.mdnew file mode 100644index 0000000..fb50fea--- /dev/null+++ b/evals/README.md@@ -0,0 +1,87 @@+# Thinking Skills — Eval Harness++Outcome-based evaluation for the thinking skills. The repo's original+`scripts/validate-skills.js` measures **structural conformance** (does a file have+the right headers/tables/checkboxes). This harness measures whether a skill+actually **improves reasoning** and **fires at the right time**.++All model calls go through the authenticated `droid` CLI (`evals/lib/droid.js`).+`claude -p` is not used — the headless subprocess fails auth (401); `droid` has+provider keys configured and can drive Claude/GPT/Gemini/DeepSeek uniformly.++## Tiers++| Tier | Script | Cost | What it measures |+|---|---|---|---|+| 0 | `run-structural.js` | free | header/format conformance + a substance-aware re-score (shows where the strict lint under-counts content-rich skills) |+| 1 | `run-rubric.js` | LLM | a judge grades each SKILL.md's content: fidelity / applicability / actionability / discrimination / discoverability (1–5) + a "would mislead an agent" flag |+| 2 | `run-routing.js` | LLM | given a natural prompt + all 39 descriptions, does the right skill get picked, and correctly NOT fire on routine requests? Reports **lenient** (accept siblings) vs **strict/unique** accuracy |+| 3 | `run-behavioral.js` | LLM | **headline.** Claude solves 108 jargon-free problems under controlled conditions; a cross-family judge does blind pairwise comparison. Reports per-skill win-rate + lift |++## Tier 3 methodology (length-controlled)++The naive test — skill vs *empty* baseline — confounds the skill's reasoning with+simply having 2–4k more words of context. So Tier 3 uses **length-matched+conditions** (`lib/conditions.js`):++- `placebo` — problem + neutral, content-free filler matched to the skill's word count+- `skill` — problem + the full SKILL.md+- `trigger` — problem + skill name + a 2–3 sentence trigger summary++Headline comparison is **`skill` vs `placebo`** (isolates content from length).+The optional **`skill` vs `trigger`** comparison tests whether the full guide+beats a one-liner — i.e. whether the SKILL.md is a textbook worth shipping or+dead weight. Solver = Claude (the real consumer); judge = a different family to+limit self-preference. Pair ordering is assigned by problem-index parity+(deterministic, position-bias-balanced).++> **Sample-size caveat:** 3 problems/skill is statistically noisy. Trust the+> cross-skill **aggregate**; treat per-skill verdicts as **directional**. Raise+> coverage by adding problems to `datasets/behavioral/<skill>.json`.++## Running++```bash+# Tier 0 (free)+EVAL_RUN=run1 node evals/run-structural.js++# Tier 1 — judge defaults to gemini-3.1-pro-preview+EVAL_RUN=run1 CONC=4 JUDGE_MODEL=gemini-3.1-pro-preview node evals/run-rubric.js [skill ...]++# Tier 2 — router defaults to claude-sonnet-4-6+EVAL_RUN=run1 CONC=4 ROUTER_MODEL=claude-sonnet-4-6 node evals/run-routing.js++# Tier 3 — length-controlled headline+EVAL_RUN=run1 CONC=4 SOLVER_MODEL=claude-sonnet-4-6 JUDGE_MODEL=gemini-3.1-pro-preview \+ PAIRS="skill:placebo" node evals/run-behavioral.js [skill ...]++# Tier 3 — trigger-vs-instruction probe+PAIRS="skill:placebo,skill:trigger" EVAL_RUN=run1 node evals/run-behavioral.js <subset>++# Tier 3 — capability curve on a weaker solver+SOLVER_MODEL=claude-haiku-4-5-20251001 OUTFILE=tier3-haiku.json EVAL_RUN=run1 \+ node evals/run-behavioral.js <subset>++# Merge everything into one scorecard+EVAL_RUN=run1 node evals/aggregate-scorecard.js # -> results/run1/scorecard.md+```++## Models & reasoning effort++`lib/droid.js` maps each model to its highest supported effort: `gpt-5.5-pro`→`xhigh`,+`gemini-3.1-pro-preview`→`high`, `deepseek-v4-pro`→`max`, `claude-opus-4-8`→`max`.+Override per call with the `effort` option / `SOLVER_EFFORT` env.++## Datasets++- `datasets/routing-cases.jsonl` — 71 cases (39 positive / 16 negative / 16 ambiguous).+- `datasets/behavioral/<skill>.json` — 3 problems/skill (36 leaf skills; the 3 meta+ routers are covered by Tier 2, not Tier 3). Prompts never name the framework and+ each targets a specific failure mode a naive solver falls into.++## Adversarial review (../reviews/)++`reviews/build-packet.js` assembles the shared context; `reviews/run-holistic.js`+and `reviews/run-perskill.js` drive gpt-5.5-pro / gemini-3.1-pro-preview /+deepseek-v4-pro at max effort; `reviews/aggregate.js` collates findings into+`reviews/digest.json` for synthesis.diff --git a/evals/aggregate-scorecard.js b/evals/aggregate-scorecard.jsnew file mode 100644index 0000000..aacd0fc--- /dev/null+++ b/evals/aggregate-scorecard.js@@ -0,0 +1,71 @@+#!/usr/bin/env node+'use strict';++/**+ * Merge tier 0-3 results in the run dir into a single scorecard.md, ranked with+ * behavioral lift as the headline metric.+ * Usage: EVAL_RUN=run1 node evals/aggregate-scorecard.js+ */++const path = require('path');+const fs = require('fs');+const { runDir, readJsonIfExists } = require('./lib/io');++function main() {+ const dir = runDir();+ const t0 = readJsonIfExists(path.join(dir, 'tier0-structural.json'));+ const t1 = readJsonIfExists(path.join(dir, 'tier1-rubric.json'));+ const t2 = readJsonIfExists(path.join(dir, 'tier2-routing.json'));+ const t3 = readJsonIfExists(path.join(dir, 'tier3-behavioral.json'));++ const skills = new Map();+ const ensure = n => { if (!skills.has(n)) skills.set(n, { skill: n }); return skills.get(n); };++ if (t0) for (const r of t0.rows) Object.assign(ensure(r.name), { strict: r.strict_pct, loose: r.loose_pct, descLen: r.descLen });+ if (t1) for (const r of (t1.results || [])) if (r.ok) Object.assign(ensure(r.skill), { rubric: +Number(r.avg).toFixed(2), weakest: r.weakest_dimension, mislead: !!r.would_mislead });+ if (t3) for (const r of (t3.skills || [])) { const hp = (r.pairs && r.pairs[t3.headline_pair]) || {}; Object.assign(ensure(r.skill), { winRate: r.win_rate, lift: r.lift, verdict: r.verdict, vocabOnly: hp.vocab_only || 0 }); }++ const rows = [...skills.values()];+ rows.sort((a, b) => (a.winRate ?? 1) - (b.winRate ?? 1) || (a.rubric ?? 5) - (b.rubric ?? 5));++ const L = [];+ L.push(`# Thinking Skills — Eval Scorecard`);+ L.push(``);+ L.push(`Run: \`${process.env.EVAL_RUN || 'latest'}\``);+ L.push(``);+ L.push(`## Headline numbers`);+ if (t0) L.push(`- **Tier 0 structural:** strict ${t0.overall_strict_pct}% vs substance-aware ${t0.overall_loose_pct}% (the strict number undercounts content-rich skills).`);+ if (t1) L.push(`- **Tier 1 rubric (${t1.judge}):** mean ${t1.overall_avg.toFixed(2)}/5; ${t1.would_mislead.length} skills flagged as potentially misleading.`);+ if (t2) L.push(`- **Tier 2 routing (${t2.router}):** lenient ${(t2.accuracy_lenient * 100).toFixed(0)}% / strict-unique ${(t2.accuracy_strict * 100).toFixed(0)}%; ` + Object.entries(t2.by_type).map(([k, v]) => `${k} ${(v.acc * 100).toFixed(0)}%`).join(', ') + `.`);+ if (t3) L.push(`- **Tier 3 behavioral (length-controlled, headline pair ${t3.headline_pair}; solver ${t3.solver} / judge ${t3.judge}):** mean win-rate ${(t3.mean_win_rate * 100).toFixed(0)}%; proven ${t3.proven.length}/${t3.n_skills}.`);+ L.push(``);+ L.push(`## Per-skill (sorted by behavioral win-rate, weakest first)`);+ L.push(``);+ L.push(`| Skill | Behavioral win-rate | Lift | Verdict | Rubric /5 | Weakest dim | Structural strict→loose |`);+ L.push(`|---|---|---|---|---|---|---|`);+ for (const r of rows) {+ L.push(`| ${r.skill} | ${r.winRate != null ? (r.winRate * 100).toFixed(0) + '%' : '—'} | ${r.lift ?? '—'} | ${r.verdict || '—'} | ${r.rubric ?? '—'} | ${r.weakest || '—'} | ${r.strict != null ? r.strict + '→' + r.loose : '—'} |`);+ }+ L.push(``);+ if (t3) {+ L.push(`## Unproven / regressing (behavioral lift ≤ 0)`);+ for (const u of t3.unproven) L.push(`- **${u.skill}** — win-rate ${(u.win_rate * 100).toFixed(0)}%, ${u.verdict}`);+ L.push(``);+ }+ if (t1 && t1.would_mislead.length) {+ L.push(`## Flagged as potentially misleading (Tier 1)`);+ for (const m of t1.would_mislead) L.push(`- **${m.skill}** — ${m.note}`);+ L.push(``);+ }+ if (t0) {+ L.push(`## Over-length descriptions (>200 chars — hurts auto-invocation)`);+ for (const d of t0.over_length_descriptions) L.push(`- ${d.name}: ${d.len}`);+ L.push(``);+ }++ const file = path.join(dir, 'scorecard.md');+ fs.writeFileSync(file, L.join('\n'));+ console.log(`scorecard -> ${file}`);+}++main();diff --git a/evals/run-behavioral.js b/evals/run-behavioral.jsnew file mode 100644index 0000000..4c9ce0b--- /dev/null+++ b/evals/run-behavioral.js@@ -0,0 +1,138 @@+#!/usr/bin/env node+'use strict';++/**+ * Tier 3 — Behavioral A/B lift (the headline signal), length-controlled.+ *+ * Per problem the SOLVER answers under multiple CONDITIONS, then a cross-family+ * JUDGE does blind, order-deterministic pairwise comparisons for each PAIR.+ *+ * conditions: placebo (length-matched filler), skill (full SKILL.md), trigger (name+2-3 sentences), empty+ * default PAIRS: "skill:placebo" -> isolates skill CONTENT from mere context length+ * add "skill:trigger" to test whether the full guide beats a one-liner.+ *+ * Solver = Claude (the real consumer). Judge = different family. Order assigned+ * by problem-index parity to balance position bias and stay reproducible.+ *+ * Usage:+ * EVAL_RUN=run1 node evals/run-behavioral.js # all skills, skill:placebo+ * PAIRS="skill:placebo,skill:trigger" node evals/run-behavioral.js thinking-inversion+ * SOLVER_MODEL=claude-haiku-4-5-20251001 PAIRS="skill:placebo" EVAL_RUN=run1-haiku node evals/run-behavioral.js <subset>+ */++const fs = require('fs');+const path = require('path');+const { droidExecAsync, maxEffortFor } = require('./lib/droid');+const { runDir, writeJson, mapPool } = require('./lib/io');+const { buildConditionPrompt } = require('./lib/conditions');+const { panelJudge, panelModels } = require('./lib/judge');+const { summarize } = require('./lib/stats');++const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';+const JUDGE = process.env.JUDGE_MODEL || 'gemini-3.1-pro-preview';+const CONC = parseInt(process.env.CONC || '3', 10);+const PAIRS = (process.env.PAIRS || 'skill:placebo').split(',').map(p => p.trim().split(':'));+const OUTFILE = process.env.OUTFILE || 'tier3-behavioral.json';+const DATA_DIR = path.join(__dirname, 'datasets', 'behavioral');+const RUBRIC = fs.readFileSync(path.join(__dirname, 'rubrics', 'behavioral-pairwise.md'), 'utf8');+const SKILLS_DIR = path.join(__dirname, '..', 'skills');++function listDatasets(argv) {+ const names = argv.length ? argv : fs.readdirSync(DATA_DIR).filter(f => f.endsWith('.json')).map(f => f.replace(/\.json$/, ''));+ return names.map(name => JSON.parse(fs.readFileSync(path.join(DATA_DIR, `${name}.json`), 'utf8')));+}+function skillContent(name) { return fs.readFileSync(path.join(SKILLS_DIR, name, 'SKILL.md'), 'utf8'); }++function judgePrompt(problem, respA, respB) {+ return `${RUBRIC}\n\n=== PROBLEM ===\n${problem.prompt}\n\n=== RESPONSE A ===\n${respA}\n\n=== RESPONSE B ===\n${respB}\n\n=== END ===\nReturn ONLY the JSON verdict.`;+}++async function runProblem(skillName, skillMd, problem, idx, neededConditions) {+ // solve each needed condition (concurrently)+ const solves = await Promise.all(neededConditions.map(c =>+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: buildConditionPrompt(c, problem.prompt, skillMd, skillName) })+ .then(r => [c, r])+ ));+ const byCond = Object.fromEntries(solves);+ const failed = neededConditions.filter(c => !byCond[c].ok);+ if (failed.length) return { id: problem.id, ok: false, error: `solver fail: ${failed.join(',')} (${byCond[failed[0]].error})` };++ // judge each pair+ const pairResults = {};+ for (const [left, right] of PAIRS) {+ const key = `${left}:${right}`;+ // deterministic order: even idx -> left is A+ const leftIsA = idx % 2 === 0;+ const respA = leftIsA ? byCond[left].text : byCond[right].text;+ const respB = leftIsA ? byCond[right].text : byCond[left].text;+ const v = await panelJudge(judgePrompt(problem, respA, respB));+ let winner = 'tie';+ if (v.winner === 'A') winner = leftIsA ? left : right;+ else if (v.winner === 'B') winner = leftIsA ? right : left;+ pairResults[key] = { ok: true, winner, vocab_only: v.vocab_only, votes: v.tally, why: (v.whys || [])[0] };+ }+ return { id: problem.id, ok: true, pairs: pairResults, failure_mode: problem.failure_mode || '' };+}++function summarizePair(problems, key, leftCond) {+ const ok = problems.filter(p => p.ok && p.pairs[key] && p.pairs[key].ok);+ const wins = ok.filter(p => p.pairs[key].winner === leftCond).length;+ const ties = ok.filter(p => p.pairs[key].winner === 'tie').length;+ const losses = ok.length - wins - ties;+ const n = ok.length || 1;+ return { n: ok.length, wins, losses, ties, win_rate: +(((wins + 0.5 * ties) / n)).toFixed(3), lift: wins - losses, vocab_only: ok.filter(p => p.pairs[key].vocab_only).length };+}++async function runSkill(ds) {+ const skillMd = skillContent(ds.skill);+ const needed = [...new Set(PAIRS.flat())];+ const problems = [];+ for (let i = 0; i < ds.problems.length; i++) problems.push(await runProblem(ds.skill, skillMd, ds.problems[i], i, needed));+ const summary = { skill: ds.skill, n: problems.filter(p => p.ok).length, pairs: {}, problems };+ for (const [left, right] of PAIRS) summary.pairs[`${left}:${right}`] = summarizePair(problems, `${left}:${right}`, left);+ const head = summary.pairs[`${PAIRS[0][0]}:${PAIRS[0][1]}`];+ summary.win_rate = head.win_rate; summary.lift = head.lift;+ summary.verdict = head.lift > 0 ? 'proven' : (head.lift === 0 ? 'unproven-tie' : 'regresses');+ return summary;+}++async function main() {+ const argv = process.argv.slice(2);+ const datasets = listDatasets(argv);+ console.log(`Tier 3 — behavioral: ${datasets.length} skills, solver=${SOLVER}(${SOLVER_EFFORT}) judges=${panelModels().join('+')}, conc=${CONC}`);+ console.log(` pairs (left vs right; win = left wins): ${PAIRS.map(p => p.join(' vs ')).join(', ')}`);++ let done = 0;+ const scored = await mapPool(datasets, CONC, async (ds) => {+ const r = await runSkill(ds);+ done++;+ const head = r.pairs[`${PAIRS[0][0]}:${PAIRS[0][1]}`];+ console.log(` [${done}/${datasets.length}] ${r.skill}: ${head.wins}W ${head.losses}L ${head.ties}T ${(head.win_rate * 100).toFixed(0)}% -> ${r.verdict}${head.vocab_only ? ` (vocab-only x${head.vocab_only})` : ''}`);+ return r;+ });++ scored.sort((a, b) => a.win_rate - b.win_rate);+ const headKey = `${PAIRS[0][0]}:${PAIRS[0][1]}`;+ // aggregate significance across ALL problems for the headline pair+ let aw = 0, al = 0, at = 0;+ for (const s of scored) { const h = s.pairs[headKey] || {}; aw += h.wins || 0; al += h.losses || 0; at += h.ties || 0; }+ const aggregate = summarize(aw, al, at);+ const out = {+ tier: 3, solver: SOLVER, solver_effort: SOLVER_EFFORT, judges: panelModels(),+ pairs: PAIRS.map(p => p.join(':')), headline_pair: headKey,+ n_skills: scored.length,+ aggregate_significance: aggregate,+ mean_win_rate: +(scored.reduce((a, r) => a + r.win_rate, 0) / (scored.length || 1)).toFixed(3),+ proven: scored.filter(r => r.verdict === 'proven').map(r => r.skill),+ unproven: scored.filter(r => r.verdict !== 'proven').map(r => ({ skill: r.skill, win_rate: r.win_rate, verdict: r.verdict })),+ skills: scored,+ };+ const file = path.join(runDir(), OUTFILE);+ writeJson(file, out);+ console.log(`\n mean win-rate (${headKey}): ${(out.mean_win_rate * 100).toFixed(0)}% proven: ${out.proven.length}/${scored.length}`);+ console.log(` AGGREGATE: ${aggregate.wins}W ${aggregate.losses}L ${aggregate.ties}T → ${(aggregate.win_rate * 100).toFixed(0)}% 95% CI [${(aggregate.ci95[0] * 100).toFixed(0)}%, ${(aggregate.ci95[1] * 100).toFixed(0)}%] p=${aggregate.p_value} ${aggregate.significant ? 'SIGNIFICANT' : 'directional'}`);+ console.log(` -> ${file}`);+}++main();diff --git a/evals/run-correctness.js b/evals/run-correctness.jsnew file mode 100644index 0000000..4459ac7--- /dev/null+++ b/evals/run-correctness.js@@ -0,0 +1,127 @@+#!/usr/bin/env node+'use strict';++/**+ * Correctness-mode eval (objective, judge-bias-free). For each ground-truth item+ * the SOLVER answers twice IN THE SAME RUN — WITH the most-relevant skill injected+ * vs WITH a length-matched placebo — and we score the answer against ground truth.+ * Paired McNemar test per skill + overall. This dodges the LLM-judge format bias+ * because the outcome is correct/incorrect, not a preference.+ *+ * Reads correctness candidates from evals/datasets/external/*.jsonl (mode=correctness,+ * with answer_idx (MCQ) or answer (yes/no) + skill_fit tags).+ *+ * Usage: EVAL_RUN=run1 SOLVER_MODEL=claude-sonnet-4-6 node evals/run-correctness.js [medical legal]+ */++const fs = require('fs');+const path = require('path');+const { droidExecAsync } = require('./lib/droid');+const { runDir, writeJson, mapPool } = require('./lib/io');+const { neutralFiller, wordCount } = require('./lib/conditions');+const { mcnemar, wilson } = require('./lib/stats');++const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';+const CONC = parseInt(process.env.CONC || '4', 10);+const EXT = path.join(__dirname, 'datasets', 'external');+const SKILLS_DIR = path.join(__dirname, '..', 'skills');++function loadItems(tags) {+ const out = [];+ for (const t of tags) {+ const f = path.join(EXT, `${t}.jsonl`);+ if (!fs.existsSync(f)) continue;+ for (const line of fs.readFileSync(f, 'utf8').trim().split('\n').filter(Boolean)) {+ const o = JSON.parse(line);+ if (o.mode === 'correctness' && (o.answer_idx || o.answer)) out.push(o);+ }+ }+ return process.env.LIMIT ? out.slice(0, parseInt(process.env.LIMIT, 10)) : out;+}++function skillContent(short) {+ const f = path.join(SKILLS_DIR, 'thinking-' + short.replace(/^thinking-/, ''), 'SKILL.md');+ return fs.existsSync(f) ? fs.readFileSync(f, 'utf8') : null;+}+function pickSkill(item) {+ if (process.env.FORCE_SKILL) { const f = process.env.FORCE_SKILL.replace(/^thinking-/, ''); return skillContent(f) ? f : null; }+ for (const s of (item.skill_fit || [])) if (skillContent(s)) return s.replace(/^thinking-/, '');+ return null;+}++function answerInstruction(item) {+ return item.answer_idx ? `\n\nEnd your response with exactly: ANSWER: <letter A-E>` : `\n\nEnd your response with exactly: ANSWER: <Yes or No>`;+}+function withSkillPrompt(item, skillMd) {+ return `Use the following thinking-skill guide to reason about the problem. Apply it substantively.\n\n=== THINKING SKILL ===\n${skillMd}\n=== END SKILL ===\n\n${item.prompt}${answerInstruction(item)}`;+}+function placeboPrompt(item, skillMd) {+ return `Some general notes:\n\n${neutralFiller(wordCount(skillMd))}\n\n${item.prompt}${answerInstruction(item)}`;+}+function extract(item, text) {+ if (item.answer_idx) { const m = text.match(/ANSWER:\s*([A-E])/i) || text.match(/\b([A-E])\b\s*\.?\s*$/); return m ? m[1].toUpperCase() : null; }+ const m = text.match(/ANSWER:\s*(yes|no)/i) || text.match(/\b(yes|no)\b\s*\.?\s*$/i); return m ? m[1].toLowerCase() : null;+}+function isCorrect(item, extracted) {+ if (extracted == null) return false;+ return item.answer_idx ? extracted === String(item.answer_idx).toUpperCase()+ : extracted === String(item.answer || '').trim().toLowerCase();+}++async function runItem(item) {+ const skill = pickSkill(item);+ if (!skill) return null;+ const skillMd = skillContent(skill);+ const [w, p] = await Promise.all([+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: withSkillPrompt(item, skillMd) }),+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: placeboPrompt(item, skillMd) }),+ ]);+ if (!w.ok || !p.ok) return { id: item.id, skill, ok: false };+ const cw = isCorrect(item, extract(item, w.text)), cp = isCorrect(item, extract(item, p.text));+ return { id: item.id, skill, ok: true, skill_correct: cw, placebo_correct: cp };+}++async function main() {+ const tags = process.argv.slice(2).length ? process.argv.slice(2) : ['medical', 'legal'];+ const items = loadItems(tags);+ console.log(`Correctness eval: ${items.length} items from [${tags.join(', ')}], solver=${SOLVER}(${SOLVER_EFFORT}), conc=${CONC}`);+ if (!items.length) { console.log(' no correctness candidates found — run the bulk ingest first.'); return; }++ let done = 0;+ const results = (await mapPool(items, CONC, async it => { const r = await runItem(it); done++; if (done % 10 === 0) console.log(` ${done}/${items.length}`); return r; })).filter(r => r && r.ok);++ // per-skill + overall paired stats+ const bySkill = {};+ for (const r of results) {+ const s = bySkill[r.skill] = bySkill[r.skill] || { n: 0, sc: 0, pc: 0, b: 0, c: 0 };+ s.n++; if (r.skill_correct) s.sc++; if (r.placebo_correct) s.pc++;+ if (r.skill_correct && !r.placebo_correct) s.b++;+ if (!r.skill_correct && r.placebo_correct) s.c++;+ }+ const skills = Object.entries(bySkill).map(([skill, s]) => ({+ skill, n: s.n, acc_with: +(s.sc / s.n).toFixed(3), acc_placebo: +(s.pc / s.n).toFixed(3),+ delta_pp: +(((s.sc - s.pc) / s.n) * 100).toFixed(1), discordant: s.b + s.c, mcnemar_p: +mcnemar(s.b, s.c).toFixed(3),+ })).sort((a, b) => b.delta_pp - a.delta_pp);++ const N = results.length, SC = results.filter(r => r.skill_correct).length, PC = results.filter(r => r.placebo_correct).length;+ const B = results.filter(r => r.skill_correct && !r.placebo_correct).length;+ const C = results.filter(r => !r.skill_correct && r.placebo_correct).length;+ const ciW = wilson(SC, N), ciP = wilson(PC, N);+ const out = {+ mode: 'correctness', solver: SOLVER, n: N,+ acc_with_skill: +(SC / N).toFixed(3), acc_with_skill_ci: ciW.map(x => +x.toFixed(3)),+ acc_placebo: +(PC / N).toFixed(3), acc_placebo_ci: ciP.map(x => +x.toFixed(3)),+ delta_pp: +(((SC - PC) / N) * 100).toFixed(1),+ mcnemar_p: +mcnemar(B, C).toFixed(3), significant: mcnemar(B, C) < 0.05,+ discordant: B + C, by_skill: skills,+ };+ const file = process.env.OUTFILE || path.join(runDir(), 'correctness.json');+ fs.mkdirSync(path.dirname(file), { recursive: true });+ writeJson(file, out);+ console.log(`\n accuracy WITH skill: ${(out.acc_with_skill * 100).toFixed(0)}% vs placebo ${(out.acc_placebo * 100).toFixed(0)}% (Δ ${out.delta_pp >= 0 ? '+' : ''}${out.delta_pp}pp)`);+ console.log(` McNemar p=${out.mcnemar_p} (${out.significant ? 'SIGNIFICANT' : 'not sig'}); discordant pairs=${out.discordant}`);+ console.log(` -> ${file}`);+}++main();diff --git a/evals/run-numeric.js b/evals/run-numeric.jsnew file mode 100644index 0000000..bf5883c--- /dev/null+++ b/evals/run-numeric.js@@ -0,0 +1,118 @@+#!/usr/bin/env node+'use strict';++/**+ * Correctness eval for NUMERIC and BINARY-PROBABILITY ground truth (paired,+ * isolated, length-controlled). Unlocks fermi-estimation (order-of-magnitude),+ * bayesian/probabilistic (forecasting → accuracy + Brier).+ *+ * numeric : score within K orders of magnitude of answer_num (default K=1)+ * binary-prob : model gives P(yes); correct if (P>0.5)==answer_bin; also Brier+ *+ * Compares FORCE_SKILL vs length-matched placebo (paired McNemar). Optional+ * trigger arm via PAIRS-like TRIGGER=1.+ *+ * Usage: EVAL_RUN=run1 FORCE_SKILL=fermi-estimation MODE=numeric node evals/run-numeric.js fermi+ * FORCE_SKILL=probabilistic MODE=binary-prob node evals/run-numeric.js forecasting+ */++const fs = require('fs');+const path = require('path');+const { droidExecAsync } = require('./lib/droid');+const { runDir, writeJson, mapPool } = require('./lib/io');+const { neutralFiller, wordCount, triggerSummary } = require('./lib/conditions');+const { mcnemar, wilson, summarize } = require('./lib/stats');++const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';+const CONC = parseInt(process.env.CONC || '4', 10);+const MODE = process.env.MODE || 'numeric';+const OOM = parseFloat(process.env.OOM || '1'); // orders of magnitude tolerance+const FORCE_SKILL = process.env.FORCE_SKILL;+const DO_TRIGGER = process.env.TRIGGER === '1';+const EXT = path.join(__dirname, 'datasets', 'external');+const SKILLS_DIR = path.join(__dirname, '..', 'skills');++function loadItems(tags) {+ const out = [];+ for (const t of tags) {+ const f = path.join(EXT, `${t}.jsonl`);+ if (!fs.existsSync(f)) continue;+ for (const l of fs.readFileSync(f, 'utf8').trim().split('\n').filter(Boolean)) out.push(JSON.parse(l));+ }+ return process.env.LIMIT ? out.slice(0, parseInt(process.env.LIMIT, 10)) : out;+}+function skillContent(s) { const f = path.join(SKILLS_DIR, 'thinking-' + s.replace(/^thinking-/, ''), 'SKILL.md'); return fs.existsSync(f) ? fs.readFileSync(f, 'utf8') : null; }++function prompts(item, skillMd, skillName) {+ const p = item.prompt;+ const skill = `Use the following thinking-skill guide to reason. Apply it substantively.\n\n=== THINKING SKILL ===\n${skillMd}\n=== END SKILL ===\n\n${p}`;+ const placebo = `Some general notes:\n\n${neutralFiller(wordCount(skillMd))}\n\n${p}`;+ const trigger = `Consider this approach if relevant:\n${triggerSummary(skillMd, skillName)}\n\n${p}`;+ return { skill, placebo, trigger };+}++function extractNum(text) {+ const m = text.match(/ANSWER:\s*\$?\s*(-?[\d,]*\.?\d+(?:\s*[eE]\s*[-+]?\d+)?(?:\s*[×x]\s*10\^?-?\d+)?)/);+ if (!m) { const all = text.match(/-?\d[\d,]*\.?\d*(?:[eE][-+]?\d+)?/g); return all ? parseFloat(all[all.length - 1].replace(/,/g, '')) : null; }+ let s = m[1].replace(/,/g, '').replace(/\s/g, '').replace(/[×x]10\^?/, 'e');+ return parseFloat(s);+}+function numericCorrect(item, text) {+ const est = extractNum(text);+ if (est == null || !isFinite(est) || est === 0) return false;+ return Math.abs(Math.log10(Math.abs(est) / Math.abs(item.answer_num))) <= OOM;+}+function extractProb(text) {+ const m = text.match(/ANSWER:\s*(0?\.\d+|1\.0+|1|0|\d{1,3})\s*%?/i);+ if (!m) return null;+ let v = parseFloat(m[1]); if (/%/.test(m[0]) || v > 1) v = v / 100; return Math.max(0, Math.min(1, v));+}++async function runItem(item, skillName, skillMd) {+ const P = prompts(item, skillMd, skillName);+ const conds = DO_TRIGGER ? ['skill', 'placebo', 'trigger'] : ['skill', 'placebo'];+ const res = await Promise.all(conds.map(c => droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: P[c] })));+ if (res.some(r => !r.ok)) return null;+ const by = Object.fromEntries(conds.map((c, i) => [c, res[i].text]));+ const score = txt => MODE === 'numeric' ? numericCorrect(item, txt) : ((extractProb(txt) > 0.5 ? 1 : 0) === item.answer_bin);+ const brier = txt => { if (MODE !== 'binary-prob') return null; const p = extractProb(txt); return p == null ? null : Math.pow(p - item.answer_bin, 2); };+ return {+ skill_correct: score(by.skill), placebo_correct: score(by.placebo),+ trigger_correct: DO_TRIGGER ? score(by.trigger) : null,+ skill_brier: brier(by.skill), placebo_brier: brier(by.placebo),+ };+}++async function main() {+ const tags = process.argv.slice(2);+ if (!FORCE_SKILL) { console.error('set FORCE_SKILL'); process.exit(1); }+ const skillMd = skillContent(FORCE_SKILL);+ if (!skillMd) { console.error('no skill ' + FORCE_SKILL); process.exit(1); }+ const items = loadItems(tags).filter(i => MODE === 'numeric' ? i.answer_num != null : i.answer_bin != null);+ console.log(`Numeric eval [${MODE}]: ${items.length} items, skill=${FORCE_SKILL}, solver=${SOLVER}(${SOLVER_EFFORT}), OOM=${OOM}, isolation ${process.env.EVAL_NO_ISOLATE === '1' ? 'OFF' : 'ON'}`);+ let done = 0;+ const r = (await mapPool(items, CONC, async it => { const x = await runItem(it, FORCE_SKILL, skillMd); done++; if (done % 10 === 0) console.log(` ${done}/${items.length}`); return x; })).filter(Boolean);++ const SC = r.filter(x => x.skill_correct).length, PC = r.filter(x => x.placebo_correct).length, N = r.length;+ const b = r.filter(x => x.skill_correct && !x.placebo_correct).length, c = r.filter(x => !x.skill_correct && x.placebo_correct).length;+ const out = {+ mode: MODE, skill: FORCE_SKILL, solver: SOLVER, n: N,+ acc_with_skill: +(SC / N).toFixed(3), acc_with_skill_ci: wilson(SC, N).map(x => +x.toFixed(3)),+ acc_placebo: +(PC / N).toFixed(3), acc_placebo_ci: wilson(PC, N).map(x => +x.toFixed(3)),+ delta_pp: +(((SC - PC) / N) * 100).toFixed(1), mcnemar_p: +mcnemar(b, c).toFixed(3), significant: mcnemar(b, c) < 0.05, discordant: b + c,+ };+ if (DO_TRIGGER) { const TC = r.filter(x => x.trigger_correct).length; out.acc_trigger = +(TC / N).toFixed(3); }+ if (MODE === 'binary-prob') {+ const sb = r.map(x => x.skill_brier).filter(v => v != null), pb = r.map(x => x.placebo_brier).filter(v => v != null);+ out.brier_skill = +(sb.reduce((a, v) => a + v, 0) / (sb.length || 1)).toFixed(3);+ out.brier_placebo = +(pb.reduce((a, v) => a + v, 0) / (pb.length || 1)).toFixed(3);+ }+ const file = process.env.OUTFILE || path.join(runDir(), `numeric-${FORCE_SKILL}.json`);+ fs.mkdirSync(path.dirname(file), { recursive: true }); writeJson(file, out);+ console.log(`\n acc WITH skill ${(out.acc_with_skill * 100).toFixed(0)}% CI[${(out.acc_with_skill_ci[0] * 100).toFixed(0)},${(out.acc_with_skill_ci[1] * 100).toFixed(0)}] vs placebo ${(out.acc_placebo * 100).toFixed(0)}% CI[${(out.acc_placebo_ci[0] * 100).toFixed(0)},${(out.acc_placebo_ci[1] * 100).toFixed(0)}] Δ${out.delta_pp >= 0 ? '+' : ''}${out.delta_pp}pp McNemar p=${out.mcnemar_p}${out.significant ? ' SIG' : ''}`);+ if (out.brier_skill != null) console.log(` Brier skill ${out.brier_skill} vs placebo ${out.brier_placebo} (lower=better)`);+ if (out.acc_trigger != null) console.log(` acc TRIGGER ${(out.acc_trigger * 100).toFixed(0)}%`);+ console.log(` -> ${file}`);+}+main();diff --git a/evals/run-routing.js b/evals/run-routing.jsnew file mode 100644index 0000000..6aba0a6--- /dev/null+++ b/evals/run-routing.js@@ -0,0 +1,98 @@+#!/usr/bin/env node+'use strict';++/**+ * Tier 2 — Invocation / routing accuracy.+ *+ * Given a realistic user prompt plus the catalog of every skill's `description`,+ * does a Claude-class model pick the correct skill — and correctly pick NONE for+ * irrelevant prompts? This tests the description fields that drive Claude Code's+ * auto-invocation, and the router's discriminability. Negatives guard against+ * "fires everywhere" noise.+ *+ * Usage: EVAL_RUN=run1 ROUTER_MODEL=claude-sonnet-4-6 node evals/run-routing.js+ */++const fs = require('fs');+const path = require('path');+const { loadAllSkills } = require('./lib/skills');+const { droidJsonAsync, maxEffortFor } = require('./lib/droid');+const { runDir, writeJson, mapPool } = require('./lib/io');++const ROUTER = process.env.ROUTER_MODEL || 'claude-sonnet-4-6';+const CONC = parseInt(process.env.CONC || '4', 10);+const DATASET = path.join(__dirname, 'datasets', 'routing-cases.jsonl');++function loadCases() {+ return fs.readFileSync(DATASET, 'utf8').split('\n').map(l => l.trim()).filter(Boolean).map(l => JSON.parse(l));+}++function buildCatalog() {+ return loadAllSkills().map(s => `- ${s.name}: ${s.description}`).join('\n');+}++function buildPrompt(catalog, userPrompt) {+ return `You are the skill-router inside an AI coding agent. Below is a catalog of optional "thinking skill" guides, each with a one-line description. A user has sent a message. Decide which SINGLE skill (if any) should be auto-loaded to help answer it best.\n\n` ++ `Rules:\n- Pick the one skill whose description best matches the user's actual need.\n- If NO skill is a clear, strong fit (the request is routine and a mental-model guide would just be noise), answer "none".\n- Do not force a match. "none" is often correct.\n\n` ++ `=== SKILL CATALOG ===\n${catalog}\n\n` ++ `=== USER MESSAGE ===\n${userPrompt}\n\n` ++ `Return ONLY JSON: {"skill": "thinking-... or none", "confidence": 0.0-1.0, "rationale": "<one sentence>"}`;+}++function judge(chosen, c) {+ const exp = c.expected; // string or null+ if (exp === null || exp === 'none') return chosen === 'none';+ const acceptable = new Set([exp, ...(c.acceptable || [])]);+ return acceptable.has(chosen);+}++// Strict: a positive/ambiguous must match the SINGLE expected skill, ignoring the+// `acceptable` set. Surfaces the discrimination problem that lenient scoring hides.+function judgeStrict(chosen, c) {+ if (c.expected === null || c.expected === 'none') return chosen === 'none';+ return chosen === c.expected;+}++async function main() {+ let cases = loadCases();+ if (process.env.LIMIT) cases = cases.slice(0, parseInt(process.env.LIMIT, 10));+ const catalog = buildCatalog();+ console.log(`Tier 2 — routing: ${cases.length} cases, router=${ROUTER} (${maxEffortFor(ROUTER)}), conc=${CONC}`);++ let n = 0;+ const results = await mapPool(cases, CONC, async (c) => {+ const r = await droidJsonAsync({ model: ROUTER, prompt: buildPrompt(catalog, c.prompt) });+ n++;+ if (!r.ok) { console.log(` [${n}/${cases.length}] ${c.id}: FAILED (${r.error})`); return { id: c.id, ok: false, error: r.error }; }+ const chosen = (r.json.skill || 'none').trim();+ const correct = judge(chosen, c);+ const correctStrict = judgeStrict(chosen, c);+ if (!correct) console.log(` [${n}/${cases.length}] ${c.id} ✗ chose ${chosen} (expected ${c.expected || 'none'}) [${c.type}]`);+ return { id: c.id, ok: true, type: c.type, expected: c.expected, chosen, correct, correctStrict, confidence: r.json.confidence, rationale: r.json.rationale };+ });++ const ok = results.filter(r => r.ok);+ const byType = {};+ for (const r of ok) {+ byType[r.type] = byType[r.type] || { total: 0, correct: 0 };+ byType[r.type].total++; if (r.correct) byType[r.type].correct++;+ }+ const acc = ok.length ? ok.filter(r => r.correct).length / ok.length : 0;+ const accStrict = ok.length ? ok.filter(r => r.correctStrict).length / ok.length : 0;+ const out = {+ tier: 2, router: ROUTER, effort: maxEffortFor(ROUTER),+ n: results.length, n_ok: ok.length,+ accuracy_lenient: acc, accuracy_strict: accStrict,+ by_type: Object.fromEntries(Object.entries(byType).map(([k, v]) => [k, { ...v, acc: +(v.correct / v.total).toFixed(2) }])),+ misroutes: ok.filter(r => !r.correct),+ discrimination_only_errors: ok.filter(r => r.correct && !r.correctStrict).map(r => ({ id: r.id, expected: r.expected, chose: r.chosen })),+ results,+ };+ const file = path.join(runDir(), 'tier2-routing.json');+ writeJson(file, out);+ console.log(`\n accuracy: lenient ${(acc * 100).toFixed(0)}% | strict/unique ${(accStrict * 100).toFixed(0)}%`);+ for (const [t, v] of Object.entries(out.by_type)) console.log(` ${t}: ${v.correct}/${v.total} (${(v.acc * 100).toFixed(0)}%)`);+ console.log(` -> ${file}`);+}++main();diff --git a/evals/run-rubric.js b/evals/run-rubric.jsnew file mode 100644index 0000000..32008e7--- /dev/null+++ b/evals/run-rubric.js@@ -0,0 +1,65 @@+#!/usr/bin/env node+'use strict';++/**+ * Tier 1 — LLM-graded reasoning-quality rubric.+ *+ * A judge model reads each SKILL.md and scores its *content* (not headers) on+ * fidelity / applicability / actionability / discrimination / discoverability.+ * This is the substance signal the structural lint cannot produce.+ *+ * Usage: EVAL_RUN=run1 JUDGE_MODEL=gemini-3.1-pro-preview node evals/run-rubric.js [skill ...]+ */++const fs = require('fs');+const path = require('path');+const { loadAllSkills, readSkill } = require('./lib/skills');+const { droidJsonAsync, maxEffortFor } = require('./lib/droid');+const { runDir, writeJson, mapPool } = require('./lib/io');++const RUBRIC = fs.readFileSync(path.join(__dirname, 'rubrics', 'reasoning-quality.md'), 'utf8');+const JUDGE = process.env.JUDGE_MODEL || 'gemini-3.1-pro-preview';+const CONC = parseInt(process.env.CONC || '4', 10);++function buildPrompt(skill) {+ return `You are grading a Claude Code thinking skill against the rubric below.\n\n` ++ `=== RUBRIC ===\n${RUBRIC}\n\n` ++ `=== SKILL FILE: ${skill.name} ===\n${skill.content}\n\n` ++ `=== END SKILL ===\n\n` ++ `Return ONLY the JSON object specified in the rubric, with "skill" set to "${skill.name}". No prose.`;+}++async function main() {+ const argv = process.argv.slice(2);+ const skills = argv.length ? argv.map(readSkill) : loadAllSkills();+ console.log(`Tier 1 — rubric grading ${skills.length} skills with judge=${JUDGE} (${maxEffortFor(JUDGE)}), conc=${CONC}`);++ let n = 0;+ const results = await mapPool(skills, CONC, async (skill) => {+ const r = await droidJsonAsync({ model: JUDGE, prompt: buildPrompt(skill) });+ n++;+ if (!r.ok) { console.log(` [${n}/${skills.length}] ${skill.name}: FAILED (${r.error})`); return { skill: skill.name, ok: false, error: r.error, raw: r.raw }; }+ const s = r.json.scores || {};+ const avg = ['fidelity','applicability','actionability','discrimination','discoverability']+ .map(k => Number(s[k]) || 0).reduce((a, b) => a + b, 0) / 5;+ console.log(` [${n}/${skills.length}] ${skill.name}: avg ${avg.toFixed(2)}/5 weakest=${r.json.weakest_dimension || '?'}${r.json.would_mislead ? ' ⚠ MISLEAD' : ''}`);+ return { skill: skill.name, ok: true, judge: JUDGE, ...r.json, avg };+ });++ const ok = results.filter(r => r.ok);+ ok.sort((a, b) => a.avg - b.avg);+ const out = {+ tier: 1, judge: JUDGE, effort: maxEffortFor(JUDGE),+ n: results.length, n_ok: ok.length,+ overall_avg: ok.length ? (ok.reduce((a, r) => a + r.avg, 0) / ok.length) : 0,+ would_mislead: ok.filter(r => r.would_mislead).map(r => ({ skill: r.skill, note: r.mislead_note })),+ weakest: ok.slice(0, 10).map(r => ({ skill: r.skill, avg: Number(r.avg.toFixed(2)), weakest_dimension: r.weakest_dimension, top_fix: r.top_fix })),+ results,+ };+ const file = path.join(runDir(), 'tier1-rubric.json');+ writeJson(file, out);+ console.log(`\n overall avg: ${out.overall_avg.toFixed(2)}/5 misleading flags: ${out.would_mislead.length}`);+ console.log(` -> ${file}`);+}++main();diff --git a/evals/run-structural.js b/evals/run-structural.jsnew file mode 100644index 0000000..88204b6--- /dev/null+++ b/evals/run-structural.js@@ -0,0 +1,114 @@+#!/usr/bin/env node+'use strict';++/**+ * Tier 0 — Structural lint (free, deterministic, no API cost).+ *+ * Reproduces the existing scripts/validate-skills.js 18-point "strict" score,+ * AND computes a "loosened" substance-aware score that recognizes real content+ * the strict regexes miss (e.g. a fenced template block, an examples section,+ * an anti-patterns paragraph). The gap between the two quantifies how much the+ * current quality number is measuring headers rather than substance — the+ * `thinking-debiasing` 72% false-negative is the motivating case.+ */++const { loadAllSkills } = require('./lib/skills');+const { runDir, writeJson } = require('./lib/io');+const path = require('path');++// ---- strict checks (copied from scripts/validate-skills.js) ----+const STRICT_SECTIONS = [+ { name: 'YAML Frontmatter', p: /^---\n[\s\S]*?name:[\s\S]*?description:[\s\S]*?---/m },+ { name: 'Overview', p: /^## Overview/m },+ { name: 'Core Principle', p: /\*\*Core Principle:\*\*/m },+ { name: 'When to Use', p: /^## When to Use/m },+ { name: 'Decision Flow', p: /```[\s\S]*?→[\s\S]*?```/m },+ { name: 'Process/Steps', p: /^### Step \d|^## The .* Process|^## .*Process$/m },+ { name: 'Examples', p: /^## .*Example|Example:|```markdown[\s\S]*?##/m },+ { name: 'Template', p: /^## .*Template|Template:/m },+ { name: 'Verification Checklist', p: /^## Verification Checklist/m },+ { name: 'Key Questions', p: /^## Key Questions/m },+ { name: 'Wisdom/Attribution', p: /Wisdom|".*"/m },+];+const STRICT_QUALITY = [+ { name: 'Description Length', f: c => (c.match(/description:\s*(.+)/) || [, ''])[1].length <= 200 },+ { name: 'Multiple Examples', f: c => ((c.match(/Example[s]?:/gi) || []).length + (c.match(/```markdown[\s\S]*?##/g) || []).length) >= 2 },+ { name: 'Tables', f: c => (c.match(/\|.*\|.*\|/g) || []).length >= 3 },+ { name: 'Checklists', f: c => (c.match(/- \[ \]/g) || []).length >= 3 },+ { name: 'Code Blocks', f: c => (c.match(/```/g) || []).length / 2 >= 3 },+ { name: 'Word Count', f: c => { const w = c.split(/\s+/).length; return w >= 500 && w <= 3000; } },+ { name: 'Anti-patterns', f: c => /anti-pattern|failure mode|common mistake|when not to|don't use/i.test(c) },+];++function strictScore(content) {+ let score = 0, max = 0;+ const missing = [];+ for (const s of STRICT_SECTIONS) { max++; if (s.p.test(content)) score++; else missing.push(s.name); }+ for (const q of STRICT_QUALITY) { max++; if (q.f(content)) score++; else missing.push(q.name); }+ return { score, max, pct: Math.round((score / max) * 100), missing };+}++// ---- loosened, substance-aware checks ----+function looseScore(content) {+ const checks = {+ has_frontmatter: /^---\n[\s\S]*?name:[\s\S]*?description:[\s\S]*?---/m.test(content),+ has_overview: /^## (Overview|Summary|What)/im.test(content),+ has_core_idea: /\*\*Core Principle/i.test(content) || /^## (Core|The Principle|The Idea)/im.test(content),+ has_when_to_use: /^## When to (Use|Apply)/im.test(content),+ // any fenced block OR a fill-in template (e.g. a markdown block with [brackets])+ has_template: /^## .*Template/im.test(content) || /```[\s\S]*?\[[^\]]+\][\s\S]*?```/m.test(content),+ // examples: a section OR >=2 worked code blocks+ has_examples: /^#{2,3} .*Example/im.test(content) || (content.match(/```/g) || []).length / 2 >= 2,+ has_verification: /^## Verification|^## Checklist|- \[ \]/im.test(content),+ has_key_questions: /^## (Key Questions|Questions)/im.test(content) || /\?\s*$/m.test(content),+ // boundaries: when-not, failure modes, pitfalls, OR a comparison table+ has_boundaries: /anti-pattern|failure mode|common (mistake|pitfall)|when not to|don't use|pitfall|trap/i.test(content),+ has_attribution: /".{10,}"|—\s*\w|\bper\b|attributed|coined|developed by/i.test(content),+ desc_ok: (content.match(/description:\s*(.+)/) || [, ''])[1].length <= 200,+ word_ok: (() => { const w = content.split(/\s+/).length; return w >= 400 && w <= 3500; })(),+ };+ const vals = Object.values(checks);+ const score = vals.filter(Boolean).length;+ return { score, max: vals.length, pct: Math.round((score / vals.length) * 100), checks };+}++function main() {+ const skills = loadAllSkills();+ const rows = skills.map(s => {+ const strict = strictScore(s.content);+ const loose = looseScore(s.content);+ return {+ name: s.name,+ descLen: s.description.length,+ strict_pct: strict.pct,+ loose_pct: loose.pct,+ gap: loose.pct - strict.pct,+ strict_missing: strict.missing,+ words: s.content.split(/\s+/).length,+ };+ });+ rows.sort((a, b) => b.gap - a.gap);++ const overallStrict = Math.round(rows.reduce((a, r) => a + r.strict_pct, 0) / rows.length);+ const overallLoose = Math.round(rows.reduce((a, r) => a + r.loose_pct, 0) / rows.length);++ const out = {+ tier: 0,+ overall_strict_pct: overallStrict,+ overall_loose_pct: overallLoose,+ false_negatives: rows.filter(r => r.gap >= 15).map(r => ({ name: r.name, strict: r.strict_pct, loose: r.loose_pct })),+ over_length_descriptions: rows.filter(r => r.descLen > 200).map(r => ({ name: r.name, len: r.descLen })),+ rows,+ };+ const file = path.join(runDir(), 'tier0-structural.json');+ writeJson(file, out);+ console.log(`Tier 0 — structural lint`);+ console.log(` strict overall: ${overallStrict}%`);+ console.log(` loosened overall: ${overallLoose}%`);+ console.log(` biggest false-negatives (strict undercounts substance):`);+ for (const r of out.false_negatives.slice(0, 8)) console.log(` ${r.name}: strict ${r.strict}% vs loose ${r.loose}%`);+ console.log(` over-length descriptions (>200): ${out.over_length_descriptions.map(d => `${d.name}(${d.len})`).join(', ') || 'none'}`);+ console.log(` -> ${file}`);+}++main();diff --git a/evals/run-swe.js b/evals/run-swe.jsnew file mode 100644index 0000000..6d39557--- /dev/null+++ b/evals/run-swe.js@@ -0,0 +1,77 @@+#!/usr/bin/env node+'use strict';++/**+ * SWE-bench fault-file localization (native-domain debugging, objective, isolated).+ * Given a real GitHub issue, the solver names the file to fix; scored against the+ * gold patch's modified file(s). Tests whether a debugging skill (scientific-method,+ * five-whys, systems, occams) improves fault localization vs a length-matched placebo.+ * Paired McNemar + Wilson CI. No repo checkout / test execution required.+ *+ * Usage: EVAL_RUN=run1 FORCE_SKILL=scientific-method node evals/run-swe.js+ */++const fs = require('fs');+const path = require('path');+const { droidExecAsync } = require('./lib/droid');+const { runDir, writeJson, mapPool } = require('./lib/io');+const { neutralFiller, wordCount } = require('./lib/conditions');+const { mcnemar, wilson } = require('./lib/stats');++const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';+const CONC = parseInt(process.env.CONC || '4', 10);+const FORCE_SKILL = process.env.FORCE_SKILL;+const FILE = path.join(__dirname, 'datasets', 'external', 'swebench.jsonl');+const SKILLS_DIR = path.join(__dirname, '..', 'skills');++function loadItems() {+ let L = fs.readFileSync(FILE, 'utf8').trim().split('\n').filter(Boolean).map(JSON.parse);+ return process.env.LIMIT ? L.slice(0, parseInt(process.env.LIMIT, 10)) : L;+}+function skillContent(s) { return fs.readFileSync(path.join(SKILLS_DIR, 'thinking-' + s.replace(/^thinking-/, ''), 'SKILL.md'), 'utf8'); }++function locatedCorrectly(item, text) {+ // correct if a gold file path or its (distinctive) basename appears in the response+ for (const g of item.gold_files) {+ if (text.includes(g)) return true;+ const base = g.split('/').pop();+ if (base && !/^(__init__|utils|index|main|app|setup)\.\w+$/.test(base) && new RegExp('\\b' + base.replace('.', '\\.') + '\\b').test(text)) return true;+ }+ return false;+}++async function runItem(item, skillMd) {+ const tail = `\n\nThink carefully and answer.`;+ const skillP = `Use the following thinking-skill guide to reason about the bug. Apply it substantively.\n\n=== THINKING SKILL ===\n${skillMd}\n=== END SKILL ===\n\n${item.prompt}${tail}`;+ const placeboP = `Some general notes:\n\n${neutralFiller(wordCount(skillMd))}\n\n${item.prompt}${tail}`;+ const [s, p] = await Promise.all([+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: skillP }),+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: placeboP }),+ ]);+ if (!s.ok || !p.ok) return null;+ return { skill_correct: locatedCorrectly(item, s.text), placebo_correct: locatedCorrectly(item, p.text) };+}++async function main() {+ if (!FORCE_SKILL) { console.error('set FORCE_SKILL'); process.exit(1); }+ const skillMd = skillContent(FORCE_SKILL);+ const items = loadItems();+ console.log(`SWE-bench localization: ${items.length} issues, skill=${FORCE_SKILL}, solver=${SOLVER}(${SOLVER_EFFORT}), isolation ${process.env.EVAL_NO_ISOLATE === '1' ? 'OFF' : 'ON'}`);+ let done = 0;+ const r = (await mapPool(items, CONC, async it => { const x = await runItem(it, skillMd); done++; if (done % 10 === 0) console.log(` ${done}/${items.length}`); return x; })).filter(Boolean);++ const SC = r.filter(x => x.skill_correct).length, PC = r.filter(x => x.placebo_correct).length, N = r.length;+ const b = r.filter(x => x.skill_correct && !x.placebo_correct).length, c = r.filter(x => !x.skill_correct && x.placebo_correct).length;+ const out = {+ mode: 'swe-localize', skill: FORCE_SKILL, solver: SOLVER, n: N,+ acc_with_skill: +(SC / N).toFixed(3), acc_with_skill_ci: wilson(SC, N).map(x => +x.toFixed(3)),+ acc_placebo: +(PC / N).toFixed(3), acc_placebo_ci: wilson(PC, N).map(x => +x.toFixed(3)),+ delta_pp: +(((SC - PC) / N) * 100).toFixed(1), mcnemar_p: +mcnemar(b, c).toFixed(3), significant: mcnemar(b, c) < 0.05, discordant: b + c,+ };+ const file = process.env.OUTFILE || path.join(runDir(), `swe-${FORCE_SKILL}.json`);+ fs.mkdirSync(path.dirname(file), { recursive: true }); writeJson(file, out);+ console.log(`\n localize WITH skill ${(out.acc_with_skill * 100).toFixed(0)}% CI[${(out.acc_with_skill_ci[0] * 100).toFixed(0)},${(out.acc_with_skill_ci[1] * 100).toFixed(0)}] vs placebo ${(out.acc_placebo * 100).toFixed(0)}% CI[${(out.acc_placebo_ci[0] * 100).toFixed(0)},${(out.acc_placebo_ci[1] * 100).toFixed(0)}] Δ${out.delta_pp >= 0 ? '+' : ''}${out.delta_pp}pp McNemar p=${out.mcnemar_p}${out.significant ? ' SIG' : ''}`);+ console.log(` -> ${file}`);+}+main();