Skip to content

28 eval-informed mental models and critical-thinking skills for Claude Code, GitHub Copilot, Codex, Cursor, and other Agent Skills-compatible tools

feat(evals): four-tier + objective eval runners

Travis Boudreauxf2a7bbb094aa
diff --git a/evals/README.md b/evals/README.md
new file mode 100644
index 0000000..fb50fea
--- /dev/null
+++ b/evals/README.md
@@ -0,0 +1,87 @@
+# Thinking Skills — Eval Harness
+
+Outcome-based evaluation for the thinking skills. The repo's original
+`scripts/validate-skills.js` measures **structural conformance** (does a file have
+the right headers/tables/checkboxes). This harness measures whether a skill
+actually **improves reasoning** and **fires at the right time**.
+
+All model calls go through the authenticated `droid` CLI (`evals/lib/droid.js`).
+`claude -p` is not used — the headless subprocess fails auth (401); `droid` has
+provider keys configured and can drive Claude/GPT/Gemini/DeepSeek uniformly.
+
+## Tiers
+
+| Tier | Script | Cost | What it measures |
+|---|---|---|---|
+| 0 | `run-structural.js` | free | header/format conformance + a substance-aware re-score (shows where the strict lint under-counts content-rich skills) |
+| 1 | `run-rubric.js` | LLM | a judge grades each SKILL.md's content: fidelity / applicability / actionability / discrimination / discoverability (1–5) + a "would mislead an agent" flag |
+| 2 | `run-routing.js` | LLM | given a natural prompt + all 39 descriptions, does the right skill get picked, and correctly NOT fire on routine requests? Reports **lenient** (accept siblings) vs **strict/unique** accuracy |
+| 3 | `run-behavioral.js` | LLM | **headline.** Claude solves 108 jargon-free problems under controlled conditions; a cross-family judge does blind pairwise comparison. Reports per-skill win-rate + lift |
+
+## Tier 3 methodology (length-controlled)
+
+The naive test — skill vs *empty* baseline — confounds the skill's reasoning with
+simply having 2–4k more words of context. So Tier 3 uses **length-matched
+conditions** (`lib/conditions.js`):
+
+- `placebo` — problem + neutral, content-free filler matched to the skill's word count
+- `skill` — problem + the full SKILL.md
+- `trigger` — problem + skill name + a 2–3 sentence trigger summary
+
+Headline comparison is **`skill` vs `placebo`** (isolates content from length).
+The optional **`skill` vs `trigger`** comparison tests whether the full guide
+beats a one-liner — i.e. whether the SKILL.md is a textbook worth shipping or
+dead weight. Solver = Claude (the real consumer); judge = a different family to
+limit self-preference. Pair ordering is assigned by problem-index parity
+(deterministic, position-bias-balanced).
+
+> **Sample-size caveat:** 3 problems/skill is statistically noisy. Trust the
+> cross-skill **aggregate**; treat per-skill verdicts as **directional**. Raise
+> coverage by adding problems to `datasets/behavioral/<skill>.json`.
+
+## Running
+
+```bash
+# Tier 0 (free)
+EVAL_RUN=run1 node evals/run-structural.js
+
+# Tier 1 — judge defaults to gemini-3.1-pro-preview
+EVAL_RUN=run1 CONC=4 JUDGE_MODEL=gemini-3.1-pro-preview node evals/run-rubric.js [skill ...]
+
+# Tier 2 — router defaults to claude-sonnet-4-6
+EVAL_RUN=run1 CONC=4 ROUTER_MODEL=claude-sonnet-4-6 node evals/run-routing.js
+
+# Tier 3 — length-controlled headline
+EVAL_RUN=run1 CONC=4 SOLVER_MODEL=claude-sonnet-4-6 JUDGE_MODEL=gemini-3.1-pro-preview \
+ PAIRS="skill:placebo" node evals/run-behavioral.js [skill ...]
+
+# Tier 3 — trigger-vs-instruction probe
+PAIRS="skill:placebo,skill:trigger" EVAL_RUN=run1 node evals/run-behavioral.js <subset>
+
+# Tier 3 — capability curve on a weaker solver
+SOLVER_MODEL=claude-haiku-4-5-20251001 OUTFILE=tier3-haiku.json EVAL_RUN=run1 \
+ node evals/run-behavioral.js <subset>
+
+# Merge everything into one scorecard
+EVAL_RUN=run1 node evals/aggregate-scorecard.js # -> results/run1/scorecard.md
+```
+
+## Models & reasoning effort
+
+`lib/droid.js` maps each model to its highest supported effort: `gpt-5.5-pro`→`xhigh`,
+`gemini-3.1-pro-preview`→`high`, `deepseek-v4-pro`→`max`, `claude-opus-4-8`→`max`.
+Override per call with the `effort` option / `SOLVER_EFFORT` env.
+
+## Datasets
+
+- `datasets/routing-cases.jsonl` — 71 cases (39 positive / 16 negative / 16 ambiguous).
+- `datasets/behavioral/<skill>.json` — 3 problems/skill (36 leaf skills; the 3 meta
+ routers are covered by Tier 2, not Tier 3). Prompts never name the framework and
+ each targets a specific failure mode a naive solver falls into.
+
+## Adversarial review (../reviews/)
+
+`reviews/build-packet.js` assembles the shared context; `reviews/run-holistic.js`
+and `reviews/run-perskill.js` drive gpt-5.5-pro / gemini-3.1-pro-preview /
+deepseek-v4-pro at max effort; `reviews/aggregate.js` collates findings into
+`reviews/digest.json` for synthesis.
diff --git a/evals/aggregate-scorecard.js b/evals/aggregate-scorecard.js
new file mode 100644
index 0000000..aacd0fc
--- /dev/null
+++ b/evals/aggregate-scorecard.js
@@ -0,0 +1,71 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Merge tier 0-3 results in the run dir into a single scorecard.md, ranked with
+ * behavioral lift as the headline metric.
+ * Usage: EVAL_RUN=run1 node evals/aggregate-scorecard.js
+ */
+
+const path = require('path');
+const fs = require('fs');
+const { runDir, readJsonIfExists } = require('./lib/io');
+
+function main() {
+ const dir = runDir();
+ const t0 = readJsonIfExists(path.join(dir, 'tier0-structural.json'));
+ const t1 = readJsonIfExists(path.join(dir, 'tier1-rubric.json'));
+ const t2 = readJsonIfExists(path.join(dir, 'tier2-routing.json'));
+ const t3 = readJsonIfExists(path.join(dir, 'tier3-behavioral.json'));
+
+ const skills = new Map();
+ const ensure = n => { if (!skills.has(n)) skills.set(n, { skill: n }); return skills.get(n); };
+
+ if (t0) for (const r of t0.rows) Object.assign(ensure(r.name), { strict: r.strict_pct, loose: r.loose_pct, descLen: r.descLen });
+ if (t1) for (const r of (t1.results || [])) if (r.ok) Object.assign(ensure(r.skill), { rubric: +Number(r.avg).toFixed(2), weakest: r.weakest_dimension, mislead: !!r.would_mislead });
+ if (t3) for (const r of (t3.skills || [])) { const hp = (r.pairs && r.pairs[t3.headline_pair]) || {}; Object.assign(ensure(r.skill), { winRate: r.win_rate, lift: r.lift, verdict: r.verdict, vocabOnly: hp.vocab_only || 0 }); }
+
+ const rows = [...skills.values()];
+ rows.sort((a, b) => (a.winRate ?? 1) - (b.winRate ?? 1) || (a.rubric ?? 5) - (b.rubric ?? 5));
+
+ const L = [];
+ L.push(`# Thinking Skills — Eval Scorecard`);
+ L.push(``);
+ L.push(`Run: \`${process.env.EVAL_RUN || 'latest'}\``);
+ L.push(``);
+ L.push(`## Headline numbers`);
+ if (t0) L.push(`- **Tier 0 structural:** strict ${t0.overall_strict_pct}% vs substance-aware ${t0.overall_loose_pct}% (the strict number undercounts content-rich skills).`);
+ if (t1) L.push(`- **Tier 1 rubric (${t1.judge}):** mean ${t1.overall_avg.toFixed(2)}/5; ${t1.would_mislead.length} skills flagged as potentially misleading.`);
+ if (t2) L.push(`- **Tier 2 routing (${t2.router}):** lenient ${(t2.accuracy_lenient * 100).toFixed(0)}% / strict-unique ${(t2.accuracy_strict * 100).toFixed(0)}%; ` + Object.entries(t2.by_type).map(([k, v]) => `${k} ${(v.acc * 100).toFixed(0)}%`).join(', ') + `.`);
+ if (t3) L.push(`- **Tier 3 behavioral (length-controlled, headline pair ${t3.headline_pair}; solver ${t3.solver} / judge ${t3.judge}):** mean win-rate ${(t3.mean_win_rate * 100).toFixed(0)}%; proven ${t3.proven.length}/${t3.n_skills}.`);
+ L.push(``);
+ L.push(`## Per-skill (sorted by behavioral win-rate, weakest first)`);
+ L.push(``);
+ L.push(`| Skill | Behavioral win-rate | Lift | Verdict | Rubric /5 | Weakest dim | Structural strict→loose |`);
+ L.push(`|---|---|---|---|---|---|---|`);
+ for (const r of rows) {
+ L.push(`| ${r.skill} | ${r.winRate != null ? (r.winRate * 100).toFixed(0) + '%' : '—'} | ${r.lift ?? '—'} | ${r.verdict || '—'} | ${r.rubric ?? '—'} | ${r.weakest || '—'} | ${r.strict != null ? r.strict + '→' + r.loose : '—'} |`);
+ }
+ L.push(``);
+ if (t3) {
+ L.push(`## Unproven / regressing (behavioral lift ≤ 0)`);
+ for (const u of t3.unproven) L.push(`- **${u.skill}** — win-rate ${(u.win_rate * 100).toFixed(0)}%, ${u.verdict}`);
+ L.push(``);
+ }
+ if (t1 && t1.would_mislead.length) {
+ L.push(`## Flagged as potentially misleading (Tier 1)`);
+ for (const m of t1.would_mislead) L.push(`- **${m.skill}** — ${m.note}`);
+ L.push(``);
+ }
+ if (t0) {
+ L.push(`## Over-length descriptions (>200 chars — hurts auto-invocation)`);
+ for (const d of t0.over_length_descriptions) L.push(`- ${d.name}: ${d.len}`);
+ L.push(``);
+ }
+
+ const file = path.join(dir, 'scorecard.md');
+ fs.writeFileSync(file, L.join('\n'));
+ console.log(`scorecard -> ${file}`);
+}
+
+main();
diff --git a/evals/run-behavioral.js b/evals/run-behavioral.js
new file mode 100644
index 0000000..4c9ce0b
--- /dev/null
+++ b/evals/run-behavioral.js
@@ -0,0 +1,138 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Tier 3 — Behavioral A/B lift (the headline signal), length-controlled.
+ *
+ * Per problem the SOLVER answers under multiple CONDITIONS, then a cross-family
+ * JUDGE does blind, order-deterministic pairwise comparisons for each PAIR.
+ *
+ * conditions: placebo (length-matched filler), skill (full SKILL.md), trigger (name+2-3 sentences), empty
+ * default PAIRS: "skill:placebo" -> isolates skill CONTENT from mere context length
+ * add "skill:trigger" to test whether the full guide beats a one-liner.
+ *
+ * Solver = Claude (the real consumer). Judge = different family. Order assigned
+ * by problem-index parity to balance position bias and stay reproducible.
+ *
+ * Usage:
+ * EVAL_RUN=run1 node evals/run-behavioral.js # all skills, skill:placebo
+ * PAIRS="skill:placebo,skill:trigger" node evals/run-behavioral.js thinking-inversion
+ * SOLVER_MODEL=claude-haiku-4-5-20251001 PAIRS="skill:placebo" EVAL_RUN=run1-haiku node evals/run-behavioral.js <subset>
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { droidExecAsync, maxEffortFor } = require('./lib/droid');
+const { runDir, writeJson, mapPool } = require('./lib/io');
+const { buildConditionPrompt } = require('./lib/conditions');
+const { panelJudge, panelModels } = require('./lib/judge');
+const { summarize } = require('./lib/stats');
+
+const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';
+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';
+const JUDGE = process.env.JUDGE_MODEL || 'gemini-3.1-pro-preview';
+const CONC = parseInt(process.env.CONC || '3', 10);
+const PAIRS = (process.env.PAIRS || 'skill:placebo').split(',').map(p => p.trim().split(':'));
+const OUTFILE = process.env.OUTFILE || 'tier3-behavioral.json';
+const DATA_DIR = path.join(__dirname, 'datasets', 'behavioral');
+const RUBRIC = fs.readFileSync(path.join(__dirname, 'rubrics', 'behavioral-pairwise.md'), 'utf8');
+const SKILLS_DIR = path.join(__dirname, '..', 'skills');
+
+function listDatasets(argv) {
+ const names = argv.length ? argv : fs.readdirSync(DATA_DIR).filter(f => f.endsWith('.json')).map(f => f.replace(/\.json$/, ''));
+ return names.map(name => JSON.parse(fs.readFileSync(path.join(DATA_DIR, `${name}.json`), 'utf8')));
+}
+function skillContent(name) { return fs.readFileSync(path.join(SKILLS_DIR, name, 'SKILL.md'), 'utf8'); }
+
+function judgePrompt(problem, respA, respB) {
+ return `${RUBRIC}\n\n=== PROBLEM ===\n${problem.prompt}\n\n=== RESPONSE A ===\n${respA}\n\n=== RESPONSE B ===\n${respB}\n\n=== END ===\nReturn ONLY the JSON verdict.`;
+}
+
+async function runProblem(skillName, skillMd, problem, idx, neededConditions) {
+ // solve each needed condition (concurrently)
+ const solves = await Promise.all(neededConditions.map(c =>
+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: buildConditionPrompt(c, problem.prompt, skillMd, skillName) })
+ .then(r => [c, r])
+ ));
+ const byCond = Object.fromEntries(solves);
+ const failed = neededConditions.filter(c => !byCond[c].ok);
+ if (failed.length) return { id: problem.id, ok: false, error: `solver fail: ${failed.join(',')} (${byCond[failed[0]].error})` };
+
+ // judge each pair
+ const pairResults = {};
+ for (const [left, right] of PAIRS) {
+ const key = `${left}:${right}`;
+ // deterministic order: even idx -> left is A
+ const leftIsA = idx % 2 === 0;
+ const respA = leftIsA ? byCond[left].text : byCond[right].text;
+ const respB = leftIsA ? byCond[right].text : byCond[left].text;
+ const v = await panelJudge(judgePrompt(problem, respA, respB));
+ let winner = 'tie';
+ if (v.winner === 'A') winner = leftIsA ? left : right;
+ else if (v.winner === 'B') winner = leftIsA ? right : left;
+ pairResults[key] = { ok: true, winner, vocab_only: v.vocab_only, votes: v.tally, why: (v.whys || [])[0] };
+ }
+ return { id: problem.id, ok: true, pairs: pairResults, failure_mode: problem.failure_mode || '' };
+}
+
+function summarizePair(problems, key, leftCond) {
+ const ok = problems.filter(p => p.ok && p.pairs[key] && p.pairs[key].ok);
+ const wins = ok.filter(p => p.pairs[key].winner === leftCond).length;
+ const ties = ok.filter(p => p.pairs[key].winner === 'tie').length;
+ const losses = ok.length - wins - ties;
+ const n = ok.length || 1;
+ return { n: ok.length, wins, losses, ties, win_rate: +(((wins + 0.5 * ties) / n)).toFixed(3), lift: wins - losses, vocab_only: ok.filter(p => p.pairs[key].vocab_only).length };
+}
+
+async function runSkill(ds) {
+ const skillMd = skillContent(ds.skill);
+ const needed = [...new Set(PAIRS.flat())];
+ const problems = [];
+ for (let i = 0; i < ds.problems.length; i++) problems.push(await runProblem(ds.skill, skillMd, ds.problems[i], i, needed));
+ const summary = { skill: ds.skill, n: problems.filter(p => p.ok).length, pairs: {}, problems };
+ for (const [left, right] of PAIRS) summary.pairs[`${left}:${right}`] = summarizePair(problems, `${left}:${right}`, left);
+ const head = summary.pairs[`${PAIRS[0][0]}:${PAIRS[0][1]}`];
+ summary.win_rate = head.win_rate; summary.lift = head.lift;
+ summary.verdict = head.lift > 0 ? 'proven' : (head.lift === 0 ? 'unproven-tie' : 'regresses');
+ return summary;
+}
+
+async function main() {
+ const argv = process.argv.slice(2);
+ const datasets = listDatasets(argv);
+ console.log(`Tier 3 — behavioral: ${datasets.length} skills, solver=${SOLVER}(${SOLVER_EFFORT}) judges=${panelModels().join('+')}, conc=${CONC}`);
+ console.log(` pairs (left vs right; win = left wins): ${PAIRS.map(p => p.join(' vs ')).join(', ')}`);
+
+ let done = 0;
+ const scored = await mapPool(datasets, CONC, async (ds) => {
+ const r = await runSkill(ds);
+ done++;
+ const head = r.pairs[`${PAIRS[0][0]}:${PAIRS[0][1]}`];
+ console.log(` [${done}/${datasets.length}] ${r.skill}: ${head.wins}W ${head.losses}L ${head.ties}T ${(head.win_rate * 100).toFixed(0)}% -> ${r.verdict}${head.vocab_only ? ` (vocab-only x${head.vocab_only})` : ''}`);
+ return r;
+ });
+
+ scored.sort((a, b) => a.win_rate - b.win_rate);
+ const headKey = `${PAIRS[0][0]}:${PAIRS[0][1]}`;
+ // aggregate significance across ALL problems for the headline pair
+ let aw = 0, al = 0, at = 0;
+ for (const s of scored) { const h = s.pairs[headKey] || {}; aw += h.wins || 0; al += h.losses || 0; at += h.ties || 0; }
+ const aggregate = summarize(aw, al, at);
+ const out = {
+ tier: 3, solver: SOLVER, solver_effort: SOLVER_EFFORT, judges: panelModels(),
+ pairs: PAIRS.map(p => p.join(':')), headline_pair: headKey,
+ n_skills: scored.length,
+ aggregate_significance: aggregate,
+ mean_win_rate: +(scored.reduce((a, r) => a + r.win_rate, 0) / (scored.length || 1)).toFixed(3),
+ proven: scored.filter(r => r.verdict === 'proven').map(r => r.skill),
+ unproven: scored.filter(r => r.verdict !== 'proven').map(r => ({ skill: r.skill, win_rate: r.win_rate, verdict: r.verdict })),
+ skills: scored,
+ };
+ const file = path.join(runDir(), OUTFILE);
+ writeJson(file, out);
+ console.log(`\n mean win-rate (${headKey}): ${(out.mean_win_rate * 100).toFixed(0)}% proven: ${out.proven.length}/${scored.length}`);
+ console.log(` AGGREGATE: ${aggregate.wins}W ${aggregate.losses}L ${aggregate.ties}T → ${(aggregate.win_rate * 100).toFixed(0)}% 95% CI [${(aggregate.ci95[0] * 100).toFixed(0)}%, ${(aggregate.ci95[1] * 100).toFixed(0)}%] p=${aggregate.p_value} ${aggregate.significant ? 'SIGNIFICANT' : 'directional'}`);
+ console.log(` -> ${file}`);
+}
+
+main();
diff --git a/evals/run-correctness.js b/evals/run-correctness.js
new file mode 100644
index 0000000..4459ac7
--- /dev/null
+++ b/evals/run-correctness.js
@@ -0,0 +1,127 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Correctness-mode eval (objective, judge-bias-free). For each ground-truth item
+ * the SOLVER answers twice IN THE SAME RUN — WITH the most-relevant skill injected
+ * vs WITH a length-matched placebo — and we score the answer against ground truth.
+ * Paired McNemar test per skill + overall. This dodges the LLM-judge format bias
+ * because the outcome is correct/incorrect, not a preference.
+ *
+ * Reads correctness candidates from evals/datasets/external/*.jsonl (mode=correctness,
+ * with answer_idx (MCQ) or answer (yes/no) + skill_fit tags).
+ *
+ * Usage: EVAL_RUN=run1 SOLVER_MODEL=claude-sonnet-4-6 node evals/run-correctness.js [medical legal]
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { droidExecAsync } = require('./lib/droid');
+const { runDir, writeJson, mapPool } = require('./lib/io');
+const { neutralFiller, wordCount } = require('./lib/conditions');
+const { mcnemar, wilson } = require('./lib/stats');
+
+const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';
+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';
+const CONC = parseInt(process.env.CONC || '4', 10);
+const EXT = path.join(__dirname, 'datasets', 'external');
+const SKILLS_DIR = path.join(__dirname, '..', 'skills');
+
+function loadItems(tags) {
+ const out = [];
+ for (const t of tags) {
+ const f = path.join(EXT, `${t}.jsonl`);
+ if (!fs.existsSync(f)) continue;
+ for (const line of fs.readFileSync(f, 'utf8').trim().split('\n').filter(Boolean)) {
+ const o = JSON.parse(line);
+ if (o.mode === 'correctness' && (o.answer_idx || o.answer)) out.push(o);
+ }
+ }
+ return process.env.LIMIT ? out.slice(0, parseInt(process.env.LIMIT, 10)) : out;
+}
+
+function skillContent(short) {
+ const f = path.join(SKILLS_DIR, 'thinking-' + short.replace(/^thinking-/, ''), 'SKILL.md');
+ return fs.existsSync(f) ? fs.readFileSync(f, 'utf8') : null;
+}
+function pickSkill(item) {
+ if (process.env.FORCE_SKILL) { const f = process.env.FORCE_SKILL.replace(/^thinking-/, ''); return skillContent(f) ? f : null; }
+ for (const s of (item.skill_fit || [])) if (skillContent(s)) return s.replace(/^thinking-/, '');
+ return null;
+}
+
+function answerInstruction(item) {
+ return item.answer_idx ? `\n\nEnd your response with exactly: ANSWER: <letter A-E>` : `\n\nEnd your response with exactly: ANSWER: <Yes or No>`;
+}
+function withSkillPrompt(item, skillMd) {
+ return `Use the following thinking-skill guide to reason about the problem. Apply it substantively.\n\n=== THINKING SKILL ===\n${skillMd}\n=== END SKILL ===\n\n${item.prompt}${answerInstruction(item)}`;
+}
+function placeboPrompt(item, skillMd) {
+ return `Some general notes:\n\n${neutralFiller(wordCount(skillMd))}\n\n${item.prompt}${answerInstruction(item)}`;
+}
+function extract(item, text) {
+ if (item.answer_idx) { const m = text.match(/ANSWER:\s*([A-E])/i) || text.match(/\b([A-E])\b\s*\.?\s*$/); return m ? m[1].toUpperCase() : null; }
+ const m = text.match(/ANSWER:\s*(yes|no)/i) || text.match(/\b(yes|no)\b\s*\.?\s*$/i); return m ? m[1].toLowerCase() : null;
+}
+function isCorrect(item, extracted) {
+ if (extracted == null) return false;
+ return item.answer_idx ? extracted === String(item.answer_idx).toUpperCase()
+ : extracted === String(item.answer || '').trim().toLowerCase();
+}
+
+async function runItem(item) {
+ const skill = pickSkill(item);
+ if (!skill) return null;
+ const skillMd = skillContent(skill);
+ const [w, p] = await Promise.all([
+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: withSkillPrompt(item, skillMd) }),
+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: placeboPrompt(item, skillMd) }),
+ ]);
+ if (!w.ok || !p.ok) return { id: item.id, skill, ok: false };
+ const cw = isCorrect(item, extract(item, w.text)), cp = isCorrect(item, extract(item, p.text));
+ return { id: item.id, skill, ok: true, skill_correct: cw, placebo_correct: cp };
+}
+
+async function main() {
+ const tags = process.argv.slice(2).length ? process.argv.slice(2) : ['medical', 'legal'];
+ const items = loadItems(tags);
+ console.log(`Correctness eval: ${items.length} items from [${tags.join(', ')}], solver=${SOLVER}(${SOLVER_EFFORT}), conc=${CONC}`);
+ if (!items.length) { console.log(' no correctness candidates found — run the bulk ingest first.'); return; }
+
+ let done = 0;
+ const results = (await mapPool(items, CONC, async it => { const r = await runItem(it); done++; if (done % 10 === 0) console.log(` ${done}/${items.length}`); return r; })).filter(r => r && r.ok);
+
+ // per-skill + overall paired stats
+ const bySkill = {};
+ for (const r of results) {
+ const s = bySkill[r.skill] = bySkill[r.skill] || { n: 0, sc: 0, pc: 0, b: 0, c: 0 };
+ s.n++; if (r.skill_correct) s.sc++; if (r.placebo_correct) s.pc++;
+ if (r.skill_correct && !r.placebo_correct) s.b++;
+ if (!r.skill_correct && r.placebo_correct) s.c++;
+ }
+ const skills = Object.entries(bySkill).map(([skill, s]) => ({
+ skill, n: s.n, acc_with: +(s.sc / s.n).toFixed(3), acc_placebo: +(s.pc / s.n).toFixed(3),
+ delta_pp: +(((s.sc - s.pc) / s.n) * 100).toFixed(1), discordant: s.b + s.c, mcnemar_p: +mcnemar(s.b, s.c).toFixed(3),
+ })).sort((a, b) => b.delta_pp - a.delta_pp);
+
+ const N = results.length, SC = results.filter(r => r.skill_correct).length, PC = results.filter(r => r.placebo_correct).length;
+ const B = results.filter(r => r.skill_correct && !r.placebo_correct).length;
+ const C = results.filter(r => !r.skill_correct && r.placebo_correct).length;
+ const ciW = wilson(SC, N), ciP = wilson(PC, N);
+ const out = {
+ mode: 'correctness', solver: SOLVER, n: N,
+ acc_with_skill: +(SC / N).toFixed(3), acc_with_skill_ci: ciW.map(x => +x.toFixed(3)),
+ acc_placebo: +(PC / N).toFixed(3), acc_placebo_ci: ciP.map(x => +x.toFixed(3)),
+ delta_pp: +(((SC - PC) / N) * 100).toFixed(1),
+ mcnemar_p: +mcnemar(B, C).toFixed(3), significant: mcnemar(B, C) < 0.05,
+ discordant: B + C, by_skill: skills,
+ };
+ const file = process.env.OUTFILE || path.join(runDir(), 'correctness.json');
+ fs.mkdirSync(path.dirname(file), { recursive: true });
+ writeJson(file, out);
+ console.log(`\n accuracy WITH skill: ${(out.acc_with_skill * 100).toFixed(0)}% vs placebo ${(out.acc_placebo * 100).toFixed(0)}% (Δ ${out.delta_pp >= 0 ? '+' : ''}${out.delta_pp}pp)`);
+ console.log(` McNemar p=${out.mcnemar_p} (${out.significant ? 'SIGNIFICANT' : 'not sig'}); discordant pairs=${out.discordant}`);
+ console.log(` -> ${file}`);
+}
+
+main();
diff --git a/evals/run-numeric.js b/evals/run-numeric.js
new file mode 100644
index 0000000..bf5883c
--- /dev/null
+++ b/evals/run-numeric.js
@@ -0,0 +1,118 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Correctness eval for NUMERIC and BINARY-PROBABILITY ground truth (paired,
+ * isolated, length-controlled). Unlocks fermi-estimation (order-of-magnitude),
+ * bayesian/probabilistic (forecasting → accuracy + Brier).
+ *
+ * numeric : score within K orders of magnitude of answer_num (default K=1)
+ * binary-prob : model gives P(yes); correct if (P>0.5)==answer_bin; also Brier
+ *
+ * Compares FORCE_SKILL vs length-matched placebo (paired McNemar). Optional
+ * trigger arm via PAIRS-like TRIGGER=1.
+ *
+ * Usage: EVAL_RUN=run1 FORCE_SKILL=fermi-estimation MODE=numeric node evals/run-numeric.js fermi
+ * FORCE_SKILL=probabilistic MODE=binary-prob node evals/run-numeric.js forecasting
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { droidExecAsync } = require('./lib/droid');
+const { runDir, writeJson, mapPool } = require('./lib/io');
+const { neutralFiller, wordCount, triggerSummary } = require('./lib/conditions');
+const { mcnemar, wilson, summarize } = require('./lib/stats');
+
+const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';
+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';
+const CONC = parseInt(process.env.CONC || '4', 10);
+const MODE = process.env.MODE || 'numeric';
+const OOM = parseFloat(process.env.OOM || '1'); // orders of magnitude tolerance
+const FORCE_SKILL = process.env.FORCE_SKILL;
+const DO_TRIGGER = process.env.TRIGGER === '1';
+const EXT = path.join(__dirname, 'datasets', 'external');
+const SKILLS_DIR = path.join(__dirname, '..', 'skills');
+
+function loadItems(tags) {
+ const out = [];
+ for (const t of tags) {
+ const f = path.join(EXT, `${t}.jsonl`);
+ if (!fs.existsSync(f)) continue;
+ for (const l of fs.readFileSync(f, 'utf8').trim().split('\n').filter(Boolean)) out.push(JSON.parse(l));
+ }
+ return process.env.LIMIT ? out.slice(0, parseInt(process.env.LIMIT, 10)) : out;
+}
+function skillContent(s) { const f = path.join(SKILLS_DIR, 'thinking-' + s.replace(/^thinking-/, ''), 'SKILL.md'); return fs.existsSync(f) ? fs.readFileSync(f, 'utf8') : null; }
+
+function prompts(item, skillMd, skillName) {
+ const p = item.prompt;
+ const skill = `Use the following thinking-skill guide to reason. Apply it substantively.\n\n=== THINKING SKILL ===\n${skillMd}\n=== END SKILL ===\n\n${p}`;
+ const placebo = `Some general notes:\n\n${neutralFiller(wordCount(skillMd))}\n\n${p}`;
+ const trigger = `Consider this approach if relevant:\n${triggerSummary(skillMd, skillName)}\n\n${p}`;
+ return { skill, placebo, trigger };
+}
+
+function extractNum(text) {
+ const m = text.match(/ANSWER:\s*\$?\s*(-?[\d,]*\.?\d+(?:\s*[eE]\s*[-+]?\d+)?(?:\s*[×x]\s*10\^?-?\d+)?)/);
+ if (!m) { const all = text.match(/-?\d[\d,]*\.?\d*(?:[eE][-+]?\d+)?/g); return all ? parseFloat(all[all.length - 1].replace(/,/g, '')) : null; }
+ let s = m[1].replace(/,/g, '').replace(/\s/g, '').replace(/[×x]10\^?/, 'e');
+ return parseFloat(s);
+}
+function numericCorrect(item, text) {
+ const est = extractNum(text);
+ if (est == null || !isFinite(est) || est === 0) return false;
+ return Math.abs(Math.log10(Math.abs(est) / Math.abs(item.answer_num))) <= OOM;
+}
+function extractProb(text) {
+ const m = text.match(/ANSWER:\s*(0?\.\d+|1\.0+|1|0|\d{1,3})\s*%?/i);
+ if (!m) return null;
+ let v = parseFloat(m[1]); if (/%/.test(m[0]) || v > 1) v = v / 100; return Math.max(0, Math.min(1, v));
+}
+
+async function runItem(item, skillName, skillMd) {
+ const P = prompts(item, skillMd, skillName);
+ const conds = DO_TRIGGER ? ['skill', 'placebo', 'trigger'] : ['skill', 'placebo'];
+ const res = await Promise.all(conds.map(c => droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: P[c] })));
+ if (res.some(r => !r.ok)) return null;
+ const by = Object.fromEntries(conds.map((c, i) => [c, res[i].text]));
+ const score = txt => MODE === 'numeric' ? numericCorrect(item, txt) : ((extractProb(txt) > 0.5 ? 1 : 0) === item.answer_bin);
+ const brier = txt => { if (MODE !== 'binary-prob') return null; const p = extractProb(txt); return p == null ? null : Math.pow(p - item.answer_bin, 2); };
+ return {
+ skill_correct: score(by.skill), placebo_correct: score(by.placebo),
+ trigger_correct: DO_TRIGGER ? score(by.trigger) : null,
+ skill_brier: brier(by.skill), placebo_brier: brier(by.placebo),
+ };
+}
+
+async function main() {
+ const tags = process.argv.slice(2);
+ if (!FORCE_SKILL) { console.error('set FORCE_SKILL'); process.exit(1); }
+ const skillMd = skillContent(FORCE_SKILL);
+ if (!skillMd) { console.error('no skill ' + FORCE_SKILL); process.exit(1); }
+ const items = loadItems(tags).filter(i => MODE === 'numeric' ? i.answer_num != null : i.answer_bin != null);
+ console.log(`Numeric eval [${MODE}]: ${items.length} items, skill=${FORCE_SKILL}, solver=${SOLVER}(${SOLVER_EFFORT}), OOM=${OOM}, isolation ${process.env.EVAL_NO_ISOLATE === '1' ? 'OFF' : 'ON'}`);
+ let done = 0;
+ const r = (await mapPool(items, CONC, async it => { const x = await runItem(it, FORCE_SKILL, skillMd); done++; if (done % 10 === 0) console.log(` ${done}/${items.length}`); return x; })).filter(Boolean);
+
+ const SC = r.filter(x => x.skill_correct).length, PC = r.filter(x => x.placebo_correct).length, N = r.length;
+ const b = r.filter(x => x.skill_correct && !x.placebo_correct).length, c = r.filter(x => !x.skill_correct && x.placebo_correct).length;
+ const out = {
+ mode: MODE, skill: FORCE_SKILL, solver: SOLVER, n: N,
+ acc_with_skill: +(SC / N).toFixed(3), acc_with_skill_ci: wilson(SC, N).map(x => +x.toFixed(3)),
+ acc_placebo: +(PC / N).toFixed(3), acc_placebo_ci: wilson(PC, N).map(x => +x.toFixed(3)),
+ delta_pp: +(((SC - PC) / N) * 100).toFixed(1), mcnemar_p: +mcnemar(b, c).toFixed(3), significant: mcnemar(b, c) < 0.05, discordant: b + c,
+ };
+ if (DO_TRIGGER) { const TC = r.filter(x => x.trigger_correct).length; out.acc_trigger = +(TC / N).toFixed(3); }
+ if (MODE === 'binary-prob') {
+ const sb = r.map(x => x.skill_brier).filter(v => v != null), pb = r.map(x => x.placebo_brier).filter(v => v != null);
+ out.brier_skill = +(sb.reduce((a, v) => a + v, 0) / (sb.length || 1)).toFixed(3);
+ out.brier_placebo = +(pb.reduce((a, v) => a + v, 0) / (pb.length || 1)).toFixed(3);
+ }
+ const file = process.env.OUTFILE || path.join(runDir(), `numeric-${FORCE_SKILL}.json`);
+ fs.mkdirSync(path.dirname(file), { recursive: true }); writeJson(file, out);
+ console.log(`\n acc WITH skill ${(out.acc_with_skill * 100).toFixed(0)}% CI[${(out.acc_with_skill_ci[0] * 100).toFixed(0)},${(out.acc_with_skill_ci[1] * 100).toFixed(0)}] vs placebo ${(out.acc_placebo * 100).toFixed(0)}% CI[${(out.acc_placebo_ci[0] * 100).toFixed(0)},${(out.acc_placebo_ci[1] * 100).toFixed(0)}] Δ${out.delta_pp >= 0 ? '+' : ''}${out.delta_pp}pp McNemar p=${out.mcnemar_p}${out.significant ? ' SIG' : ''}`);
+ if (out.brier_skill != null) console.log(` Brier skill ${out.brier_skill} vs placebo ${out.brier_placebo} (lower=better)`);
+ if (out.acc_trigger != null) console.log(` acc TRIGGER ${(out.acc_trigger * 100).toFixed(0)}%`);
+ console.log(` -> ${file}`);
+}
+main();
diff --git a/evals/run-routing.js b/evals/run-routing.js
new file mode 100644
index 0000000..6aba0a6
--- /dev/null
+++ b/evals/run-routing.js
@@ -0,0 +1,98 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Tier 2 — Invocation / routing accuracy.
+ *
+ * Given a realistic user prompt plus the catalog of every skill's `description`,
+ * does a Claude-class model pick the correct skill — and correctly pick NONE for
+ * irrelevant prompts? This tests the description fields that drive Claude Code's
+ * auto-invocation, and the router's discriminability. Negatives guard against
+ * "fires everywhere" noise.
+ *
+ * Usage: EVAL_RUN=run1 ROUTER_MODEL=claude-sonnet-4-6 node evals/run-routing.js
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { loadAllSkills } = require('./lib/skills');
+const { droidJsonAsync, maxEffortFor } = require('./lib/droid');
+const { runDir, writeJson, mapPool } = require('./lib/io');
+
+const ROUTER = process.env.ROUTER_MODEL || 'claude-sonnet-4-6';
+const CONC = parseInt(process.env.CONC || '4', 10);
+const DATASET = path.join(__dirname, 'datasets', 'routing-cases.jsonl');
+
+function loadCases() {
+ return fs.readFileSync(DATASET, 'utf8').split('\n').map(l => l.trim()).filter(Boolean).map(l => JSON.parse(l));
+}
+
+function buildCatalog() {
+ return loadAllSkills().map(s => `- ${s.name}: ${s.description}`).join('\n');
+}
+
+function buildPrompt(catalog, userPrompt) {
+ return `You are the skill-router inside an AI coding agent. Below is a catalog of optional "thinking skill" guides, each with a one-line description. A user has sent a message. Decide which SINGLE skill (if any) should be auto-loaded to help answer it best.\n\n` +
+ `Rules:\n- Pick the one skill whose description best matches the user's actual need.\n- If NO skill is a clear, strong fit (the request is routine and a mental-model guide would just be noise), answer "none".\n- Do not force a match. "none" is often correct.\n\n` +
+ `=== SKILL CATALOG ===\n${catalog}\n\n` +
+ `=== USER MESSAGE ===\n${userPrompt}\n\n` +
+ `Return ONLY JSON: {"skill": "thinking-... or none", "confidence": 0.0-1.0, "rationale": "<one sentence>"}`;
+}
+
+function judge(chosen, c) {
+ const exp = c.expected; // string or null
+ if (exp === null || exp === 'none') return chosen === 'none';
+ const acceptable = new Set([exp, ...(c.acceptable || [])]);
+ return acceptable.has(chosen);
+}
+
+// Strict: a positive/ambiguous must match the SINGLE expected skill, ignoring the
+// `acceptable` set. Surfaces the discrimination problem that lenient scoring hides.
+function judgeStrict(chosen, c) {
+ if (c.expected === null || c.expected === 'none') return chosen === 'none';
+ return chosen === c.expected;
+}
+
+async function main() {
+ let cases = loadCases();
+ if (process.env.LIMIT) cases = cases.slice(0, parseInt(process.env.LIMIT, 10));
+ const catalog = buildCatalog();
+ console.log(`Tier 2 — routing: ${cases.length} cases, router=${ROUTER} (${maxEffortFor(ROUTER)}), conc=${CONC}`);
+
+ let n = 0;
+ const results = await mapPool(cases, CONC, async (c) => {
+ const r = await droidJsonAsync({ model: ROUTER, prompt: buildPrompt(catalog, c.prompt) });
+ n++;
+ if (!r.ok) { console.log(` [${n}/${cases.length}] ${c.id}: FAILED (${r.error})`); return { id: c.id, ok: false, error: r.error }; }
+ const chosen = (r.json.skill || 'none').trim();
+ const correct = judge(chosen, c);
+ const correctStrict = judgeStrict(chosen, c);
+ if (!correct) console.log(` [${n}/${cases.length}] ${c.id} ✗ chose ${chosen} (expected ${c.expected || 'none'}) [${c.type}]`);
+ return { id: c.id, ok: true, type: c.type, expected: c.expected, chosen, correct, correctStrict, confidence: r.json.confidence, rationale: r.json.rationale };
+ });
+
+ const ok = results.filter(r => r.ok);
+ const byType = {};
+ for (const r of ok) {
+ byType[r.type] = byType[r.type] || { total: 0, correct: 0 };
+ byType[r.type].total++; if (r.correct) byType[r.type].correct++;
+ }
+ const acc = ok.length ? ok.filter(r => r.correct).length / ok.length : 0;
+ const accStrict = ok.length ? ok.filter(r => r.correctStrict).length / ok.length : 0;
+ const out = {
+ tier: 2, router: ROUTER, effort: maxEffortFor(ROUTER),
+ n: results.length, n_ok: ok.length,
+ accuracy_lenient: acc, accuracy_strict: accStrict,
+ by_type: Object.fromEntries(Object.entries(byType).map(([k, v]) => [k, { ...v, acc: +(v.correct / v.total).toFixed(2) }])),
+ misroutes: ok.filter(r => !r.correct),
+ discrimination_only_errors: ok.filter(r => r.correct && !r.correctStrict).map(r => ({ id: r.id, expected: r.expected, chose: r.chosen })),
+ results,
+ };
+ const file = path.join(runDir(), 'tier2-routing.json');
+ writeJson(file, out);
+ console.log(`\n accuracy: lenient ${(acc * 100).toFixed(0)}% | strict/unique ${(accStrict * 100).toFixed(0)}%`);
+ for (const [t, v] of Object.entries(out.by_type)) console.log(` ${t}: ${v.correct}/${v.total} (${(v.acc * 100).toFixed(0)}%)`);
+ console.log(` -> ${file}`);
+}
+
+main();
diff --git a/evals/run-rubric.js b/evals/run-rubric.js
new file mode 100644
index 0000000..32008e7
--- /dev/null
+++ b/evals/run-rubric.js
@@ -0,0 +1,65 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Tier 1 — LLM-graded reasoning-quality rubric.
+ *
+ * A judge model reads each SKILL.md and scores its *content* (not headers) on
+ * fidelity / applicability / actionability / discrimination / discoverability.
+ * This is the substance signal the structural lint cannot produce.
+ *
+ * Usage: EVAL_RUN=run1 JUDGE_MODEL=gemini-3.1-pro-preview node evals/run-rubric.js [skill ...]
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { loadAllSkills, readSkill } = require('./lib/skills');
+const { droidJsonAsync, maxEffortFor } = require('./lib/droid');
+const { runDir, writeJson, mapPool } = require('./lib/io');
+
+const RUBRIC = fs.readFileSync(path.join(__dirname, 'rubrics', 'reasoning-quality.md'), 'utf8');
+const JUDGE = process.env.JUDGE_MODEL || 'gemini-3.1-pro-preview';
+const CONC = parseInt(process.env.CONC || '4', 10);
+
+function buildPrompt(skill) {
+ return `You are grading a Claude Code thinking skill against the rubric below.\n\n` +
+ `=== RUBRIC ===\n${RUBRIC}\n\n` +
+ `=== SKILL FILE: ${skill.name} ===\n${skill.content}\n\n` +
+ `=== END SKILL ===\n\n` +
+ `Return ONLY the JSON object specified in the rubric, with "skill" set to "${skill.name}". No prose.`;
+}
+
+async function main() {
+ const argv = process.argv.slice(2);
+ const skills = argv.length ? argv.map(readSkill) : loadAllSkills();
+ console.log(`Tier 1 — rubric grading ${skills.length} skills with judge=${JUDGE} (${maxEffortFor(JUDGE)}), conc=${CONC}`);
+
+ let n = 0;
+ const results = await mapPool(skills, CONC, async (skill) => {
+ const r = await droidJsonAsync({ model: JUDGE, prompt: buildPrompt(skill) });
+ n++;
+ if (!r.ok) { console.log(` [${n}/${skills.length}] ${skill.name}: FAILED (${r.error})`); return { skill: skill.name, ok: false, error: r.error, raw: r.raw }; }
+ const s = r.json.scores || {};
+ const avg = ['fidelity','applicability','actionability','discrimination','discoverability']
+ .map(k => Number(s[k]) || 0).reduce((a, b) => a + b, 0) / 5;
+ console.log(` [${n}/${skills.length}] ${skill.name}: avg ${avg.toFixed(2)}/5 weakest=${r.json.weakest_dimension || '?'}${r.json.would_mislead ? ' ⚠ MISLEAD' : ''}`);
+ return { skill: skill.name, ok: true, judge: JUDGE, ...r.json, avg };
+ });
+
+ const ok = results.filter(r => r.ok);
+ ok.sort((a, b) => a.avg - b.avg);
+ const out = {
+ tier: 1, judge: JUDGE, effort: maxEffortFor(JUDGE),
+ n: results.length, n_ok: ok.length,
+ overall_avg: ok.length ? (ok.reduce((a, r) => a + r.avg, 0) / ok.length) : 0,
+ would_mislead: ok.filter(r => r.would_mislead).map(r => ({ skill: r.skill, note: r.mislead_note })),
+ weakest: ok.slice(0, 10).map(r => ({ skill: r.skill, avg: Number(r.avg.toFixed(2)), weakest_dimension: r.weakest_dimension, top_fix: r.top_fix })),
+ results,
+ };
+ const file = path.join(runDir(), 'tier1-rubric.json');
+ writeJson(file, out);
+ console.log(`\n overall avg: ${out.overall_avg.toFixed(2)}/5 misleading flags: ${out.would_mislead.length}`);
+ console.log(` -> ${file}`);
+}
+
+main();
diff --git a/evals/run-structural.js b/evals/run-structural.js
new file mode 100644
index 0000000..88204b6
--- /dev/null
+++ b/evals/run-structural.js
@@ -0,0 +1,114 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Tier 0 — Structural lint (free, deterministic, no API cost).
+ *
+ * Reproduces the existing scripts/validate-skills.js 18-point "strict" score,
+ * AND computes a "loosened" substance-aware score that recognizes real content
+ * the strict regexes miss (e.g. a fenced template block, an examples section,
+ * an anti-patterns paragraph). The gap between the two quantifies how much the
+ * current quality number is measuring headers rather than substance — the
+ * `thinking-debiasing` 72% false-negative is the motivating case.
+ */
+
+const { loadAllSkills } = require('./lib/skills');
+const { runDir, writeJson } = require('./lib/io');
+const path = require('path');
+
+// ---- strict checks (copied from scripts/validate-skills.js) ----
+const STRICT_SECTIONS = [
+ { name: 'YAML Frontmatter', p: /^---\n[\s\S]*?name:[\s\S]*?description:[\s\S]*?---/m },
+ { name: 'Overview', p: /^## Overview/m },
+ { name: 'Core Principle', p: /\*\*Core Principle:\*\*/m },
+ { name: 'When to Use', p: /^## When to Use/m },
+ { name: 'Decision Flow', p: /```[\s\S]*?→[\s\S]*?```/m },
+ { name: 'Process/Steps', p: /^### Step \d|^## The .* Process|^## .*Process$/m },
+ { name: 'Examples', p: /^## .*Example|Example:|```markdown[\s\S]*?##/m },
+ { name: 'Template', p: /^## .*Template|Template:/m },
+ { name: 'Verification Checklist', p: /^## Verification Checklist/m },
+ { name: 'Key Questions', p: /^## Key Questions/m },
+ { name: 'Wisdom/Attribution', p: /Wisdom|".*"/m },
+];
+const STRICT_QUALITY = [
+ { name: 'Description Length', f: c => (c.match(/description:\s*(.+)/) || [, ''])[1].length <= 200 },
+ { name: 'Multiple Examples', f: c => ((c.match(/Example[s]?:/gi) || []).length + (c.match(/```markdown[\s\S]*?##/g) || []).length) >= 2 },
+ { name: 'Tables', f: c => (c.match(/\|.*\|.*\|/g) || []).length >= 3 },
+ { name: 'Checklists', f: c => (c.match(/- \[ \]/g) || []).length >= 3 },
+ { name: 'Code Blocks', f: c => (c.match(/```/g) || []).length / 2 >= 3 },
+ { name: 'Word Count', f: c => { const w = c.split(/\s+/).length; return w >= 500 && w <= 3000; } },
+ { name: 'Anti-patterns', f: c => /anti-pattern|failure mode|common mistake|when not to|don't use/i.test(c) },
+];
+
+function strictScore(content) {
+ let score = 0, max = 0;
+ const missing = [];
+ for (const s of STRICT_SECTIONS) { max++; if (s.p.test(content)) score++; else missing.push(s.name); }
+ for (const q of STRICT_QUALITY) { max++; if (q.f(content)) score++; else missing.push(q.name); }
+ return { score, max, pct: Math.round((score / max) * 100), missing };
+}
+
+// ---- loosened, substance-aware checks ----
+function looseScore(content) {
+ const checks = {
+ has_frontmatter: /^---\n[\s\S]*?name:[\s\S]*?description:[\s\S]*?---/m.test(content),
+ has_overview: /^## (Overview|Summary|What)/im.test(content),
+ has_core_idea: /\*\*Core Principle/i.test(content) || /^## (Core|The Principle|The Idea)/im.test(content),
+ has_when_to_use: /^## When to (Use|Apply)/im.test(content),
+ // any fenced block OR a fill-in template (e.g. a markdown block with [brackets])
+ has_template: /^## .*Template/im.test(content) || /```[\s\S]*?\[[^\]]+\][\s\S]*?```/m.test(content),
+ // examples: a section OR >=2 worked code blocks
+ has_examples: /^#{2,3} .*Example/im.test(content) || (content.match(/```/g) || []).length / 2 >= 2,
+ has_verification: /^## Verification|^## Checklist|- \[ \]/im.test(content),
+ has_key_questions: /^## (Key Questions|Questions)/im.test(content) || /\?\s*$/m.test(content),
+ // boundaries: when-not, failure modes, pitfalls, OR a comparison table
+ has_boundaries: /anti-pattern|failure mode|common (mistake|pitfall)|when not to|don't use|pitfall|trap/i.test(content),
+ has_attribution: /".{10,}"|—\s*\w|\bper\b|attributed|coined|developed by/i.test(content),
+ desc_ok: (content.match(/description:\s*(.+)/) || [, ''])[1].length <= 200,
+ word_ok: (() => { const w = content.split(/\s+/).length; return w >= 400 && w <= 3500; })(),
+ };
+ const vals = Object.values(checks);
+ const score = vals.filter(Boolean).length;
+ return { score, max: vals.length, pct: Math.round((score / vals.length) * 100), checks };
+}
+
+function main() {
+ const skills = loadAllSkills();
+ const rows = skills.map(s => {
+ const strict = strictScore(s.content);
+ const loose = looseScore(s.content);
+ return {
+ name: s.name,
+ descLen: s.description.length,
+ strict_pct: strict.pct,
+ loose_pct: loose.pct,
+ gap: loose.pct - strict.pct,
+ strict_missing: strict.missing,
+ words: s.content.split(/\s+/).length,
+ };
+ });
+ rows.sort((a, b) => b.gap - a.gap);
+
+ const overallStrict = Math.round(rows.reduce((a, r) => a + r.strict_pct, 0) / rows.length);
+ const overallLoose = Math.round(rows.reduce((a, r) => a + r.loose_pct, 0) / rows.length);
+
+ const out = {
+ tier: 0,
+ overall_strict_pct: overallStrict,
+ overall_loose_pct: overallLoose,
+ false_negatives: rows.filter(r => r.gap >= 15).map(r => ({ name: r.name, strict: r.strict_pct, loose: r.loose_pct })),
+ over_length_descriptions: rows.filter(r => r.descLen > 200).map(r => ({ name: r.name, len: r.descLen })),
+ rows,
+ };
+ const file = path.join(runDir(), 'tier0-structural.json');
+ writeJson(file, out);
+ console.log(`Tier 0 — structural lint`);
+ console.log(` strict overall: ${overallStrict}%`);
+ console.log(` loosened overall: ${overallLoose}%`);
+ console.log(` biggest false-negatives (strict undercounts substance):`);
+ for (const r of out.false_negatives.slice(0, 8)) console.log(` ${r.name}: strict ${r.strict}% vs loose ${r.loose}%`);
+ console.log(` over-length descriptions (>200): ${out.over_length_descriptions.map(d => `${d.name}(${d.len})`).join(', ') || 'none'}`);
+ console.log(` -> ${file}`);
+}
+
+main();
diff --git a/evals/run-swe.js b/evals/run-swe.js
new file mode 100644
index 0000000..6d39557
--- /dev/null
+++ b/evals/run-swe.js
@@ -0,0 +1,77 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * SWE-bench fault-file localization (native-domain debugging, objective, isolated).
+ * Given a real GitHub issue, the solver names the file to fix; scored against the
+ * gold patch's modified file(s). Tests whether a debugging skill (scientific-method,
+ * five-whys, systems, occams) improves fault localization vs a length-matched placebo.
+ * Paired McNemar + Wilson CI. No repo checkout / test execution required.
+ *
+ * Usage: EVAL_RUN=run1 FORCE_SKILL=scientific-method node evals/run-swe.js
+ */
+
+const fs = require('fs');
+const path = require('path');
+const { droidExecAsync } = require('./lib/droid');
+const { runDir, writeJson, mapPool } = require('./lib/io');
+const { neutralFiller, wordCount } = require('./lib/conditions');
+const { mcnemar, wilson } = require('./lib/stats');
+
+const SOLVER = process.env.SOLVER_MODEL || 'claude-sonnet-4-6';
+const SOLVER_EFFORT = process.env.SOLVER_EFFORT || 'medium';
+const CONC = parseInt(process.env.CONC || '4', 10);
+const FORCE_SKILL = process.env.FORCE_SKILL;
+const FILE = path.join(__dirname, 'datasets', 'external', 'swebench.jsonl');
+const SKILLS_DIR = path.join(__dirname, '..', 'skills');
+
+function loadItems() {
+ let L = fs.readFileSync(FILE, 'utf8').trim().split('\n').filter(Boolean).map(JSON.parse);
+ return process.env.LIMIT ? L.slice(0, parseInt(process.env.LIMIT, 10)) : L;
+}
+function skillContent(s) { return fs.readFileSync(path.join(SKILLS_DIR, 'thinking-' + s.replace(/^thinking-/, ''), 'SKILL.md'), 'utf8'); }
+
+function locatedCorrectly(item, text) {
+ // correct if a gold file path or its (distinctive) basename appears in the response
+ for (const g of item.gold_files) {
+ if (text.includes(g)) return true;
+ const base = g.split('/').pop();
+ if (base && !/^(__init__|utils|index|main|app|setup)\.\w+$/.test(base) && new RegExp('\\b' + base.replace('.', '\\.') + '\\b').test(text)) return true;
+ }
+ return false;
+}
+
+async function runItem(item, skillMd) {
+ const tail = `\n\nThink carefully and answer.`;
+ const skillP = `Use the following thinking-skill guide to reason about the bug. Apply it substantively.\n\n=== THINKING SKILL ===\n${skillMd}\n=== END SKILL ===\n\n${item.prompt}${tail}`;
+ const placeboP = `Some general notes:\n\n${neutralFiller(wordCount(skillMd))}\n\n${item.prompt}${tail}`;
+ const [s, p] = await Promise.all([
+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: skillP }),
+ droidExecAsync({ model: SOLVER, effort: SOLVER_EFFORT, prompt: placeboP }),
+ ]);
+ if (!s.ok || !p.ok) return null;
+ return { skill_correct: locatedCorrectly(item, s.text), placebo_correct: locatedCorrectly(item, p.text) };
+}
+
+async function main() {
+ if (!FORCE_SKILL) { console.error('set FORCE_SKILL'); process.exit(1); }
+ const skillMd = skillContent(FORCE_SKILL);
+ const items = loadItems();
+ console.log(`SWE-bench localization: ${items.length} issues, skill=${FORCE_SKILL}, solver=${SOLVER}(${SOLVER_EFFORT}), isolation ${process.env.EVAL_NO_ISOLATE === '1' ? 'OFF' : 'ON'}`);
+ let done = 0;
+ const r = (await mapPool(items, CONC, async it => { const x = await runItem(it, skillMd); done++; if (done % 10 === 0) console.log(` ${done}/${items.length}`); return x; })).filter(Boolean);
+
+ const SC = r.filter(x => x.skill_correct).length, PC = r.filter(x => x.placebo_correct).length, N = r.length;
+ const b = r.filter(x => x.skill_correct && !x.placebo_correct).length, c = r.filter(x => !x.skill_correct && x.placebo_correct).length;
+ const out = {
+ mode: 'swe-localize', skill: FORCE_SKILL, solver: SOLVER, n: N,
+ acc_with_skill: +(SC / N).toFixed(3), acc_with_skill_ci: wilson(SC, N).map(x => +x.toFixed(3)),
+ acc_placebo: +(PC / N).toFixed(3), acc_placebo_ci: wilson(PC, N).map(x => +x.toFixed(3)),
+ delta_pp: +(((SC - PC) / N) * 100).toFixed(1), mcnemar_p: +mcnemar(b, c).toFixed(3), significant: mcnemar(b, c) < 0.05, discordant: b + c,
+ };
+ const file = process.env.OUTFILE || path.join(runDir(), `swe-${FORCE_SKILL}.json`);
+ fs.mkdirSync(path.dirname(file), { recursive: true }); writeJson(file, out);
+ console.log(`\n localize WITH skill ${(out.acc_with_skill * 100).toFixed(0)}% CI[${(out.acc_with_skill_ci[0] * 100).toFixed(0)},${(out.acc_with_skill_ci[1] * 100).toFixed(0)}] vs placebo ${(out.acc_placebo * 100).toFixed(0)}% CI[${(out.acc_placebo_ci[0] * 100).toFixed(0)},${(out.acc_placebo_ci[1] * 100).toFixed(0)}] Δ${out.delta_pp >= 0 ? '+' : ''}${out.delta_pp}pp McNemar p=${out.mcnemar_p}${out.significant ? ' SIG' : ''}`);
+ console.log(` -> ${file}`);
+}
+main();