Skip to content

28 eval-informed mental models and critical-thinking skills for Claude Code, GitHub Copilot, Codex, Cursor, and other Agent Skills-compatible tools

docs(repo): eval-informed metadata, README banner, agent guidelines

Travis Boudreauxaf0c31e8b29c
diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
index 3e04165..6dbb5d4 100644
--- a/.claude-plugin/marketplace.json
+++ b/.claude-plugin/marketplace.json
@@ -5,14 +5,14 @@
"email": "[email protected]"
},
"metadata": {
- "description": "Mental Models and Thinking Frameworks for Claude Code - 39 battle-tested frameworks for better decision-making",
+ "description": "Mental models and thinking skills for Claude Code, with eval-informed guidance for better reasoning",
"version": "1.0.0"
},
"plugins": [
{
"name": "thinking-skills",
"source": "./",
- "description": "39 Mental Models and Frameworks for Critical Thinking including first-principles, Bayesian reasoning, systems thinking, TRIZ, OODA, pre-mortem analysis, and more",
+ "description": "39 eval-informed thinking skills for Claude Code, including first-principles, Bayesian reasoning, systems thinking, TRIZ, OODA, pre-mortem analysis, and more",
"version": "1.0.0",
"category": "productivity",
"tags": [
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 1c0bc47..d5ebc2d 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
{
"name": "thinking-skills",
- "description": "39 Mental Models and Frameworks for Critical Thinking in Claude Code",
+ "description": "39 eval-informed thinking skills and mental models for Claude Code",
"version": "1.0.0",
"author": {
"name": "TJ Boudreaux",
diff --git a/.gitignore b/.gitignore
index 1461716..e60655f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,6 +4,8 @@
.claude/
# transient run logs
*.log
+# local backups of global Claude state; do not commit
+backups/
# scratch eval runs (regenerable)
evals/results/smoke/
evals/results/pilot/
diff --git a/AGENTS.md b/AGENTS.md
new file mode 100644
index 0000000..783928c
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1,26 @@
+# Agent Guidelines
+
+This repository packages 39 Claude Code thinking skills. Treat it as a plugin marketplace project first and an eval research project second.
+
+## Default Workflow
+
+- Preserve the public skill count at 39 unless a change intentionally adds/removes a shipped skill and updates README, plugin metadata, routing cases, and eval docs together.
+- Keep skill frontmatter descriptions situation-named and under 200 characters.
+- Prefer agent-native instructions over human facilitation language.
+- Add explicit "When NOT to Use" boundaries for every non-router skill.
+- Do not commit local backups, downloaded third-party datasets, transient logs, or scratch eval runs.
+
+## Verification
+
+- Run `node scripts/validate-skills.js` after skill edits.
+- Run `EVAL_RUN=<name> node evals/run-structural.js` after catalog edits.
+- Run routing or behavioral evals only when the changed surface warrants the cost.
+- Use current post-edit evidence only; historical pre-edit results are context, not proof.
+
+## Local Backups
+
+Global Claude assets may be backed up under `backups/` for safety. The directory is gitignored. Do not add backup archives to commits.
+
+## Images
+
+README and marketing images live in `assets/`. Keep generated source paths out of README links; copy selected assets into the repo and leave generator cache files untouched.
diff --git a/CLAUDE.md b/CLAUDE.md
new file mode 100644
index 0000000..0a083a7
--- /dev/null
+++ b/CLAUDE.md
@@ -0,0 +1,17 @@
+# Claude Project Notes
+
+Follow `AGENTS.md` for repository workflow.
+
+Key project facts:
+- Shipped skill count: 39.
+- `thinking-scientific-method` is the shipped hypothesis-differential debugging skill. Do not reintroduce a separate v2 variant.
+- Eval claims must identify whether they are pre-edit, post-edit, directional, or statistically significant.
+- Backups of global Claude state belong in `backups/` and must remain uncommitted.
+
+Common verification commands:
+
+```bash
+node scripts/validate-skills.js
+EVAL_RUN=local node evals/run-structural.js
+EVAL_RUN=local node evals/run-routing.js
+```
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 19032af..f4d0ae0 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -128,6 +128,24 @@ The `description` field is critical - it helps Claude Code understand when to in
- Focus on the use case, not the theory
- Include keywords that users might mention
+### Eval Expectations
+
+Structural quality is not enough. Any change that alters a skill's reasoning procedure or scope should include the narrowest relevant eval evidence:
+
+- Run `node scripts/validate-skills.js` for format regressions.
+- Run `EVAL_RUN=<name> node evals/run-structural.js` after skill catalog changes.
+- For routing/description changes, run `EVAL_RUN=<name> node evals/run-routing.js`.
+- For behavioral claims, use length-controlled skill-vs-placebo comparisons and document the dataset, model, N, and p-value.
+- For debugging-skill claims, prefer `evals/run-swe.js` on SWE-bench localization.
+
+Do not claim a skill is "proven" or "firm" from pre-edit results. Re-run after edits, especially after trimming content.
+
+### Generated and Local Artifacts
+
+- Do not commit `backups/`; it contains local backups of global Claude state.
+- Do not commit third-party downloaded datasets under `evals/datasets/external/*.jsonl` unless they are authored here and license-cleared.
+- README images should live under `assets/` and be referenced with relative paths.
+
## Pull Request Process
1. **Fork** the repository
diff --git a/README.md b/README.md
index c27ffbc..6d10590 100644
--- a/README.md
+++ b/README.md
@@ -8,10 +8,13 @@ A comprehensive collection of thinking skills for [Claude Code](https://claude.a
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Skills Count](https://img.shields.io/badge/Skills-39-blue)](https://github.com/tjboudreaux/cc-thinking-skills)
+![Claude Code Thinking Skills banner](assets/readme-banner.png)
+
## Features
- **39 Thinking Frameworks** - Comprehensive mental models for better decision-making
-- **Battle-Tested** - Based on proven frameworks from cognitive science and systems thinking
+- **Eval-Informed** - Includes an isolated evaluation harness and evidence notes for elevate/kill decisions
+- **Battle-Tested Foundations** - Based on proven frameworks from cognitive science and systems thinking
- **Claude Code Native** - Designed specifically for Claude Code's skill system
- **Quality Scripts** - Tools to validate and enhance skill quality
- **Zero Configuration** - Just install and invoke with skill names
@@ -112,7 +115,7 @@ Once installed, invoke any skill by name in Claude Code:
| `thinking-circle-of-competence` | Know the boundaries of expertise | Delegation, learning decisions |
| `thinking-triz` | Resolve technical contradictions | Engineering design, innovation |
| `thinking-five-whys-plus` | Enhanced root cause analysis with bias guards | Debugging, incident postmortems |
-| `thinking-scientific-method` | Hypothesis-driven investigation | Debugging, A/B testing, experimentation |
+| `thinking-scientific-method` | Hypothesis-differential debugging | Fault localization, ambiguous symptoms |
| `thinking-thought-experiment` | Structured imagination for exploration | Architecture, edge cases, philosophy |
### Estimation & Risk
@@ -144,6 +147,18 @@ Once installed, invoke any skill by name in Claude Code:
This collection includes scripts to maintain and improve skill quality:
+### Outcome Evals
+
+The `evals/` and `experiments/` directories contain the current outcome-based harness:
+
+- **Structural lint** for frontmatter and format checks
+- **Routing evals** for skill discoverability and false-positive control
+- **Length-controlled behavioral evals** using skill-vs-placebo prompts
+- **Objective SWE-bench localization evals** for debugging skills
+- **SQLite dashboard** for reviewing eval and experiment results
+
+Current evidence is documented in `analysis/ELEVATE-OR-KILL.md`. The strongest post-improvement result is `thinking-scientific-method`, now implemented as hypothesis-differential debugging: SWE-bench fault localization improved from a flat original to **+9.3pp, p=0.002** after the agent-native rewrite. Other debugging skills need careful revalidation after trimming; do not treat old pre-trim results as current.
+
### Validate Skills
Check all skills against quality criteria:
@@ -215,6 +230,14 @@ Every system has exactly one constraint limiting throughput. Optimizing anything
- Resource allocation
- Identifying bottlenecks
+### Scientific Method / Hypothesis-Differential Debugging
+Localize an ambiguous bug by enumerating falsifiable hypotheses, ranking them by likelihood x cheapness-to-check, and making the cheapest discriminating observation first.
+
+**When to use:**
+- A symptom could plausibly come from several files/functions/components
+- You can inspect code, logs, diffs, traces, or tests now
+- You need to localize the fault before applying root-cause analysis
+
### Cynefin Framework
Classify problems by the relationship between cause and effect: Clear, Complicated, Complex, or Chaotic. Each domain requires a different approach.
diff --git a/assets/readme-banner.png b/assets/readme-banner.png
new file mode 100644
index 0000000..af21690
Binary files /dev/null and b/assets/readme-banner.png differ