An agentic skills framework & software development methodology that works.
experiment: ground-up two-principle rewrite of writing-good-tests
Jesse Vincentcaa1826cbade
diff --git a/skills/test-driven-development/writing-good-tests.md b/skills/test-driven-development/writing-good-tests.mdindex 3cae73bc2..d3c4482fd 100644--- a/skills/test-driven-development/writing-good-tests.md+++ b/skills/test-driven-development/writing-good-tests.md@@ -5,393 +5,194 @@ adding cleanup/helper methods for tests.## Overview-Good tests verify real behavior. Mocks exist to isolate the code under-test — they are never the thing being tested.--**Core principle:** Test what the code does, not what the mocks do — and-make every test able to fail.--Strict TDD produces every rule below naturally: a test written first and-watched failing against real code has already proven it can fail, and-only earns a mock when the real dependency proves slow or external. A-test asserting on a mock means TDD was skipped somewhere.--## The Iron Laws+A test exists to catch a specific break. Two principles govern everything+here:```-1. Every test can fail — name the production change that would fail it-2. Assert on real behavior, never on mock behavior-3. Production classes carry production methods only-4. Understand a dependency's side effects before mocking it+1. Every test names the break it catches+2. Every test exercises the real thing```-## Rule 1: Write Tests That Can Fail+Strict TDD produces both naturally: a test written first and watched+failing against real code has already proven it can fail, and only earns+a mock when the real dependency proves slow or external.-Before writing or changing a test, name the production change that would-make it fail. If you cannot, redesign the test around an observable-behavior — a test that cannot fail protects nothing.+## Principle 1: Name the Break-Derive expected values independently of the code under test: literals,-hand-checked fixtures, small worked examples, or invariant assertions.-Keep test logic simple enough to review by inspection — table-driven-tests with literal `want` values are the preferred shape.+Before writing the test body, answer: **what production change should+make this test fail — and is that change a bug or a decision?** A test+earns its place by catching a wrong branch, missing side effect, wrong+argument, boundary case, or broken contract.-```typescript-// ✅ GOOD: literal, hand-derived expectation-test('builds tag query', () => {- expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"');-});-```+**Derive expectations independently.** Use literals and hand-checked+fixtures; table-driven tests with literal `want` values are the preferred+shape. An expectation computed by the code under test — or its helpers —+passes no matter what that code does:```typescript-// ❌ The violation: expectation computed by the logic under test-test('builds tag query', () => {- const expected = buildSearchQuery({ tag: 'urgent' }); // same builder!- expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected); // always true-});--// ❌ Subtler: the expectation reuses the same helper the code calls-test('formats timestamp', () => {- expect(render(entry)).toContain(formatTime(entry.ts)); // mirrors implementation-});-```--A mirror assertion re-derives the answer with the answer's own machinery:-it passes no matter what that machinery does.--**Name the break, not just the change.** A test earns its place by-catching a wrong branch, missing side effect, wrong argument, boundary,-or broken contract. If only intentional decisions can fail it — a-constant's value, exact message wording — it is a change detector: it-fires on redesign and sleeps through bugs.--**The string-presence trap.** Asserting that a script, skill, or config-contains an exact line counterfeits falsifiability: it proves only that-the source is the source, breaking on every rewording and surviving every-real regression. Run scripts and assert outputs, side effects, or exit-codes; test agent-instructing documents by their consumer's behavior.-Text containment is never the observable.+// ❌ Mirror assertion: the same builder computes both sides — always true+const expected = buildSearchQuery({ tag: 'urgent' });+expect(buildSearchQuery({ tag: 'urgent' })).toBe(expected);++// ✅ Hand-derived literal+expect(buildSearchQuery({ tag: 'urgent' })).toBe('tag:"urgent"');+```++**No change detectors.** If only intentional decisions can fail a test —+a constant's value, exact message wording, private structure — it fires+on redesign and sleeps through bugs. Test the behavior that depends on+the decision: not `expect(MAX_RETRIES).toBe(5)` but "a failing call is+retried 5 times and the 6th attempt never happens."++**Behavior, not text.** Asserting that a script, skill, or config+contains an exact line proves only that the source is the source. Run+scripts against controlled inputs and assert outputs, side effects, or+exit codes. Documents that instruct agents are tested by the consuming+agent's behavior (superpowers:writing-skills); prose for humans earns no+test at all.++**Your code, not the framework.** Test the contract your code makes at+its boundaries — the route you register, the query you emit, the payload+you produce. Upstream mechanics are their maintainers' tests to write+(the classic: asserting your router invokes a registered handler — that+is the framework's test, not yours). When upstream behavior genuinely+surprised you, write one narrow characterization test naming the+assumption. The same boundary applies inside your code: constructors,+getters, constants, and trivial forwarding earn tests only when they+validate, normalize, default, derive, enforce, or cause side effects —+otherwise assert the first consumer-visible result that depends on them.### Gate Function```BEFORE writing the test body:- Ask: "What production change should make this test fail?"-- IF you cannot name one:- STOP - Redesign the test around an observable behavior+ Name the production change that would make this test fail.- IF the only answer is "the source text changed":- STOP - Run the artifact and assert its effects instead+ Cannot name one → redesign around an observable behavior+ "The source text changed" → run the artifact and assert its effects+ Only intentional decisions → change detector; test the behavior+ that depends on the decision- Ask: "What BREAK would this catch?"-- IF every failing change is an intentional decision, never a bug:- STOP - That is a change detector; test the behavior that- depends on the decision instead-- Ask: "Is the expected value derived independently of the code under test?"-- IF it reuses the code's own logic or helpers:- STOP - Replace it with a literal or hand-checked fixture+ Confirm the expected value is derived without the code under test.+ IF it reuses the code's logic or helpers:+ Replace it with a literal or hand-checked fixture```-## Rule 2: Assert on Real Behavior--```typescript-// ✅ GOOD: Test the real component-test('renders sidebar', () => {- render(<Page />); // Sidebar unmocked- expect(screen.getByRole('navigation')).toBeInTheDocument();-});-```+## Principle 2: Exercise the Real Thing-If the sidebar must be mocked for isolation, assert on Page's behavior-with the sidebar present — the mock itself earns no assertions.+**The mock earns no assertions.** A mock assertion passes when the mock+is present and fails when it is absent — it says nothing about the+component. Assert the real component's behavior; if the mock is what you+are checking, unmock it or delete the assertion.```typescript-// ❌ The violation: asserting that the mock exists-test('renders sidebar', () => {- render(<Page />);- expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument();-});-```--A mock assertion passes when the mock is present and fails when it is-absent — it says nothing about the component. **your human partner's-correction:** "Are we testing the behavior of a mock?"--### Gate Function+// ✅ Real behavior+expect(screen.getByRole('navigation')).toBeInTheDocument();+// ❌ Mock existence+expect(screen.getByTestId('sidebar-mock')).toBeInTheDocument();```-BEFORE asserting on any mock element:- Ask: "Am I testing real component behavior or just mock existence?"- IF testing mock existence:- STOP - Delete the assertion or unmock the component+**your human partner's correction:** "Are we testing the behavior of a+mock?"- Test real behavior instead-```--## Rule 3: Keep Test Cleanup in Test Utilities--```typescript-// ✅ GOOD: Test utilities own test cleanup-// Session has no destroy() - it's stateless in production--// In test-utils/-export async function cleanupSession(session: Session) {- const workspace = session.getWorkspaceInfo();- if (workspace) {- await workspaceManager.destroyWorkspace(workspace.id);- }-}--// In tests-afterEach(() => cleanupSession(session));-```+**Mock at the right level.** Learn every side effect of the real method+before replacing it; mock the slow or external operation and keep what+the test depends on real. When unsure, run the test against the real+implementation first and observe what actually needs to happen.```typescript-// ❌ The violation: destroy() exists only for tests-class Session {- async destroy() { // Looks like production API!- await this._workspaceManager?.destroyWorkspace(this.id);- // ... cleanup- }-}--// In tests-afterEach(() => session.destroy());-```--A test-only method pollutes the production class, is dangerous if-production code ever calls it, and confuses object lifecycle with entity-lifecycle.--### Gate Function+// ❌ The mock swallows the config write that duplicate detection reads+vi.mock('ToolCatalog', () => ({+ discoverAndCacheTools: vi.fn().mockResolvedValue(undefined)+}));+// ✅ Mock only the slow server startup; the config write stays real+vi.mock('MCPServerManager');```-BEFORE adding any method to a production class:- Ask: "Is this only used by tests?"- IF yes:- STOP - Put it in test utilities instead+**Make doubles specific.** When arguments, call counts, or ordering are+part of the contract, assert them — a fake that accepts anything verifies+nothing. Give each branch (success, error, malformed) its own fixture or+spy, so the wrong branch cannot satisfy the expectation.- Ask: "Does this class own this resource's lifecycle?"+**Mirror real data completely.** Mock the complete structure as it exists+in reality — all documented fields — not just the ones your test reads.+Partial mocks fail silently when downstream code reads an omitted field:+the test passes while integration breaks.- IF no:- STOP - Wrong class for this method-```+**Production classes carry production methods only.** Cleanup that only+tests need lives in test utilities, never as a `destroy()` on the+production class. Ask: is this method called only from tests? Does this+class own this resource's lifecycle? Wrong answers → test utility.-## Rule 4: Mock at the Right Level--Learn what the real method does — every side effect — before replacing-it. Mock the slow or external operation and preserve the behavior your-test depends on.--Make doubles specific to their contract: when arguments, call counts, or-ordering matter, assert them — a fake that accepts anything verifies-nothing. And give each branch its own double: success, error, and-malformed paths each get their own fixture or spy, so the wrong branch-cannot satisfy the expectation.--```typescript-// ✅ GOOD: Mock the slow part, preserve behavior the test needs-test('detects duplicate server', () => {- vi.mock('MCPServerManager'); // Just mock slow server startup-- await addServer(config); // Config written- await addServer(config); // Duplicate detected ✓-});-```--```typescript-// ❌ The violation: the mock swallows the side effect the test depends on-test('detects duplicate server', () => {- // Mock prevents the config write that duplicate detection reads!- vi.mock('ToolCatalog', () => ({- discoverAndCacheTools: vi.fn().mockResolvedValue(undefined)- }));-- await addServer(config);- await addServer(config); // Should throw - but won't!-});-```+**Prefer real components over complex mocks.** When mock setup outgrows+the test logic, mocks miss methods the real components have, or tests+break when the mock changes, switch to an integration test with real+components. **your human partner's question:** "Do we need to be using a+mock here?"### Gate Function```-BEFORE mocking any method:- STOP - Understand before replacing-- 1. Ask: "What side effects does the real method have?"- 2. Ask: "Does this test depend on any of those side effects?"- 3. Ask: "Do I fully understand what this test needs?"-- IF the test depends on side effects:- Mock at the lower level (the actual slow/external operation)- OR use test doubles that preserve the necessary behavior- — keep the high-level method the test depends on real-- IF unsure what the test depends on:- Run the test with the real implementation FIRST- Observe what actually needs to happen- THEN add minimal mocking at the right level-- Warning signs:- - "I'll mock this to be safe"- - "This might be slow, better mock it"- - Mocking before tracing the dependency chain-```--## Rule 5: Mirror Real Data Completely--Mock the COMPLETE data structure as it exists in reality, not just the-fields your immediate test uses.--```typescript-// ✅ GOOD: Mirror real API completeness-const mockResponse = {- status: 'success',- data: { userId: '123', name: 'Alice' },- metadata: { requestId: 'req-789', timestamp: 1234567890 }- // All fields real API returns-};-```+BEFORE adding a mock or test helper:+ List the real method's side effects; keep the ones the test+ depends on real — mock the slow/external level below them.-```typescript-// ❌ The violation: only the fields you thought you needed-const mockResponse = {- status: 'success',- data: { userId: '123', name: 'Alice' }- // Missing: metadata that downstream code uses-};--// Later: breaks when code accesses response.metadata.requestId-```+ Mock responses mirror the complete real structure.-Partial mocks hide structural assumptions and fail silently when-downstream code reads an omitted field: the test passes while integration-breaks.--### Gate Function--```-BEFORE creating mock responses:- Check: "What fields does the real API response contain?"-- Actions:- 1. Examine the actual API response from docs/examples- 2. Include ALL fields the system might consume downstream- 3. Verify the mock matches the real response schema completely-- If uncertain: include all documented fields-```--## Rule 6: Test Your Code, Not the Framework--Test the contract your code makes at its boundaries — the route you-register, the query you emit, the payload shape you produce, the value-handoff between layers. Dependencies' documented mechanics are their-maintainers' tests to write.--```typescript-// ✅ GOOD: your contract at the boundary-test('GET /sessions/:id returns 404 for unknown id', async () => {- const res = await request(app).get('/sessions/nope');- expect(res.status).toBe(404);- expect(res.body.error).toMatch(/not found/); // contract, not exact copy-});-```+ A method only tests call lives in test utilities, not production.-```typescript-// ❌ The violation: re-proving the router works as documented-test('router calls handler for matching route', () => {- const handler = vi.fn();- router.get('/x', handler);- router.handle(makeRequest('/x'));- expect(handler).toHaveBeenCalled();-});+ About to assert on the mock itself?+ Unmock it or delete the assertion.```-When upstream behavior genuinely surprised you (a quoting rule, an event-ordering), write one narrow characterization test around your integration-point and name the assumption in the test name or a comment.--The same boundary applies inside your own code: test behavior, not that-the implementation is written the way it is currently written. Plain-constructor assignment, getters, constants, trivial forwarding, and-data-only structs earn tests only when they validate, normalize, default,-derive, enforce, or cause side effects — otherwise assert the first-consumer-visible result that depends on them.--## Rule 7: Tests Ship With the Implementation--Testing is part of implementation. The TDD cycle — failing test, minimal-implementation, refactor — is what "complete" means; "implementation-complete, ready for testing" describes an unfinished task.+## Tests Ship With the Implementation-Ship the tests the behavior needs — and only those. Trivial-code changes-(Rule 6) and prose for humans (READMEs, comments, docs) earn no test:-there is no behavior to protect, and a test written to satisfy process-costs maintenance forever. Skills and prompts follow their own discipline-— pressure-test the consuming agent when an edit changes behavior-(superpowers:writing-skills) — never their text.--## Rule 8: Prefer Real Components Over Complex Mocks--Integration tests with real components are often simpler than elaborate-mocks. Reach for one when you see:--- Mock setup longer than the test logic-- Mocking everything to make the test pass-- Mocks missing methods the real components have-- Tests breaking when the mock changes--**your human partner's question:** "Do we need to be using a mock here?"+The TDD cycle — failing test, minimal implementation, refactor — is what+"complete" means. Ship the tests the behavior needs and only those:+trivial code and human prose earn none, and a test written to satisfy+process costs maintenance forever.## The Mutation Check-Before finishing, mentally mutate the production code. At least one test+Before finishing, mentally mutate the production code; at least one testshould fail for each realistic mutation:- Wrong constant or argument- Wrong branch handler-- Missing state change or side effect (row not written, event not emitted)+- Missing state change or side effect- Empty or default return- Missing validation for zero, empty, nil, unauthorized, or malformed input-A mutation no test can catch marks the behavior as unprotected — or the+A mutation nothing catches marks the behavior as unprotected — or thetest as tautological.## Quick Reference| When you... | Do ||-------------|-----|-| Write any test | Name the production change that would make it fail |-| Build an expected value | Derive it independently — literal or hand-checked fixture |+| Write any test | Name the break it catches — a bug, not a decision |+| Build an expected value | Derive it by hand; never with the code under test |+| Test a script or document | Run it / pressure-test its consumer; never grep its text |+| Reach for a dependency test | Test your boundary contract, not their documented mechanics || Want to assert on a mocked element | Test the real component, or unmock it |-| Need cleanup that only tests use | Put it in test utilities |-| Are about to mock a method | Learn its side effects first; mock the slow/external level |+| Are about to mock a method | Learn its side effects; mock the slow/external level || Build a mock response | Mirror the real structure completely |-| Reach for a dependency test | Test your boundary contract, not their documented mechanics |-| Finish an implementation | Tests already exist (TDD) — or it is unfinished |-| Finish a test file | Run the mutation check |+| Need cleanup only tests use | Put it in test utilities || Watch mock setup balloon | Switch to an integration test with real components |+| Finish a test file | Run the mutation check |## Warning Signs-- An assertion checks for a `*-mock` test ID-- A method is called only from test files-- Mock setup is more than half the test-- The test fails when you remove the mock-- You can't explain why the mock is needed-- Mocking "just to be safe"- Setup and assertion share the same object, guaranteeing equality- The test can fail only through a panic, crash, or missing selector-- The test would still matter if only the framework remained+- The test fails on every intentional change, never on accidental breakage- Expected values are hidden behind loops, builders, or helpers-- The test greps source text instead of observing behavior-- The test asserts that a removed function, file, or symbol stays removed-- The test exists for coverage, checking no side effect, boundary, or outcome-- The test fails on every intentional change and never on accidental breakage+- The test greps source text, or asserts a removed symbol stays removed+- The test would still matter if only the framework remained+- The test exists for coverage, checking no side effect or outcome+- An assertion checks a `*-mock` test ID, or fails if you remove the mock+- A method is called only from test files+- Mock setup is more than half the test, or you can't explain why the mock is needed+- Mocking "just to be safe"