An agentic skills framework & software development methodology that works.
Spec: positive-instruction redesign — audit results, micro-test method, writing-plans variants
Jesse Vincent69a00350ff2d
diff --git a/docs/superpowers/specs/2026-06-10-positive-instruction-redesign-design.md b/docs/superpowers/specs/2026-06-10-positive-instruction-redesign-design.mdnew file mode 100644index 000000000..a19c938ca--- /dev/null+++ b/docs/superpowers/specs/2026-06-10-positive-instruction-redesign-design.md@@ -0,0 +1,164 @@+# Positive-Instruction Redesign of Skill Guidance — Design Spec++**Status:** Proposed (follow-up to the 2026-06-09 SDD review-dispatch work; separate PR per the one-problem-per-PR rule)+**Driver:** Measured evidence (2026-06-10) that some negative instructions in skill prose backfire, while others work — and that the difference is predictable.++## The measured finding this spec generalizes++Micro-tests on 2026-06-10 (opus, 5 reps per phrasing, programmatic scoring;+harness described below) measured how guidance phrasing changes what a+controller composes:++| Case | Phrasing | Result |+|---|---|---|+| Dispatch composition ("don't restate the brief") | prohibition | **4.4** spec values re-typed — *worse than no guidance* (3.6) |+| Dispatch composition | positive recipe ("your dispatch should contain: (1)…(5)") | **3.0, zero variance** — adopted |+| Dispatch composition | recipe + nuance clause ("quote only the fragment…") | 3.8, noisy — nuance dilutes recipes |+| Test-rerun directive ("do not ask reviewer to re-run tests") | prohibition | **0/5 violations** — works fine (control: 3/5) |+| Test-rerun directive | positive recipe | 0/5 — equal, but longer |++**The doctrine** (use this to classify any negative instruction):++1. **Tripwires work.** Phrase-level self-checks on concrete tokens ("if the+ prompt you are writing contains 'do not flag' … stop") fire reliably.+2. **Recognition tables work.** Red-Flags/rationalization tables read at+ decision time, not composition time.+3. **Discrete-directive prohibitions work.** "Do not ask X to do Y" holds+ when the model has no competing incentive to do Y.+4. **Composition prohibitions backfire** when the model has its own agenda+ for the output (e.g., restating specs feels like helpful curation).+ Only a positive composition recipe moves these — and adding nuance+ clauses to a winning recipe makes it worse, not better.+5. **Ties go to the shorter phrasing.** Codex re-reads SKILL.md ~500× per+ long session (measured 2026-06-10); prose length is a real cost.++## Audit results (2026-06-10, all ~30 skills + prompt templates)++Counts: 3 tripwires (keep), 14 recognition tables (keep), ~20 policy gates+(keep — "never push without permission" is policy, not composition+shaping), 5 composition-prohibitions:++| # | Location | Disposition |+|---|---|---|+| 1 | `subagent-driven-development/task-reviewer-prompt.md` — "Cite, don't narrate" | **Queued in PR #1717 batch**: lead with the positive half ("Your report should point at evidence: file:line for every finding…"), drop the prohibition half (dead weight — the positive half already exists and carries the load) |+| 2 | `subagent-driven-development/SKILL.md` — "Do not add open-ended directives" | **Keep as-is**: micro-test could not elicit the failure in 15 samples; no evidence either way; shorter wins |+| 3 | `subagent-driven-development/SKILL.md` — "Do not ask a reviewer to re-run tests" | **Keep as-is**: measured 0/5 violations; the prohibition also usefully propagates itself into dispatches |+| 4 | `subagent-driven-development/SKILL.md` — "do not re-review on top of it" | **Queued in PR #1717 batch**: replace with the three-element checklist ("Before re-dispatching the reviewer, confirm the fix report contains: the covering tests, the command run, and the output") |+| 5 | `writing-plans/SKILL.md` — the "No Placeholders" banned-patterns list | **This spec's main subject** — see below |++Borderline, deferred with #5: `task-reviewer-prompt.md` "Don't flag+pre-existing file sizes — focus on what this change contributed" (positive+half present and load-bearing; low impact; test alongside #5 if convenient).++## The writing-plans change (deferred item #5)++### Current state++`skills/writing-plans/SKILL.md`, "No Placeholders": one positive sentence+("Every step must contain the actual content an engineer needs") followed+by a six-bullet banned-patterns list ("never write them: 'TBD', 'TODO',+'Add appropriate error handling', 'Write tests for the above', 'Similar to+Task N', …").++### Why it matters and why it is genuinely uncertain++- Plans are the **largest generated artifact** in the workflow, and the+ model has a real competing incentive to emit placeholders (they are the+ path of least effort under length pressure) — the incentive structure of+ the case where prohibition measurably backfired.+- But the banned items are **discrete, recognizable tokens** — the shape+ of the case where prohibition measurably held.+- **The list is load-bearing elsewhere:** the skill's Self-Review section+ references it ("Placeholder scan: search your plan for red flags — any+ of the patterns from the 'No Placeholders' section above"). The tokens+ double as the review-time scan inventory, and review-time recognition is+ the category that works. A naive swap to a positive checklist breaks+ that reference and discards good tripwire tokens.++### Variants to test++- **V0 (current):** positive sentence + banned list at composition time;+ Self-Review references the list.+- **V1 (auditor's checklist):** composition-time positive recipe only —+ "Before finalizing a step, confirm it has: the literal code to write, a+ runnable command with expected output, types and method names defined+ within this plan, error handling shown explicitly. A step is complete+ when an engineer could implement it without asking any follow-up+ questions." Self-Review keeps a generic placeholder scan.+- **V2 (restructure by mechanism — predicted winner):** composition time+ gets only V1's positive recipe; the named patterns move wholesale into+ the Self-Review placeholder-scan step, reframed as recognition ("when+ you scan, look for: 'TBD', 'TODO', 'Similar to Task N', …"). Same+ tokens, relocated from the category that primes to the category that+ detects.+- **V3 (control):** positive sentence only, no list anywhere.++### Micro-test design++- **Task:** opus writes a 2-3 task implementation plan from a deliberately+ under-specified spec (under-specification is what tempts placeholders).+ Use a fixture spec with: one well-specified task, one task whose error+ handling the spec hand-waves, one task similar to the first (tempting+ "Similar to Task 1").+- **Sampling:** 5+ reps per variant, default temperature, model+ `claude-opus-4-8` (the model that writes plans in practice).+- **Programmatic scoring** (lower is better unless noted):+ - banned-token count: `TBD|TODO|implement later|fill in details|appropriate error handling|handle edge cases|Similar to Task|Write tests for the above`+ - steps lacking a fenced code block where the step changes code+ - references to types/functions not defined anywhere in the plan output+ - (higher is better) runnable commands with expected output per task+- **Two-stage scoring for V2:** also test the Self-Review half — feed each+ generated plan back with the variant's Self-Review section and measure+ whether the scan actually catches seeded placeholders (insert 2 known+ placeholders into a fixture plan; detection rate is the metric).+- **Acceptance:** adopt a variant only if it beats V0 on banned-token count+ without losing code-block coverage or self-review detection rate.+ Expected cost: ~$6-10 total.++### PR scoping++Separate PR (writing-plans is a different skill; its "No Placeholders"+list is tuned content where the contributor guidelines demand eval+evidence). The PR must include: the micro-test harness + results table,+before/after text, and the V2 relocation rationale.++## The micro-test harness (method, so it isn't lost)++`/tmp/sdd-exp/micro/run-micro.py` and `/tmp/sdd-exp/micro2/run-micro2.py`+(2026-06-10; to be committed to superpowers-evals as+`docs/superpowers/skills/micro-testing-prompt-guidance.md` + scripts):++- One API call per sample: system prompt = the skill-guidance variant in+ realistic surrounding context; user = a realistic mid-workflow scenario;+ output = the composed artifact (dispatch prompt, plan, report).+- Programmatic scoring with greps for unambiguous markers; **manually+ inspect every match before trusting a verdict** — one of tonight's+ "violations" was the controller correctly quoting the prohibition, and+ automated negation detection mislabeled another.+- ~$0.15-0.30/sample, seconds per iteration vs $12/50-min full eval runs.+ Iterate phrasings here; confirm winners in full runs only when the+ change is structural.+- Always include a no-guidance control — tonight it revealed both a+ backfire (restating: prohibition worse than nothing) and a working+ prohibition (test-reruns: 3/5 control failures vs 0/5 with either+ phrasing).++## Also explicitly not-dropped (tested-and-declined, with data)++Recorded so nobody re-proposes them without new evidence — full numbers in+the 2026-06-09 SDD design spec's Cost-iterations section:++- **Controller turn batching / parallel tool calls in one message:** the+ controller emits exactly one tool call per message (0 multi-tool+ messages across every measured run, with and without guidance). 46% of+ controller turns are thinking/narration with no tool call — a+ prompt-immune floor.+- **Pipelined reviews via parallel calls:** dead for the same reason.+- **Pipelined reviews via `run_in_background`:** mechanism adopted when+ offered (7/28 dispatches) but benefit below the run-to-run noise floor+ on 45-min scenarios (reviews are only ~30-60s each); adds dual+ result-stream coordination. Worth revisiting only for plans whose+ reviews are individually long.+- **Nuance clauses appended to winning recipes:** measurably degrade them+ (C2: 3.8 noisy vs C: 3.0 consistent). Iterate by re-deriving the recipe,+ not by appending caveats.