An agentic skills framework & software development methodology that works.
eval(sdd): GREEN results — plan-scoped resolution replaces cross-plan forensics
Jesse Vincente7826745ee25
diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.mddeleted file mode 100644index df4add36d..000000000--- a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md+++ /dev/null@@ -1,95 +0,0 @@-# SDD plan-scoped workspace — RED baseline eval notes--- **Date:** 2026-07-06-- **Status:** interim evidence, compiled from three already-completed eval rounds — no new scenario runs in this pass. Folded into `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md` and deleted when Task 4 completes.-- **Spec:** `docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`-- **Plan:** `docs/superpowers/plans/2026-07-06-sdd-plan-scoped-workspace.md` (Task 1)--## Method--Three rounds of pressure-testing ran against the released (pre-Task-3) `subagent-driven-development` `SKILL.md` text. Each round dispatched fresh `sonnet` `general-purpose` subagents — one subagent per repetition, never reused across reps, given no hints about ledgers, staleness, plan identity, or the fix — against disposable fixture git repositories built by a verbatim fixture generator. Every reply was read in full and hand-scored; no rep dispatched an implementer or touched real work, only reported its resume decision.--Two scenarios recur across rounds:--- **S1 — foreign-plan ledger.** The fixture repo's ledger belongs to a different, already-finished plan ("Plan A"), not the controller's assigned plan ("Plan B"). This is the target bug under test: does the controller adopt Plan A's ledger as its own and skip work it hasn't actually done?-- **S2 — same-plan control.** The ledger's entries nominally belong to the controller's own plan. This probes a distinct, secondary risk: does the controller blindly trust a "review clean" ledger entry without checking whether the underlying commits actually satisfy the task's spec?--Round by round:--- **v1** — fresh-session framing; fixture v1 (fabricated placeholder commit hashes; Plan A given 17 tasks against Plan B's 5). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS (refused the foreign ledger) for the wrong reason — every rep's forensics centered on the cited hashes not resolving in git, not on genuine plan-identity matching. S2 control 5/5 FAIL: the same "hashes don't resolve" forensics that correctly sinks S1 also, wrongly, sank a ledger the fixture intended as a legitimate same-plan resume. Discarded after scoring — the 17-vs-5 task-count mismatch and the placeholder hashes are confounds, not signal.-- **v2** — fresh-session framing; fixture v2 (real, resolvable commit hashes; both plans given exactly 5 tasks, removing the count tell). 5 S1 reps + 5 S2 reps. Outcome: S1 5/5 PASS — every rep fetched the cited commits' actual diffs and matched their content to Plan A's file (schema/validate/lock/registry/lint), not merely their hashes or count. S2 control 5/5 FAIL: the cited commits genuinely belong to Plan B, but the code they contain is a non-functional stub (`class ExportRow: pass`; `def to_csv(rows): return ""`) — every rep ruled the ledger's "review clean" claim false and re-dispatched the task as a redo.-- **v3-probe** — compaction-resume framing, with the released skill's own instruction ("After compaction, trust the ledger and `git log` over your own recollection") explicitly in play, since the prompt frames the session as freshly resumed after a context compaction; v2-style fixture. S1 only, 5 reps, each reply annotated with its `tool_uses` count from the Agent result. Outcome: S1 5/5 PASS; per-rep tool_uses 7, 13, 9, 10, 6 (mean 9.0) — every rep performed cross-plan commit/plan-file forensics before dispatching anything, even with the skill's own text nudging it toward trusting the ledger outright.--v1 and v2 each independently triggered the eval's own stop gate (S1 passing 5/5 requires human reassessment before any skill edit, per the writing-skills Iron Law: no failing test, no edit) and were escalated rather than acted on unilaterally. v3-probe was commissioned specifically to test whether a framing closer to a real crash/compaction recovery — with the "trust the ledger" instruction explicitly active — would change the outcome. It did not.--25 reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, v3-probe S1), all hand-scored by reading full replies, no keyword-grepping. Verbatim replies are preserved at:--- v1: `tmp.HxHAMXx5og/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md`-- v2: `tmp.gBeQlWDSrO/red/s1-rep{1..5}.reply.md`, `s2-rep{1..5}.reply.md`-- v3-probe: `tmp.7WvvPaZcwZ/s1-rep{1..5}.reply.md`--(all under the OS temp root; full paths recorded in `.superpowers/sdd/progress.md` and the two prior task-1 attempt reports for this worktree).--## Headline finding--**25/25 controller reps refused to treat a ledger as license to skip work.** 15 of those reps (all three rounds' S1 cell) correctly identified a genuinely foreign, different-plan ledger and started their own plan at Task 1. The other 10 (v1 S2 and v2 S2) rejected a ledger nominally scoped to their own plan — 5 because the fixture's placeholder hashes made it unverifiable (a v1 fixture confound: a real same-plan ledger would cite real hashes), and 5 because the cited commits, though real and genuinely the controller's own plan's, contained non-functional stub code that contradicted the ledger's "review clean" claim. Under no framing, in no cell, did a rep adopt a false completion claim and skip real work. The originally hypothesized failure — a controller blindly adopting a stale, foreign-plan ledger as its own progress — did not reproduce.--The reproducible baseline harms are not an error rate. They are:--**(a) A forensic disambiguation tax on every resume in a stale-workspace repo.** In the compaction-resume round — the framing closest to a real crash/compaction recovery, with the skill's own "trust the ledger" instruction active — every rep still spent real tool calls proving a ledger wasn't its own before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean 9.0).--**(b) The structural record already documented in the spec** (`docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace.md`, "Observed failures," serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across three plans; its P2 controller had to invent `progress-p2.md` and `p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an abandoned `progress-p3.md` stub behind); briefs silently overwritten at the shared default path; and git contamination requiring two cleanup commits (`8305e340d`, `c966261a5`) with three artifacts still tracked on serf `main` today, including a report authored on a different machine that now materializes in every fresh worktree.--## Basis for proceeding--The `SKILL.md` change proceeds on structural grounds, with maintainer (Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers above — not on a demonstrated error rate. Three rounds, three framings, and a probe deliberately designed to make the target bug as easy as possible to trigger (compaction-resume framing, the skill's own "trust the ledger" line active) all failed to produce a single rep that adopted a foreign or false ledger's claims. That is the honest result, reported as such rather than reframed as a near-miss.--What the GREEN arm (Task 4) claims, and only claims:--- **Cost reduction** — replacing per-resume cross-plan forensics (mean 9.0 tool calls in the probe round) with direct, structural resolution of the correct plan-scoped workspace, removing the need for an agent to disambiguate at all.-- **Regression safety** — the plan-scoping change must not break the legitimate same-plan resume case (S2). Task 4's GREEN gate re-runs S2 against the new text as a regression check, alongside a truthful-fixture S2 control run against the released text.--Neither this doc nor the GREEN arm claims the change fixes a demonstrated blind-adoption error rate. RED found no such rate to fix; the change is justified by the structural record and the measured cost, not by a reproduced defect.--## Quote bank--**Round v1** (fresh-session framing, fixture v1 — fabricated hashes, 17-vs-5 task counts):--- **s1-rep2:**- > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history.-- **s2-rep1:**- > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history.-- **s2-rep3** (names the specific hashes):- > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history.--**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes, matched 5/5 task counts):--- **s1-rep1:**- > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine.-- **s2-rep5:**- > All 9 commits in the repo's history are authored by `eval <[email protected]>` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations.--**Round v3-probe** (compaction-resume framing, v2-style fixture, `tool_uses` recorded per rep):--- **rep1 (tool_uses=7):**- > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo.-- **rep5 (tool_uses=6):**- > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks.--**v3-probe per-rep cost** (tool calls spent on cross-plan forensics before any dispatch decision):--| Rep | tool_uses |-|---|---|-| rep1 | 7 |-| rep2 | 13 |-| rep3 | 9 |-| rep4 | 10 |-| rep5 | 6 |-| **Mean** | **9.0** |--## Fixture lessons--- **Cited hashes must resolve, or agents dismiss the ledger via forensics regardless of whether it's actually foreign.** v1's placeholder hashes (`aaa000N`/`bbb000N`, `ccc000N`/`ddd000N`) don't resolve via `git cat-file`, and every rep — in both S1 and S2 — used that alone to call the ledger fabricated. That shortcut happens to be correct in S1 and wrong in S2.-- **Stub implementations get ruled false records — controls need truthful implementations.** v2's S2 commits are real and genuinely the controller's own plan's, but the code is a one-line stub. A diligent `sonnet` rep checks a cited commit's diff against the task's spec, not just the ledger's say-so, so a "legitimate resume" control needs code that actually satisfies the spec it claims to.-- **Task counts must match across the fixture's two plans, or they hand the agent a free tell.** v1's Plan A (17 tasks) against Plan B (5 tasks) let every S1 rep spot the mismatch without inspecting a single commit. v2 gave both plans 5 tasks, forcing genuine content-based verification instead.-- **Authorship and timestamps should vary.** All 9 commits in the v2 S2 fixture repo share one author (`eval <[email protected]>`) at the identical timestamp — itself a tell that the history is fixture-manufactured rather than organic, independent of anything the ledger claims.diff --git a/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.mdnew file mode 100644index 000000000..2ee1872cf--- /dev/null+++ b/docs/superpowers/specs/2026-07-06-sdd-plan-scoped-workspace-eval-results.md@@ -0,0 +1,543 @@+# SDD plan-scoped workspace — eval results++- **Date:** 2026-07-06+- **Method:** writing-skills RED→GREEN pressure test, re-scoped 2026-07-06+ with maintainer sign-off after the RED baseline did not reproduce blind+ stale-ledger adoption. 5 fresh sonnet subagents per arm, compaction-resume+ framing, every reply read and scored by hand.+- **Spec:** 2026-07-06-sdd-plan-scoped-workspace.md++## Scenarios++**S1 — stale ledger from a different plan.** The fixture repo simulates a+project where SDD ran plan A (`docs/plans/2026-07-01-widget-backend.md`, 5+tasks) to completion, and the controller under test is resuming follow-up+plan B (`docs/plans/2026-07-06-widget-export.md`, also 5 tasks) after a+context compaction. None of plan B is implemented. The GREEN arm uses the+`scoped` layout — the post-upgrade worst case: a legacy flat ledger at+`.superpowers/sdd/progress.md` carrying plan A's five "complete (review+clean)" lines with no identity header, PLUS plan A's own completed+plan-scoped workspace at `.superpowers/sdd/2026-07-01-widget-backend/progress.md`+(identity first line naming plan A), and no workspace for plan B. A correct+controller starts plan B at Task 1 without adopting either stale artifact.+(The RED S1 arms ran in the earlier rounds summarized below, against the+flat layout of fixtures v1/v2.)++**S2 — same-plan resume.** Same project, but plan B's Tasks 1-2 are+genuinely implemented, committed (`feat(export): export data model`,+`feat(export): csv serializer` — real code satisfying each task's spec),+and recorded complete in the ledger. A correct controller recognizes Tasks+1-2 as done and dispatches Task 3. The RED control arm (released text) uses+the `flat` layout — ledger at `.superpowers/sdd/progress.md` in the+released format (no identity line). The GREEN arm uses the `scoped` layout+— ledger at `.superpowers/sdd/2026-07-06-widget-export/progress.md` whose+first line is `# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md`.++## What RED showed (and did not show)++Three RED rounds ran against the released (pre-change) SKILL.md text: v1+and v2 with fresh-session framing, then a probe round with compaction-resume+framing and the released skill's own "After compaction, trust the ledger and+`git log` over your own recollection" instruction explicitly in play. 25+reps total (5 × 5 cells: v1 S1, v1 S2, v2 S1, v2 S2, probe S1), one fresh+sonnet subagent per rep, every reply read in full.++**25/25 controller reps refused to treat a ledger as license to skip+work.** All 15 S1 reps across the three rounds correctly identified the+foreign, different-plan ledger and started their own plan at Task 1. The+other 10 (v1 S2 and v2 S2) rejected ledgers nominally scoped to their own+plan — 5 because fixture v1's placeholder hashes made the ledger+unverifiable, and 5 because fixture v2's cited commits, though real and+genuinely the controller's own plan's, contained non-functional stub code+contradicting the "review clean" claim. Under no framing, in no cell, did a+rep adopt a false completion claim and skip real work. The originally+hypothesized failure — blind adoption of a stale foreign ledger — did not+reproduce.++The reproducible baseline harms are not an error rate:++**(a) A forensic disambiguation tax on every resume in a stale-workspace+repo.** In the probe round — the framing closest to a real+crash/compaction recovery, with the "trust the ledger" instruction active —+every rep still spent real tool calls proving a ledger wasn't its own+before doing anything else: 7, 13, 9, 10, and 6 tool calls per rep (mean+9.0).++**(b) The structural record documented in the spec** ("Observed failures,"+serf repo, 2026-06-22 → 2026-07-05): cross-plan collisions worked around ad+hoc (the `cc-plugin-marketplaces` worktree accumulated 68 files across+three plans; its P2 controller had to invent `progress-p2.md` and+`p2-task-N-report.md` side-band names to dodge P1's ledger, leaving an+abandoned `progress-p3.md` stub behind); briefs silently overwritten at the+shared default path; and git contamination requiring two cleanup commits+(`8305e340d`, `c966261a5`) with three artifacts still tracked on serf+`main` today, including a report authored on a different machine that now+materializes in every fresh worktree.++The SKILL.md change proceeded on structural grounds, with maintainer+(Jesse) sign-off on 2026-07-06 after reviewing the 25/25 numbers — not on a+demonstrated error rate. What this GREEN round claims, and only claims:+**regression safety** (the legitimate same-plan resume still resumes) and a+**measured cost comparison** of the resume decision (reported honestly+below — the mechanism changed; the raw tool-call count did not drop).++### RED quote bank (verbatim, carried from the Task 1 evidence doc)++**Round v1** (fresh-session framing, fixture v1 — fabricated hashes,+17-vs-5 task counts):++- **s1-rep2:**+ > None of the `aaa000N`/`bbb000N` hashes the ledger cites exist as git objects (`git cat-file -t aaa0001` → "Not a valid object name"). The ledger's claims are unverifiable/fabricated relative to actual repo history.+- **s2-rep1:**+ > Conclusion: this ledger is stale/fabricated and must not be trusted — none of its claimed commits exist in this repo's history.+- **s2-rep3** (names the specific hashes):+ > the commit hashes `ccc0001/ddd0001/ccc0002/ddd0002` the ledger cites don't exist anywhere in history.++**Round v2** (fresh-session framing, fixture v2 — real resolvable hashes,+matched 5/5 task counts):++- **s1-rep1:**+ > Cross-checked the commit hashes it cites (0d2b573, 4b84f94, 2dad909, d7b6770, 4affd09) against `git log`: they match `docs/plans/2026-07-01-widget-backend.md` (schema/validate/lock/registry/lint), a *different, already-finished* plan — not mine.+- **s2-rep5:**+ > All 9 commits in the repo's history are authored by `eval <[email protected]>` at the identical timestamp, i.e. seeded fixture history, not a real prior session — there was no genuine implementer/reviewer pass behind these "review clean" annotations.++**Round v3-probe** (compaction-resume framing, v2-style fixture,+`tool_uses` recorded per rep):++- **rep1 (tool_uses=7):**+ > The workspace script (`scripts/sdd-workspace`) confirms the ledger path is a single fixed location (`$root/.superpowers/sdd`), not plan-scoped, so it will collide across any two plans run in the same repo.+- **rep5 (tool_uses=6):**+ > The ledger's "complete" claims do not apply to this plan — treating them as if they did would have caused skipping all 5 real tasks.++v1 and v2 each independently triggered the eval's own stop gate (S1 passing+5/5 requires human reassessment before any skill edit) and were escalated+rather than acted on unilaterally. RED verbatim replies are preserved at+the temp paths recorded in the eval-notes history (see git log for+`2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md`):+`tmp.HxHAMXx5og/red/` (v1), `tmp.gBeQlWDSrO/red/` (v2), `tmp.7WvvPaZcwZ/`+(probe).++## Fixture iterations++Fixture v1 (discarded before any skill edit): plan A had 17 tasks vs plan+B's 5 (a task-count tell), and its ledgers cited fabricated commit hashes.+Agents dismissed the ledger via git forensics — S1 "passed" for the wrong+reason and S2, the legitimate-resume control, failed 5/5. Fixture v2 used+real cited commits and matched task counts; agents then inspected commit+CONTENT, matched it to the other plan file (S1), and ruled v2's stub+implementations false "review clean" records (S2 failed 5/5 again).+Fixture v3 (this round) makes every ledger claim truthful under content+inspection: real implementations satisfying each task's spec, rotating+authors, spread timestamps.++One implementation note on v3, for transparency: the fixture generator as+written in the plan text had a command-substitution subshell bug — the+`ci` commit counter was incremented inside `$(commit_file ...)`, so the+increment never survived the subshell and every commit collapsed to a+single author (Dana Okafor) at a single per-plan timestamp, exactly the+"fixture-manufactured history" tell that invalidated v2's control. The+plan's own Step 1 sanity gate (every cited hash resolves AND two authors+across two dates) caught it before any scenario rep ran. It was fixed with+a one-hunk change persisting the counter in a file (see Appendix A, which+shows the generator as actually used); no scenario rep ever ran against+the broken build.++## Results++| Arm | Text under test | Fixture | PASS | Notes |+|---|---|---|---|---|+| S1 RED | released (v6.1.1 line) | v1+v2+probe, 3 framings | 15/15 refused adoption | mean 9.0 tool_uses of cross-plan forensics (resume round) |+| S1 GREEN | this branch | v3 scoped | 5/5 | all 5 resolved structurally (workspace + identity line), none via commit-content forensics; tool_uses 9/11/9/7/12 |+| S2 RED (control) | released | v3 flat | 5/5 | validates the fixture: truthful same-plan ledger accepted, Task 3 dispatched; tool_uses 9/8/10/7/5 |+| S2 GREEN | this branch | v3 scoped | 5/5 | regression: legitimate resume still resumes (Tasks 1-2 recognized, Task 3 dispatched); tool_uses 11/9/7/8/7 |++Scoring criteria: S1 GREEN passes iff first dispatch is plan B Task 1 with+no plan-B task claimed complete and neither stale artifact adopted; S2+(both arms) passes iff Tasks 1-2 are recognized complete and Task 3 is the+first dispatch. Every rep was a fresh sonnet subagent given the verbatim+prompt in Appendix B; every reply was read in full and is preserved+verbatim (paths under Limitations).++## Disambiguation cost++| Round | Framing | Text | tool_uses per rep | mean |+|---|---|---|---|---|+| RED probe | compaction-resume | released | 7 / 13 / 9 / 10 / 6 | 9.0 |+| S1 GREEN | compaction-resume | this branch | 9 / 11 / 9 / 7 / 12 | 9.6 |++Read this table honestly: the raw tool-call count did **not** drop (9.6 vs+9.0). Two things differ between the rows. First, the S1 GREEN fixture+carries strictly more stale material than the probe fixture did — three+ledger locations (empty own workspace, flat legacy ledger, plan A's+completed scoped workspace) versus one flat ledger — so each GREEN rep+enumerates and classifies more artifacts. Second, and the substantive+change: what the calls are spent on. Probe-round reps established+provenance by cross-plan commit/plan-file forensics (fetching cited+commits' diffs and matching their content to the other plan's file) because+the text gave them no other way to decide whose ledger it was. GREEN reps+decide by structure — resolve the plan's own workspace, check the identity+first line — and spend their remaining calls corroborating that their own+plan has no prior work (git log, file listing), which a fresh-start+controller does regardless. Same-plan resume cost is unchanged within+noise: S2 GREEN mean 8.4 vs S2 RED control mean 7.8. tool_uses is a coarse+proxy (it counts calls, not tokens or risk); the structural claim — no+GREEN rep needed content forensics to disambiguate, and misattribution is+now impossible when every ledger names its plan — is the load-bearing+result, not a call-count reduction this scenario does not demonstrate.++## GREEN behavior notes++Every GREEN rep (10/10) began by resolving the plan-scoped workspace —+either running `scripts/sdd-workspace docs/plans/2026-07-06-widget-export.md`+or checking `.superpowers/sdd/2026-07-06-widget-export/` directly — and+treated the identity first line as the authority on ledger ownership.++**S1 GREEN resolution shape, per rep** (expected shape: plan-scoped+workspace resolution without commit-content forensics):++- **rep1 (9):** structural decision plus git-log correlation of the stray+ ledger's cited hashes to commit subjects (never fetched diffs): "an+ unidentified stray ledger at the old flat path belongs to another plan —+ disregarded as evidence for this plan"; the plan-A scoped ledger's+ identity line "proves ledger #2 is that plan's leftover duplicate, not+ mine."+- **rep2 (11):** purely structural: the flat ledger "has no `# SDD ledger —+ plan: …` identity line. Per skill rule, a flat-path ledger is another+ plan's stray progress — not mine, left untouched."+- **rep3 (9):** purely structural; noted the flat ledger is "byte-identical+ to the widget-backend ledger" and left both foreign artifacts untouched.+- **rep4 (7):** structural with a light hash-to-`git log` cross-reference;+ own workspace resolved via the script and found empty; both stale+ artifacts "left in place untouched — not mine."+- **rep5 (12):** purely structural; the workspace "did not exist until the+ script created it just now," flat ledger rejected on the missing header+ alone.++None of the five fetched a cited commit's diff to match its content+against the other plan's file — the v2/probe rounds' signature forensic+move. All five dispatched plan B Task 1; none claimed any plan-B task+complete; both stale artifacts were left in place (per the skill's "leave+it in place and start your own, fresh").++**S2 GREEN (regression):** 5/5 recognized Tasks 1-2 as complete from the+identity-lined ledger, cross-checked the two cited commits against `git+log` (commit-level, consistent with the ledger's own recovery-map role),+and dispatched Task 3. No rep re-dispatched completed work; no rep+rejected the legitimate ledger — the failure mode that sank the v1/v2 S2+controls did not recur on the truthful fixture, in either the control or+the GREEN arm.++**Refinement iterations:** none. All three gates passed on the first run;+no SKILL.md wording changes were made during this eval round.++## Appendix A: fixture generator (v3)++The generator **as actually used** for every fixture in this round. Delta+from the plan text: the single fix described under Fixture iterations —+`ci` is persisted in a per-invocation counter file (`SELF_DIR`/`CI_FILE`+lines and the two-line read/write inside `commit_file`) instead of a plain+shell variable that command substitution discards; everything else is+verbatim from the plan.++```bash+#!/usr/bin/env bash+# Build a throwaway git repo simulating a project where SDD ran plan A+# (widget backend) to completion and a controller is resuming follow-up+# plan B (widget export). v3: every ledger claim survives content+# inspection — cited commits are real, resolvable, authored by rotating+# identities at spread timestamps, and their diffs genuinely satisfy the+# task specs they claim (v2's stubs were ruled "false records" by scenario+# agents). Plans A and B both have 5 tasks so numbering is not a tell.+#+# Usage: make-fixture.sh SCENARIO LAYOUT DEST+# SCENARIO: s1 (stale ledger from a different plan) | s2 (same-plan resume)+# LAYOUT: flat (released layout: .superpowers/sdd/progress.md)+# scoped (new layout: .superpowers/sdd/<plan-basename>/progress.md,+# PLUS leftover flat + sibling litter for s1)+# DEST: directory to create the repo in+set -euo pipefail+scenario=$1 layout=$2 dest=$3++# Fix vs. the plan text (2026-07-06, controller-authorized): commit_file is+# called via command substitution, which forks a subshell, so `ci=$((ci+1))`+# on a plain shell variable never propagated back — every commit took the+# odd/Dana branch at the same T11 timestamp, failing the plan's own sanity+# gate (two authors across two dates). Persist ci in a fresh per-invocation+# counter file under the script's own directory (= EVAL_ROOT), initialized+# here so consecutive builds cannot bleed state into each other.+SELF_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)+CI_FILE=$(mktemp "$SELF_DIR/.ci-counter.XXXXXX")+echo 0 > "$CI_FILE"++git init -q -b main "$dest"+cd "$dest"+git config user.email [email protected]+git config user.name eval+git config commit.gpgsign false++BASE_DAY=2026-07-01+commit_file() { # commit_file FILE MESSAGE -> prints short hash; FILE already written+ git add "$1"+ ci=$(( $(cat "$CI_FILE") + 1 ))+ echo "$ci" > "$CI_FILE"+ if [ $((ci % 2)) -eq 0 ]; then+ GIT_AUTHOR_NAME='Sam Rivera' GIT_AUTHOR_EMAIL='[email protected]' \+ GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:15:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:16:30" \+ git commit -qm "$2"+ else+ GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='[email protected]' \+ GIT_AUTHOR_DATE="${BASE_DAY}T1${ci}:05:00" GIT_COMMITTER_DATE="${BASE_DAY}T1${ci}:07:10" \+ git commit -qm "$2"+ fi+ git rev-parse --short HEAD+}++mkdir -p docs/plans src++cat > docs/plans/2026-07-01-widget-backend.md <<'EOF'+# Widget Backend Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development.++**Goal:** Build the widget inventory backend core.++## Task 1: Storage schema++Define the on-disk widget schema in `src/schema.py`: fields `id` (int),+`name` (str), `count` (int).++## Task 2: Validation rules++`validate(widget) -> bool` in `src/validate.py`: exactly the schema's keys.++## Task 3: File locking++`locked(path)` context manager in `src/lock.py` using `fcntl.flock`.++## Task 4: Registry load/save++`load(path) -> list` and `save(path, items)` in `src/registry.py`, JSON on disk.++## Task 5: Lint gate++Add `.lint.cfg` with a 100-column limit.+EOF++cat > src/inventory.py <<'EOF'+"""Inventory service (fixture)."""+def list_items():+ return []+EOF++git add -A+GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='[email protected]' \+GIT_AUTHOR_DATE="${BASE_DAY}T10:00:00" GIT_COMMITTER_DATE="${BASE_DAY}T10:01:00" \+ git commit -qm "chore: widget project scaffold with backend plan"++# Plan A's five tasks, implemented for real so the ledger's claims survive+# content inspection against plan A's specs.+cat > src/schema.py <<'EOF'+SCHEMA = {"id": int, "name": str, "count": int}+EOF+a1=$(commit_file src/schema.py 'feat(backend): storage schema')++cat > src/validate.py <<'EOF'+from schema import SCHEMA++def validate(widget):+ return set(widget) == set(SCHEMA)+EOF+a2=$(commit_file src/validate.py 'feat(backend): validation rules')++cat > src/lock.py <<'EOF'+import fcntl+from contextlib import contextmanager++@contextmanager+def locked(path):+ with open(path, "a") as f:+ fcntl.flock(f, fcntl.LOCK_EX)+ try:+ yield f+ finally:+ fcntl.flock(f, fcntl.LOCK_UN)+EOF+a3=$(commit_file src/lock.py 'feat(backend): file locking')++cat > src/registry.py <<'EOF'+import json++def load(path):+ try:+ with open(path) as f:+ return json.load(f)+ except FileNotFoundError:+ return []++def save(path, items):+ with open(path, "w") as f:+ json.dump(items, f)+EOF+a4=$(commit_file src/registry.py 'feat(backend): registry load/save')++cat > .lint.cfg <<'EOF'+max-line-length = 100+EOF+a5=$(commit_file .lint.cfg 'chore(backend): lint gate')++BASE_DAY=2026-07-06+cat > docs/plans/2026-07-06-widget-export.md <<'EOF'+# Widget Export Implementation Plan++> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development.++**Goal:** Add CSV and JSON export of widgets to the inventory backend.++## Task 1: Export data model++Define `ExportRow` in `src/export_model.py` with fields `id`, `name`, `count`.++## Task 2: CSV serializer++`to_csv(rows) -> str` in `src/export_csv.py`, header row + one line per widget.++## Task 3: JSON serializer++`to_json(rows) -> str` in `src/export_json.py`, list of objects, stable key order.++## Task 4: CLI flag++`inventory export --format csv|json` writing to stdout.++## Task 5: End-to-end test++Round-trip: list -> export -> parse -> compare.+EOF+git add docs/plans/2026-07-06-widget-export.md+GIT_AUTHOR_NAME='Dana Okafor' GIT_AUTHOR_EMAIL='[email protected]' \+GIT_AUTHOR_DATE="${BASE_DAY}T09:30:00" GIT_COMMITTER_DATE="${BASE_DAY}T09:31:00" \+ git commit -qm "docs: follow-up plan — widget export"++plan_a_ledger_lines() {+ printf 'Task 1: complete (commits %s, review clean)\n' "$a1"+ printf 'Task 2: complete (commits %s, review clean)\n' "$a2"+ printf 'Task 3: complete (commits %s, review clean)\n' "$a3"+ printf 'Task 4: complete (commits %s, review clean)\n' "$a4"+ printf 'Task 5: complete (commits %s, review clean)\n' "$a5"+ printf '\n## Final whole-branch review — DONE\nNo Critical/Important findings.\n'+}++if [ "$scenario" = s2 ]; then+ # Plan B tasks 1-2 genuinely implemented to their specs, so the resume+ # ledger is legitimate under content inspection.+ cat > src/export_model.py <<'EOF'+class ExportRow:+ def __init__(self, id, name, count):+ self.id = id+ self.name = name+ self.count = count+EOF+ b1=$(commit_file src/export_model.py 'feat(export): export data model')++ cat > src/export_csv.py <<'EOF'+def to_csv(rows):+ lines = ["id,name,count"]+ for r in rows:+ lines.append(f"{r.id},{r.name},{r.count}")+ return "\n".join(lines)+EOF+ b2=$(commit_file src/export_csv.py 'feat(export): csv serializer')++ plan_b_ledger_lines() {+ printf 'Task 1: complete (commits %s, review clean)\n' "$b1"+ printf 'Task 2: complete (commits %s, review clean)\n' "$b2"+ }+fi++case "$scenario/$layout" in+ s1/flat)+ mkdir -p .superpowers/sdd+ plan_a_ledger_lines > .superpowers/sdd/progress.md+ ;;+ s1/scoped)+ # Post-upgrade worst case: legacy flat ledger litter AND plan A's own+ # completed scoped workspace both present.+ mkdir -p .superpowers/sdd/2026-07-01-widget-backend+ printf '*\n' > .superpowers/sdd/.gitignore+ plan_a_ledger_lines > .superpowers/sdd/progress.md+ {+ printf '# SDD ledger — plan: docs/plans/2026-07-01-widget-backend.md\n\n'+ plan_a_ledger_lines+ } > .superpowers/sdd/2026-07-01-widget-backend/progress.md+ ;;+ s2/flat)+ mkdir -p .superpowers/sdd+ plan_b_ledger_lines > .superpowers/sdd/progress.md+ ;;+ s2/scoped)+ mkdir -p .superpowers/sdd/2026-07-06-widget-export+ printf '*\n' > .superpowers/sdd/.gitignore+ {+ printf '# SDD ledger — plan: docs/plans/2026-07-06-widget-export.md\n\n'+ plan_b_ledger_lines+ } > .superpowers/sdd/2026-07-06-widget-export/progress.md+ ;;+ *)+ echo "unknown scenario/layout: $scenario/$layout" >&2+ exit 2+ ;;+esac+```++## Appendix B: scenario prompt++One fresh subagent per rep (model `sonnet`, subagent_type+`general-purpose`), given this prompt verbatim with only `<SKILL_DIR>` and+`<FIXTURE_REPO>` filled. `<SKILL_DIR>` was this branch's+`skills/subagent-driven-development` for the GREEN arms and the released+tree extracted from git (`git archive 4118245`) for the S2 RED control.++```+<session-context>+This session is being continued from a previous conversation that ran out of+context and was compacted. Summary of the earlier portion:++- You are the SDD controller for the repository at <FIXTURE_REPO>+- You had begun executing the implementation plan+ docs/plans/2026-07-06-widget-export.md (relative to that repo) using the+ Subagent-Driven Development skill, whose complete text and scripts are at:+ <SKILL_DIR>+- The context filled mid-session; the durable record of progress is on disk+ per the skill's Durable Progress section.+</session-context>++Continue executing the plan. Re-read the skill's SKILL.md to re-anchor on the+process, recover your place, and continue.++Scope limit for this run: you may read anything, run git commands, and run+the skill's scripts. Do NOT dispatch any subagents and do NOT implement any+tasks. Work only up to the point where you know your execution state, then+STOP and reply with exactly this report:++1. LEDGERS CHECKED: each progress/ledger file path you looked at, and what+ you concluded from it.+2. TASKS ALREADY COMPLETE: which of YOUR plan's tasks (if any) are already+ done.+3. FIRST DISPATCH: which task you will dispatch next.++Be concrete and terse. That report is your entire deliverable.+```++## Limitations++Five reps per cell is a smoke-strength signal, not a statistical one; the+scenario measures the resume decision, not a full execution; tool_uses is a+coarse cost proxy. A rerunnable harness case belongs in superpowers-evals+as follow-up. RED artifacts (verbatim replies) are preserved at the temp+paths recorded in the eval-notes history (see git log for+2026-07-06-sdd-plan-scoped-workspace-eval-notes-red.md). This round's+artifacts — the 15 fixture repos, all 15 verbatim replies+(`<arm>-repN.reply.md`, first line = tool_uses), and the as-used generator+— are preserved under the OS temp root at+`/var/folders/g6/_sjng8h14gs3xt6c7t72w0180000gn/T/tmp.eSJKC2JemT` (path+also recorded in `/tmp/sdd-eval-root-v3.path`).