Worktrunk is a CLI for Git worktree management, designed for parallel AI agent workflows
Verify changelog claims against their own sources, and gate on a clean run (#4051)
Maximilian Roos4d88a39a168c
diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.mdindex 77e9b700b..3c0da26ab 100644--- a/.claude/skills/release/SKILL.md+++ b/.claude/skills/release/SKILL.md@@ -227,14 +227,6 @@ Link when there's substantial documentation the user would benefit from reading**The gate cuts both ways.** Checking only accuracy pushes every entry longer: "understates" and "not covered" have no counterweight, so each pass adds and none subtracts. That asymmetry is what drove the ratchet above. An entry that is too long, too internal, or ranked above one more readers will notice is reported on the same footing as one that is wrong.-The subagent should:-1. Take the list of drafted changelog entries-2. For each entry, find the commit(s) it describes and read the actual diff-3. Verify the entry accurately describes what changed-4. Check for missing changes that should be documented-5. Check each entry against the length ceiling and the ordering rule-6. Report inaccuracies, omissions, overlong entries, and misordering-**Subagent prompt template:**```@@ -243,15 +235,27 @@ Verify these changelog entries for version X.Y.Z are accurate.Previous version: [e.g., v0.1.9]Commits to check: git log v<previous>..HEAD-Entries to verify:-[paste drafted entries]+Entries to verify: the top section of CHANGELOG.md as it stands on disk. Read it+there rather than from a paste:+awk '/^## /{if (f) exit; f=1} f' CHANGELOG.md++Verify claim by claim, not entry by entry: an entry carries several independent+claims, and one verdict over the whole entry waves through every claim that is not+its headline.-For EACH entry:1. Find the relevant commit(s) using git log and git show-2. Read the actual diff, not just the commit message-3. Confirm the entry accurately describes the user-facing change-4. Flag if the entry overstates, understates, or misdescribes the change-5. Flag if the entry runs over 60 words (80 for one of the two or three headline+2. Read the diff, not the commit message. The diff settles what changed. It does+ not settle what the behavior was before, what the user sees, or what a file it+ doesn't touch does. Settle a "previously" / "no longer" / "so X broke" claim by+ reading the old file (`git show <sha>^:<path>`) and confirming the old behavior+ there. The new code's handling of the old case is not that confirmation: a case+ added together with a comment about why it produces nothing reads in a diff+ exactly like a case that used to produce something. Settle a claim about output+ or a version floor against the rendered output and the docs, and a claim about+ another component against that component's own file+3. Flag any claim its source does not support, whether it overstates,+ understates, or misdescribes+4. Flag if the entry runs over 60 words (80 for one of the two or three headlineentries), restates the PR description, or explains mechanism the reader cannotact on — report these as seriously as an inaccuracy, and quote a shorterrewrite that keeps every user-facing claim@@ -265,11 +269,13 @@ Report format:- Entry: [entry text]Status: ✅ Accurate / ⚠️ Needs revision / ❌ IncorrectLength: [word count] — ✅ / ⚠️ over ceiling- Evidence: [what you found in the diff]+ Evidence: [for each claim, the source you read and what it said]Suggested fix: [if needed]```-**Do not finalize the changelog until the subagent confirms every entry is accurate and within the ceiling.**+**The pass ends on a clean run, not on the first run's findings.** A rewrite the verifier suggests is a new draft with no more evidence behind it than one you wrote yourself, so applying it leaves that entry unverified again. An entry you edit while the pass runs is in the same state. Re-run the verifier over the section as it now stands, and finalize only once a run comes back clean.++`evals/README.md` beside this skill holds four entries from a shipped release, three of them wrong, for scoring a change to this template against what the last wording missed.**If verification finds problems:** Escalate to the user. Show them the subagent's findings and ask how to proceed. Don't attempt to resolve ambiguous changelog entries autonomously — the user knows the intent behind their changes better than you do.diff --git a/.claude/skills/release/evals/README.md b/.claude/skills/release/evals/README.mdnew file mode 100644index 000000000..bf524437f--- /dev/null+++ b/.claude/skills/release/evals/README.md@@ -0,0 +1,37 @@+# Changelog-verification cases++Four entries from a shipped release, three of which reached users wrong. They+score the "MANDATORY: Verify Each Changelog Entry" prompt template: run it as a+subagent prompt and check which of the three it reports.++The template reads the top section of `CHANGELOG.md` on disk, so run the arm in a+checkout at `v0.77.0`, where that section is the 29 entries these cases come+from. Run the template unedited: pointing it at the section some other way scores+a prompt nobody ships, and the section it would otherwise read is whatever+release is in progress.++```bash+git worktree add /tmp/wt-eval-0.77.0 v0.77.0 # remove it afterwards+```++That checkout carries this skill at `v0.77.0` too, whose template is the one+being replaced. Pass the arm's template as the prompt and withhold the `Skill`+tool, so the run cannot load the old one off disk instead.++Offer the run the notes from the per-group agents as a map, the way a release+does. Handed only these four entries every wording found all three, so what the+cases measure is attention spread across 29 entries, a missing-entries sweep,+and the attribution checks.++None of them has its truthmaker wholly outside the commit, and that is hard to+fix here: this repo's commits routinely restate the surrounding facts in added+comments and doc lines, so the falsifying text usually arrives in the diff. Read+the rows as scoring how much of the diff the verifier reads, not whether it goes+looking beyond one.++| Case | Entry | The claim | What the source says |+|------|-------|-----------|----------------------|+| Before-state, with a confirming-looking diff | `wt list --format=json` gains a `marker` field | "`state` no longer carries `detached`" | `git show 981a207e2^:src/commands/list/model/state.rs` has no `Detached` variant — the PR adds it. The diff's no-op arm for `Detached` reads like the removal of a case that used to emit something. |+| Fact in the diff, in an added comment rather than the code | `wt config state marker` set and clear no-op | "an agent plugin's hooks failed on every event" | Every shipped marker hook ends `\|\| true` (`git grep 'state marker' v0.76.0 -- plugins hooks`), so the hook exited 0. `29a611b7c` says so in a comment it adds beside the fix, so a verifier that reads the whole diff has the contradiction without leaving it. |+| Non-headline clause, settled in the diff | The FAQ lists the files Worktrunk writes | "for Claude Code, Codex, OpenCode, Pi, and Gemini" | The FAQ's new table has three rows. Gemini appears nowhere; Codex appears only in the line saying its install writes nothing. |+| Guardrail | LLM commit messages keep diffs for quoted paths | all of it | Accurate. A run that calls this entry wrong has gone too far. |