Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Automate mutation testing end-to-end with run! orchestrator (#70126)

Eric Normand3ba0f90cf022
diff --git a/.claude/skills/mutation-testing/SKILL.md b/.claude/skills/mutation-testing/SKILL.md
index 0661ff3c4bd..2f2274ad6aa 100644
--- a/.claude/skills/mutation-testing/SKILL.md
+++ b/.claude/skills/mutation-testing/SKILL.md
@@ -5,227 +5,110 @@ description: Run mutation testing on a Clojure namespace, generate tests to kill
# Mutation Testing Skill
-This skill runs mutation testing on a target namespace, writes tests to kill surviving mutations, and creates draft PRs with Linear issues for each function.
+This skill runs mutation testing end-to-end: generates a coverage report, groups functions, shells out to `claude -p` to write tests, verifies mutations are killed, and creates draft PRs with Linear issues.
## Prerequisites
- A running nREPL connected to the Metabase dev environment
- `LINEAR_API_KEY` environment variable set with a valid Linear personal API key
- `gh` CLI authenticated with GitHub
+- `claude` CLI available on PATH
## Reference Files
-- `dev/src/dev/coverage.clj` — mutation testing tool (generates reports, runs mutations)
-- `dev/src/dev/mutation_testing.clj` — Linear API client and PR template helpers
-- `mutation-testing-report.lib.card.md` — example report from the pilot
+- `dev/src/dev/coverage.clj` — mutation testing engine (generates reports, runs mutations)
+- `dev/src/dev/mutation_testing.clj` — orchestration, Linear API, PR helpers
## Invocation
-The argument is a Clojure namespace, e.g.:
+The argument is a Clojure namespace, optionally followed by a base branch:
```
/mutation-testing metabase.lib.order-by
+/mutation-testing metabase.lib.order-by --base-branch release-x.52.x
+/mutation-testing metabase.lib.order-by --project-id abc-123
+/mutation-testing metabase.lib.order-by --base-branch release-x.52.x --project-id abc-123
```
-## Workflow
+- `--base-branch`: defaults to the value in config (set via `set-config!`), or `"master"` if not configured.
+- `--project-id`: if provided, issues are added to this existing Linear project instead of creating a new one.
-### Step 1: Parse the namespace
+## Steps
-From the argument (e.g., `metabase.lib.order-by`), derive:
-- **Test namespace**: append `-test` (e.g., `metabase.lib.order-by-test`)
-- **Short name**: the last segment (e.g., `order-by`)
-- **Source path**: `src/metabase/lib/order_by.cljc` (replace `-` with `_` in path segments)
-- **Test path**: `test/metabase/lib/order_by_test.cljc`
+### 1. Parse arguments
-### Step 2: Set up Linear and load tools
+Parse `$ARGUMENTS` to extract:
+- **namespace** (required) — the first positional argument
+- **`--base-branch`** (optional) — if present, the next argument is the base branch name
+- **`--project-id`** (optional) — if present, the next argument is an existing Linear project ID
+
+### 2. Load and configure
```clojure
-(require '[dev.coverage :as cov] :reload)
(require '[dev.mutation-testing :as mut-test] :reload)
+(require '[dev.coverage :as cov] :reload)
```
If this is the first invocation (or the REPL was restarted), set up the Linear team:
```clojure
-;; Find the team
(mut-test/list-teams!)
-;; Set team ID from the output
-(mut-test/set-config! {:team-id "<team-id>"})
-```
-
-If config is already set from a previous invocation, skip this.
-
-### Step 3: Generate baseline report and create Linear project
-
-```clojure
-(cov/generate-report
- '<target-ns>
- ['<test-ns>]
- "mutation-testing-report.lib.<short-name>.before.md")
-```
-
-Read the generated report file. It has three sections:
-- **Uncovered Functions** — never called by any test
-- **Partially Covered Functions** — called but with surviving mutations
-- **Fully Covered Functions** — all mutations killed
-
-Then create a Linear project for this namespace (uses report stats in the description):
-
-```clojure
-(mut-test/create-project-for-namespace!
- "<target-ns>"
- "mutation-testing-report.lib.<short-name>.before.md")
-;; => {:project-id "...", :name "Mutation Testing: metabase.lib.order-by"}
-;; Automatically sets :project-id in config for subsequent create-issue! calls
+(mut-test/set-config! {:team-id "<id>"})
```
-### Step 4: Plan the work
-
-Before creating branches, analyze the report to plan how to group the work:
-
-1. **Identify killable functions.** For each function with surviving mutations, assess whether any mutations are killable. Run `cov/test-mutations` to see the full mutation list. If ALL mutations for a function are unkillable (schema-only, semantically equivalent, etc.), skip that function — no branch or PR needed.
-
-2. **Group private functions by public entry point.** Private functions cannot be tested directly. Group them with the public function that exercises them. Create one branch/PR per group, not per private function.
-
-3. **Track mutations per group.** The `:mutations-before` and `:not-killed` fields in the PR should only reflect mutations for functions covered by that specific PR — never mutations from other functions.
-
-### Step 5: Process each group
-
-For each group of functions that has killable mutations:
-
-#### 5a. Create a branch
+### 3. Run
```clojure
-(mut-test/create-branch! "<target-ns>" "<primary-fn-name>")
-;; => "mutation-testing-lib-order-by-orderable-columns"
+(mut-test/run! '<target-ns>)
+;; Or with options:
+(mut-test/run! '<target-ns> {:base-branch "release-x.52.x"})
+(mut-test/run! '<target-ns> {:project-id "abc-123"})
+(mut-test/run! '<target-ns> {:base-branch "release-x.52.x" :project-id "abc-123"})
```
-Use the primary public function name for the branch. If the group covers private functions too, name the branch after the public entry point.
-
-#### 5b. Read context
-
-- Read the source function(s) from the source file
-- Read the full test file to understand existing test patterns, helpers, and metadata providers used
-- If the function is private, identify which public function(s) call it
+Pass the opts map with `:base-branch` and/or `:project-id` if those flags were provided in the arguments.
-#### 5c. Write tests
+This will:
+1. Generate a baseline mutation testing report
+2. Create a Linear project for the namespace
+3. Group functions by coverage relationships
+4. For each group: create branch → invoke Claude to write tests → verify → retry if needed → commit & push → create Linear issue → create draft PR
+5. Print a summary with PR links
-Generate the **simplest tests** that kill the surviving mutations while still making semantic sense. Guidelines:
+### 4. Handle failures
-- **Never call private functions directly.** Always test through public API functions. Use the coverage data to identify which public functions exercise the private function.
-- **Keep tests simple.** Each test should verify one meaningful behavior. Don't over-engineer tests just to chase mutation kills.
-- **Follow existing patterns.** Match the style of the existing tests: same metadata providers, same helper functions, same assertion patterns.
-- **Insert tests near related existing tests** for the same function, not at the end of the file. This minimizes merge conflicts between branches. If there are no existing tests for the function, find the most logical location based on the order of functions in the source file.
-
-**IMPORTANT: Use `file_edit` (not `clojure_edit`) when editing test files.** The `clojure_edit` tool reformats the entire file, introducing unnecessary whitespace changes that pollute the diff. Use `file_edit` with exact string matching to make surgical insertions that only touch the lines you intend to change.
-
-#### 5d. Verify mutations are killed
-
-Use the REPL to check that the new tests kill the targeted mutations:
+If a group fails mid-processing, `run!` catches the error and continues to the next group. To retry a single group manually:
```clojure
-;; Reload the test namespace to pick up new tests
-(require '<test-ns> :reload)
-
-;; Test mutations for the specific function
-(cov/test-mutations
- '<target-ns>/<fn-name>
- ;; Set of test names that cover this function
- #{'<test-ns>/<test-name-1> '<test-ns>/<test-name-2>})
-```
-
-Check the `:survived` key in the result. If mutations still survive:
-- Try writing additional tests
-- If a mutation is truly unkillable (semantically equivalent), note it for the PR description
-- If a mutation reveals dead/unreachable code, consider suggesting removal as a code improvement
-
-#### 5e. Handle unkillable mutations and improvements
+;; Get the parsed namespace info
+(def parsed (mut-test/parse-namespace '<target-ns>))
-**Do NOT edit the source namespace just for documentation.** Instead:
-
-- **Unkillable mutations**: Note them in the PR description with rationale
-- **Code improvements** (dead code removal, expression simplification) that directly relate to surviving mutants:
- 1. Make the change in the source file
- 2. Verify all tests still pass with the change
- 3. Note the file path, line range, and the improved code
- 4. **Reset the change**: `git checkout -- <source-file>` — do NOT commit it
- 5. After creating the PR, use `add-suggested-change!` to post the improvement as a GitHub suggested change (Step 5h)
- 6. The reviewer decides whether to accept it
-
-#### 5f. Commit and push
-
-```clojure
-;; Only commit the test file — never commit source changes for improvements
-(mut-test/commit-and-push! "<target-ns>" "<primary-fn-name>"
- ["test/metabase/lib/<short_name>_test.cljc"])
-```
+;; Run coverage to get the data
+(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
-#### 5g. Create Linear issue and draft PR
+;; Group and find the one you want
+(def groups (mut-test/group-functions coverage-results))
-```clojure
-;; Create a Linear issue (use the primary function name)
-(def issue (mut-test/create-issue-for-function! "<target-ns>" "<primary-fn-name>"))
-;; => {:identifier "QUE-1234", :url "https://linear.app/...", ...}
-
-;; Create draft PR linked to the Linear issue
-;; :fn-names lists ALL functions covered by this PR
-;; :mutations-before counts only mutations for functions in this PR
-;; :not-killed lists only unkillable mutations for functions in this PR
-(def pr-url
- (mut-test/create-draft-pr!
- {:target-ns "<target-ns>"
- :fn-name "<primary-fn-name>"
- :linear-identifier (:identifier issue)
- :mutations-before 9 ;; surviving mutations for these functions only
- :tests-added 2 ;; number of new test functions
- :killed ["Replace :segment-id with :segment-id__"
- "Replace cycle-path with nil"]
- :not-killed [{:description "Replace acc with nil"
- :rationale "reduce always has single iteration"}]
- :suggested-changes []}))
-;; => "https://github.com/metabase/metabase/pull/12345"
+;; Process just that group
+(mut-test/process-group! parsed (nth groups <index>))
```
-#### 5h. Add code improvement suggestions (if any)
-
-If you identified code improvements in step 5e (and reset them), post each as a GitHub suggested change:
-
-```clojure
-(mut-test/add-suggested-change!
- {:pr-url pr-url
- :path "src/metabase/lib/<short_name>.cljc"
- :start-line 42 ;; first line of the code to replace
- :end-line 45 ;; last line of the code to replace
- :suggestion "(improved-code-here)" ;; the replacement code
- :comment "**Suggested improvement:** <description of the change and why it's related to the surviving mutant>"})
-```
+### 5. Verify the prompt (dry run)
-### Step 6: Return to master and repeat
+To inspect what Claude will see without invoking it:
```clojure
-(mut-test/return-to-master!)
+(def parsed (mut-test/parse-namespace '<target-ns>))
+(def coverage-results (cov/test-namespace (:target-ns parsed) [(:test-ns parsed)]))
+(def groups (mut-test/group-functions coverage-results))
+(println (mut-test/build-test-prompt
+ (merge (select-keys parsed [:target-ns :test-ns :source-path :test-path])
+ (select-keys (first groups) [:fn-names :mutations]))))
```
-Repeat Step 5 for the next group.
-
-### Step 7: Summary
-
-Print a summary of what was done:
-- Link to the Linear project for this namespace
-- Number of functions processed
-- Number of functions skipped (all mutations unkillable)
-- Number of draft PRs created
-- Number of mutations killed vs. unkillable
-- List of test PRs with links
-- List of mutation rule PRs with links (if any were created)
-
-## Suggesting New Mutation Rules
-
-If you notice patterns in surviving mutations that suggest the mutation testing tool should have additional rules (e.g., a common Clojure form that isn't being mutated), open a separate PR proposing additions to `dev.coverage/mutation-rules` with an explanation.
-
## Configuration
-The `dev.mutation-testing` namespace reads `LINEAR_API_KEY` from the environment and stores team/project IDs in an atom.
-
-- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})` (Step 2)
-- **Project ID**: Set automatically when `create-project-for-namespace!` is called (Step 3) — one project per namespace
+- **Team ID**: Set once per REPL session via `(mut-test/list-teams!)` and `(mut-test/set-config! {:team-id "..."})`
+- **Project ID**: Set automatically when `run!` calls `create-project-for-namespace!`
+- **Base Branch**: Defaults to `"master"`. Override via `(mut-test/set-config! {:base-branch "release-x.52.x"})` or pass `{:base-branch "..."}` as the second arg to `run!`
+- **Project ID**: If set (via `set-config!` or `run!` opts), `run!` reuses the existing Linear project instead of creating a new one
diff --git a/dev/src/dev/mutation_testing.clj b/dev/src/dev/mutation_testing.clj
index 6a8ba303c39..72d3c51674c 100644
--- a/dev/src/dev/mutation_testing.clj
+++ b/dev/src/dev/mutation_testing.clj
@@ -7,11 +7,15 @@
Configuration:
- LINEAR_API_KEY env var must be set
- Call (set-config! {:team-id \"...\" :project-id \"...\"}) before creating issues"
+ (:refer-clojure :exclude [run!])
(:require
[cheshire.core :as json]
[clj-http.client :as http]
+ [clojure.java.io :as io]
[clojure.java.shell :as shell]
- [clojure.string :as str]))
+ [clojure.set :as set]
+ [clojure.string :as str]
+ [dev.coverage :as coverage]))
(set! *warn-on-reflection* true)
@@ -20,14 +24,21 @@
(defonce ^:private config (atom {}))
(defn set-config!
- "Set configuration for Linear API calls.
+ "Set configuration for mutation testing.
Required keys:
- :team-id — Linear team ID
- - :project-id — Linear project ID (can be set after creating a project)"
+ - :project-id — Linear project ID (can be set after creating a project)
+
+ Optional keys:
+ - :base-branch — git branch to create feature branches from (default: \"master\")
+ - :project-id — Linear project ID to add issues to (skips project creation in run!)"
[m]
(swap! config merge m))
+(defn- base-branch []
+ (get @config :base-branch "master"))
+
(defn- api-key []
(or (System/getenv "LINEAR_API_KEY")
(throw (ex-info "LINEAR_API_KEY environment variable is not set" {}))))
@@ -215,14 +226,14 @@
"Generate a branch name for mutation testing a function.
Example: mutation-testing-lib-order-by-orderable-columns"
[target-ns fn-name]
- (let [short-ns (last (clojure.string/split (str target-ns) #"\."))]
+ (let [short-ns (last (str/split (str target-ns) #"\."))]
(str "mutation-testing-lib-" short-ns "-" fn-name)))
(defn create-branch!
- "Create and checkout a new branch from master for mutation testing a function."
+ "Create and checkout a new branch from the base branch for mutation testing a function."
[target-ns fn-name]
(let [branch (branch-name target-ns fn-name)]
- (sh "git" "checkout" "master")
+ (sh "git" "checkout" (base-branch))
(sh "git" "pull")
(sh "git" "checkout" "-b" branch)
branch))
@@ -255,10 +266,11 @@
(let [title (pr-title target-ns fn-name)
body (pr-description opts)
result (sh "gh" "pr" "create" "--draft"
+ "--base" (base-branch)
"--title" title
"--body" body
"--label" "no-backport")]
- (clojure.string/trim (:out result))))
+ (str/trim (:out result))))
(defn add-pr-comment!
"Add a general comment to a PR. `pr-url-or-number` can be a PR URL or number."
@@ -308,12 +320,17 @@
"-f" "start_side=RIGHT"]))]
(apply sh args)))
-(defn return-to-master!
- "Checkout master branch."
+(defn return-to-base!
+ "Checkout the configured base branch."
[]
- (sh "git" "checkout" "master")
+ (sh "git" "checkout" (base-branch))
nil)
+(defn return-to-master!
+ "Checkout the configured base branch. Deprecated: use return-to-base! instead."
+ []
+ (return-to-base!))
+
;;; --- Report helpers ---
(defn report-stats
@@ -409,6 +426,358 @@
(create-issue! {:title (linear-issue-title target-ns fn-name)
:description (linear-issue-description target-ns fn-name)}))
+;;; --- Orchestration ---
+
+(defn parse-namespace
+ "Parse a namespace symbol into all derived paths and names.
+ Auto-detects .cljc vs .clj by checking which file exists."
+ [target-ns-sym]
+ (let [target-ns (str target-ns-sym)
+ test-ns (str target-ns "-test")
+ short-name (last (str/split target-ns #"\."))
+ ns-path (-> target-ns
+ (str/replace "." "/")
+ (str/replace "-" "_"))
+ test-path-base (-> test-ns
+ (str/replace "." "/")
+ (str/replace "-" "_"))
+ source-ext (cond
+ (.exists (io/file (str "src/" ns-path ".cljc"))) ".cljc"
+ (.exists (io/file (str "src/" ns-path ".clj"))) ".clj"
+ :else ".cljc")
+ test-ext (cond
+ (.exists (io/file (str "test/" test-path-base ".cljc"))) ".cljc"
+ (.exists (io/file (str "test/" test-path-base ".clj"))) ".clj"
+ :else ".cljc")]
+ {:target-ns (symbol target-ns)
+ :test-ns (symbol test-ns)
+ :short-name short-name
+ :source-path (str "src/" ns-path source-ext)
+ :test-path (str "test/" test-path-base test-ext)
+ :report-path (str "mutation-testing-report." target-ns ".before.md")}))
+
+(defn group-functions
+ "Group functions from coverage results for batch processing.
+ Takes the output of coverage/test-namespace.
+
+ Logic:
+ - Public functions with surviving mutations each become a group
+ - Private functions get assigned to the public function with the most test overlap
+ - Functions with zero surviving mutations are skipped
+ - Uncovered or unassignable private functions get their own group"
+ [coverage-results]
+ (let [with-survivors (into {} (filter (fn [[_ v]] (seq (:survived v))) coverage-results))
+ public? (fn [fn-sym]
+ (let [v (find-var fn-sym)]
+ (and v (not (:private (meta v))))))
+ public-fns (into {} (filter (fn [[k _]] (public? k)) with-survivors))
+ private-fns (into {} (remove (fn [[k _]] (public? k)) with-survivors))
+ assignments (into {}
+ (for [[priv-fn priv-data] private-fns
+ :let [priv-tests (set (:tests priv-data))
+ best (when (seq priv-tests)
+ (->> public-fns
+ (map (fn [[pub-fn pub-data]]
+ [pub-fn (count (set/intersection
+ priv-tests
+ (set (:tests pub-data))))]))
+ (filter (fn [[_ n]] (pos? n)))
+ (sort-by second >)
+ first))]
+ :when best]
+ [priv-fn (first best)]))
+ unassigned-private (into {} (remove (fn [[k _]] (contains? assignments k)) private-fns))
+ public-groups (for [[pub-fn pub-data] public-fns
+ :let [attached (keep (fn [[priv-fn assigned-to]]
+ (when (= assigned-to pub-fn) priv-fn))
+ assignments)
+ all-fn-names (vec (cons pub-fn attached))
+ all-mutations (vec (concat (:survived pub-data)
+ (mapcat #(:survived (get coverage-results %))
+ attached)))
+ all-tests (apply set/union
+ (keep #(:tests (get coverage-results %))
+ all-fn-names))]]
+ {:primary-fn pub-fn
+ :fn-names all-fn-names
+ :mutations all-mutations
+ :tests (or all-tests #{})})
+ private-groups (for [[priv-fn priv-data] unassigned-private]
+ {:primary-fn priv-fn
+ :fn-names [priv-fn]
+ :mutations (vec (:survived priv-data))
+ :tests (or (:tests priv-data) #{})})]
+ (vec (concat public-groups private-groups))))
+
+(defn build-test-prompt
+ "Build a prompt for Claude to write tests that kill surviving mutations."
+ [{:keys [target-ns test-ns source-path test-path fn-names mutations]}]
+ (let [fn-sources (str/join "\n\n"
+ (for [fn-name fn-names
+ :let [fn-info (coverage/find-function-source fn-name)
+ source (when fn-info (coverage/read-function-from-file fn-info))]
+ :when source]
+ (str ";;; " fn-name "\n" source)))
+ mutation-list (str/join "\n\n"
+ (map-indexed
+ (fn [i {:keys [description mutation]}]
+ (str "### Mutation " (inc i) ": " description "\n```clojure\n" mutation "\n```"))
+ mutations))
+ test-content (slurp test-path)]
+ (str "You are writing mutation tests for functions in `" target-ns "`.\n"
+ "\n"
+ "## Source Code of Functions Under Test\n"
+ "\n"
+ "```clojure\n"
+ fn-sources
+ "\n```\n"
+ "\n"
+ "## Existing Test File (" test-path ")\n"
+ "\n"
+ "```clojure\n"
+ test-content
+ "\n```\n"
+ "\n"
+ "## Surviving Mutations to Kill\n"
+ "\n"
+ "Each mutation below represents a change to the source code that no existing test catches.\n"
+ "Write the simplest tests that would fail if these mutations were applied.\n"
+ "\n"
+ mutation-list
+ "\n"
+ "\n"
+ "## Instructions\n"
+ "\n"
+ "- Write the simplest tests that kill these surviving mutations\n"
+ "- Follow the patterns in the existing test file (same helpers, same assertion style)\n"
+ "- Never call private functions directly — test through public API\n"
+ "- Insert tests near related existing tests, not at the end of the file\n"
+ "- Use the Edit tool to modify the test file at: " test-path "\n"
+ "- After writing tests, verify they compile by running: clj-nrepl-eval -p $(clj-nrepl-eval --discover-ports | head -1) \"(require '" test-ns " :reload)\"\n"
+ "- Each test should verify one meaningful behavior — don't over-engineer\n")))
+
+(defn invoke-claude!
+ "Shell out to claude CLI to write tests. Returns the output string."
+ [prompt]
+ (let [result (shell/sh "claude" "-p"
+ "--allowedTools" "Edit,Read,Bash(clj-nrepl-eval*)"
+ :in prompt)]
+ (when-not (zero? (:exit result))
+ (throw (ex-info "Claude invocation failed" {:exit (:exit result)
+ :err (:err result)})))
+ (:out result)))
+
+(defn- all-test-names
+ "Get all test var names from a test namespace."
+ [test-ns-sym]
+ (require test-ns-sym :reload)
+ (let [ns-obj (find-ns test-ns-sym)]
+ (->> (ns-interns ns-obj)
+ vals
+ (filter #(:test (meta %)))
+ (map #(symbol (str test-ns-sym) (name (.sym ^clojure.lang.Var %))))
+ set)))
+
+(defn- count-tests
+ "Count test vars in a namespace."
+ [test-ns-sym]
+ (count (filter #(:test (meta %)) (vals (ns-interns (find-ns test-ns-sym))))))
+
+(defn verify-and-retry!
+ "Verify mutations are killed. Retry with Claude if needed.
+ Returns {:status :success|:partial, :killed [...], :survived [...]}."
+ [{:keys [fn-names test-ns test-path max-retries]
+ :or {max-retries 2}}]
+ (loop [retries-left max-retries]
+ (let [test-names (all-test-names test-ns)
+ fn-results (into {}
+ (for [fn-name fn-names
+ :let [result (coverage/test-mutations fn-name test-names)]
+ :when result]
+ [fn-name result]))
+ all-killed (vec (mapcat :killed (vals fn-results)))
+ all-survived (vec (mapcat :survived (vals fn-results)))]
+ (if (empty? all-survived)
+ {:status :success :killed all-killed :survived []}
+ (if (pos? retries-left)
+ (do
+ (println " " (count all-survived) "mutations still survive, retrying..."
+ "(" retries-left "retries left)")
+ (let [retry-prompt (str "Some mutations still survive after your previous tests. "
+ "Please write additional tests to kill them.\n\n"
+ "## Still-Surviving Mutations\n\n"
+ (str/join "\n\n"
+ (for [{:keys [description mutation]} all-survived]
+ (str "### " description "\n```clojure\n" mutation "\n```")))
+ "\n\n## Instructions\n"
+ "- The test file is at: " test-path "\n"
+ "- Use the Edit tool to add more tests\n"
+ "- Focus specifically on the mutations listed above\n"
+ "- Do NOT duplicate existing tests\n")]
+ (invoke-claude! retry-prompt))
+ (recur (dec retries-left)))
+ {:status :partial :killed all-killed :survived all-survived})))))
+
+(defn process-group!
+ "Process one group of functions: branch → tests → verify → commit → PR.
+ Returns {:pr-url ... :issue ... :killed ... :not-killed ...}."
+ [session group]
+ (let [{:keys [target-ns test-ns test-path]} session
+ {:keys [primary-fn fn-names mutations]} group
+ primary-fn-name (name primary-fn)]
+ ;; 1. Create branch
+ (println " Creating branch...")
+ (create-branch! (str target-ns) primary-fn-name)
+
+ ;; 2. Count tests before
+ (require test-ns :reload)
+ (let [tests-before (count-tests test-ns)]
+
+ ;; 3. Build prompt and invoke Claude
+ (println " Invoking Claude to write tests...")
+ (let [prompt (build-test-prompt {:target-ns target-ns
+ :test-ns test-ns
+ :source-path (:source-path session)
+ :test-path test-path
+ :fn-names fn-names
+ :mutations mutations})]
+ (invoke-claude! prompt))
+
+ ;; 4. Verify and retry
+ (println " Verifying mutations killed...")
+ (let [verify-result (verify-and-retry! {:fn-names fn-names
+ :test-ns test-ns
+ :test-path test-path
+ :max-retries 2})
+ killed (:killed verify-result)
+ survived (:survived verify-result)
+ tests-after (count-tests test-ns)
+ tests-added (- tests-after tests-before)]
+
+ ;; 5. Commit and push
+ (println " Committing and pushing...")
+ (commit-and-push! (str target-ns) primary-fn-name [test-path])
+
+ ;; 6. Create Linear issue
+ (println " Creating Linear issue...")
+ (let [issue (create-issue-for-function! (str target-ns) primary-fn-name)
+
+ ;; 7. Create draft PR
+ _ (println " Creating draft PR...")
+ pr-url (create-draft-pr!
+ {:target-ns (str target-ns)
+ :fn-name primary-fn-name
+ :fn-names (mapv name fn-names)
+ :linear-identifier (:identifier issue)
+ :mutations-before (count mutations)
+ :tests-added tests-added
+ :killed (mapv :description killed)
+ :not-killed (mapv (fn [m] {:description (:description m)
+ :rationale "Could not be killed automatically"})
+ survived)
+ :suggested-changes []})]
+
+ ;; 8. Return to base branch
+ (return-to-base!)
+
+ (println " Done! PR:" pr-url)
+ {:pr-url pr-url
+ :issue issue
+ :killed killed
+ :not-killed survived
+ :tests-added tests-added})))))
+
+(defn print-summary!
+ "Print a summary of the mutation testing run."
+ [{:keys [project groups results]}]
+ (println "\n========================================")
+ (println " Mutation Testing Summary")
+ (println "========================================")
+ (println "Project:" (:name project))
+ (println "Groups:" (count groups))
+ (let [successful (filter :pr-url results)
+ failed (filter :error results)
+ total-killed (count (mapcat :killed successful))
+ total-unkilled (count (mapcat :not-killed successful))
+ total-tests (reduce + 0 (keep :tests-added successful))]
+ (println "PRs created:" (count successful))
+ (when (seq failed)
+ (println "Errors:" (count failed)))
+ (println "Tests added:" total-tests)
+ (println "Mutations killed:" total-killed)
+ (println "Mutations not killed:" total-unkilled)
+ (println)
+ (when (seq successful)
+ (println "PRs:")
+ (doseq [r successful]
+ (println " " (:pr-url r))))
+ (when (seq failed)
+ (println "\nFailed:")
+ (doseq [r failed]
+ (println " " (:primary-fn r) "-" (:error r))))
+ (println "========================================")))
+
+(defn run!
+ "Main entry point. Runs the full mutation testing workflow for a namespace.
+
+ Prerequisites:
+ - nREPL running
+ - LINEAR_API_KEY env var set
+ - gh CLI authenticated
+ - team-id configured via (set-config! {:team-id \"...\"})
+
+ Options (optional second arg):
+ - :base-branch — git branch to create feature branches from (default: from config, or \"master\")
+ - :project-id — existing Linear project ID to use (skips creating a new project)"
+ ([target-ns-sym] (run! target-ns-sym {}))
+ ([target-ns-sym opts]
+ (when-let [bb (:base-branch opts)]
+ (set-config! {:base-branch bb}))
+ (when-let [pid (:project-id opts)]
+ (set-config! {:project-id pid}))
+ (let [parsed (parse-namespace target-ns-sym)
+ {:keys [target-ns test-ns report-path]} parsed]
+ ;; 1. Generate report
+ (println "Generating mutation testing report for" (str target-ns) "...")
+ (coverage/generate-report target-ns [test-ns] report-path)
+ (println "Report written to" report-path)
+
+ ;; 2. Create or reuse Linear project
+ (let [project (if (:project-id @config)
+ (do (println "Using existing Linear project:" (:project-id @config))
+ {:project-id (:project-id @config)
+ :name (str "Existing project " (:project-id @config))})
+ (do (println "Creating Linear project...")
+ (let [p (create-project-for-namespace! (str target-ns) report-path)]
+ (println "Created project:" (:name p))
+ p)))]
+
+ ;; 3. Run coverage and group functions
+ (println "Running coverage analysis and grouping...")
+ (let [coverage-results (coverage/test-namespace target-ns [test-ns])
+ groups (group-functions coverage-results)]
+ (println "Found" (count groups) "function groups to process")
+
+ ;; 4. Process each group
+ (let [results (vec
+ (for [group groups]
+ (do
+ (println "\n--- Processing:" (:primary-fn group)
+ "(" (count (:mutations group)) "surviving mutations) ---")
+ (try
+ (process-group! parsed group)
+ (catch Exception e
+ (println "ERROR:" (.getMessage e))
+ (try (return-to-base!) (catch Exception _))
+ {:error (.getMessage e)
+ :primary-fn (:primary-fn group)})))))]
+
+ ;; 5. Print summary
+ (print-summary! {:project project
+ :groups groups
+ :results results})
+ results))))))
+
(comment
;; Setup:
;; 1. Set LINEAR_API_KEY env var before starting the REPL