Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Auto-fix modules config.edn (#77541)

Ben Grabowba83271710df
diff --git a/.claude/skills/clojure-write/SKILL.md b/.claude/skills/clojure-write/SKILL.md
index 350cca74658..e08ab1f37c1 100644
--- a/.claude/skills/clojure-write/SKILL.md
+++ b/.claude/skills/clojure-write/SKILL.md
@@ -102,3 +102,7 @@ breakage keeps happening, *that* is the signal a comment was warranted.
- End all files with a newline
- When editing tabular code, where the columns line up, try to keep them aligned
- Spaces on a line with nothing after it is not allowed
+- After changing module boundaries (adding/removing/renaming a `src` namespace, a cross-module `require`
+ or `:model/X` reference, or a new module), run `./bin/mage fix-modules-config` to regenerate
+ `.clj-kondo/config/modules/config.edn` and keep `metabase.core.modules-test` green. No-op when nothing
+ drifted; see "Module Boundaries" in the project `CLAUDE.md`.
diff --git a/CLAUDE.md b/CLAUDE.md
index 22cdee4d79e..ca3924fc0a2 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -58,6 +58,25 @@ For module-scoped runs — useful when validating a branch's blast radius — pa
Once again, do not use `clj -X:dev:test` directly — its progress-bar output is hard to parse.
+## Module Boundaries
+
+The linter config at `.clj-kondo/config/modules/config.edn` records each module's `:api`, `:uses`,
+`:model-exports`, and `:model-imports`. `metabase.core.modules-test` fails when it drifts from the source.
+
+After **any** backend change that could shift module boundaries, regenerate it:
+
+```bash
+./bin/mage fix-modules-config
+```
+
+Changes that shift boundaries include: adding/removing/renaming a `src` namespace, adding or dropping a
+cross-module `require` or `:model/X` reference, or creating a new module. When unsure, just run it — it is
+a no-op (exits `unchanged`) when nothing drifted.
+
+It piggybacks on a running dev nREPL (~5s) and auto-spawns a JVM if none is running (~15s). It only edits
+the four generated keys; structural changes it can't safely make (a new module needs a human `:team`, or
+modules need reordering) are printed as `WARNING:` lines for you to resolve by hand.
+
## Tool Preferences
If `clojure-mcp` tools are available, prefer them over shell-based alternatives for Clojure development.
\ No newline at end of file
diff --git a/bb.edn b/bb.edn
index 3ccd336c5ab..b97953d6076 100644
--- a/bb.edn
+++ b/bb.edn
@@ -265,6 +265,15 @@
:requires [[mage.modules]]
:task (task! (mage.modules/cli-can-skip-driver-tests (:arguments parsed)))}
+ fix-modules-config
+ {:doc "Regenerate .clj-kondo/config/modules/config.edn so it passes metabase.core.modules-test."
+ :examples [["./bin/mage fix-modules-config" "Fix the modules config (uses the running dev REPL if available, else a JVM)"]
+ ["./bin/mage fix-modules-config --port 59498" "Use the dev REPL on a specific nREPL port"]]
+ ;; Cold-JVM fallback invokes `clojure -X:dev dev.modules-config/fix-config!`.
+ :options [["-p" "--port PORT" "nREPL port to use, defaults to value in .nrepl-port"]]
+ :requires [[mage.modules]]
+ :task (task! (mage.modules/cli-fix-config (cli/parse! (current-task))))}
+
ls
{:doc "List all mage public and private tasks"
:examples [["./bin/mage ls" "List all tasks"]]
diff --git a/dev/resources/dev/modules_config_test/expected-config.edn b/dev/resources/dev/modules_config_test/expected-config.edn
new file mode 100644
index 00000000000..f0a70083777
--- /dev/null
+++ b/dev/resources/dev/modules_config_test/expected-config.edn
@@ -0,0 +1,44 @@
+;;
+;; Synthetic modules config used by dev.modules-config-test. Small, but structurally a full config:
+;; a header comment block plus a `:metabase/modules` map. Each module here is shaped to exercise a
+;; different branch of the rewriter (sentinel preservation, resorting, append, removal, comment
+;; preservation, and the library-module skip). See the test for the intent behind each.
+;;
+{:metabase/modules
+ {alpha
+ {:team "Test Team"
+ ;; :api is a sentinel — must survive even though a real :api is computed for alpha.
+ :api :any
+ ;; unsorted + carries an element comment; rewriting to sorted order drops the comment.
+ :uses #{beta gamma}
+ ;; stale: missing :model/Gamma, must be rewritten.
+ :model-imports #{:model/Beta :model/Gamma}
+ :model-exports #{:model/Alpha}}
+
+ beta
+ {:team "Test Team"
+ ;; :api missing entirely — must be appended.
+ :uses #{}
+ :model-exports #{:model/Beta}
+ :api #{metabase.beta.core}}
+
+ connection-pool
+ {:team "Test Team"
+ ;; Library-sourced module absent from the deps graph; the rewriter must leave it untouched.
+ :api #{metabase.connection-pool}}
+
+ gamma
+ {:team "Test Team"
+ :api :any
+ :uses :any
+ ;; bypass sentinel — preserved; also means gamma drives no model-imports.
+ :model-imports :bypass
+ :model-exports #{:model/Gamma}}
+
+ enterprise/zeta
+ {:team "Test Team"
+ :api #{}
+ ;; already correct + sorted; the element comment must be preserved untouched.
+ :uses #{alpha ; core dependency
+ beta}
+ :model-imports #{:model/Alpha}}}}
diff --git a/dev/resources/dev/modules_config_test/input-config.edn b/dev/resources/dev/modules_config_test/input-config.edn
new file mode 100644
index 00000000000..65807f60f1b
--- /dev/null
+++ b/dev/resources/dev/modules_config_test/input-config.edn
@@ -0,0 +1,42 @@
+;;
+;; Synthetic modules config used by dev.modules-config-test. Small, but structurally a full config:
+;; a header comment block plus a `:metabase/modules` map. Each module here is shaped to exercise a
+;; different branch of the rewriter (sentinel preservation, resorting, append, removal, comment
+;; preservation, and the library-module skip). See the test for the intent behind each.
+;;
+{:metabase/modules
+ {alpha
+ {:team "Test Team"
+ ;; :api is a sentinel — must survive even though a real :api is computed for alpha.
+ :api :any
+ ;; unsorted + carries an element comment; rewriting to sorted order drops the comment.
+ :uses #{gamma ; used heavily
+ beta}
+ ;; stale: missing :model/Gamma, must be rewritten.
+ :model-imports #{:model/Beta}}
+
+ beta
+ {:team "Test Team"
+ ;; :api missing entirely — must be appended.
+ :uses #{}
+ :model-exports #{:model/Beta}}
+
+ connection-pool
+ {:team "Test Team"
+ ;; Library-sourced module absent from the deps graph; the rewriter must leave it untouched.
+ :api #{metabase.connection-pool}}
+
+ gamma
+ {:team "Test Team"
+ :api :any
+ :uses :any
+ ;; bypass sentinel — preserved; also means gamma drives no model-imports.
+ :model-imports :bypass}
+
+ enterprise/zeta
+ {:team "Test Team"
+ :api #{}
+ ;; already correct + sorted; the element comment must be preserved untouched.
+ :uses #{alpha ; core dependency
+ beta}
+ :model-exports #{:model/Zeta}}}}
diff --git a/dev/resources/dev/modules_config_test/mocks.edn b/dev/resources/dev/modules_config_test/mocks.edn
new file mode 100644
index 00000000000..a5ea7f96214
--- /dev/null
+++ b/dev/resources/dev/modules_config_test/mocks.edn
@@ -0,0 +1,36 @@
+;;
+;; Mock inputs for dev.modules-config-test — the four values dev.modules-config/compute-desired takes.
+;; Shaped like the real dev.deps-graph outputs, but tiny and hand-authored so the expected output is
+;; predictable. `:config` mirrors (kondo-config): connection-pool is intentionally absent (the deps graph
+;; dissocs it), which is what makes the rewriter leave that module untouched.
+;;
+{:config
+ {alpha {}
+ beta {}
+ gamma {:model-imports :bypass}
+ enterprise/zeta {}}
+
+ ;; one entry per source file: {:module M :namespace NS :deps [{:module M :namespace NS} ...]}
+ :dependencies
+ [{:module alpha :namespace metabase.alpha.core
+ :deps [{:module beta :namespace metabase.beta.core}
+ {:module gamma :namespace metabase.gamma.core}]}
+ {:module beta :namespace metabase.beta.core :deps []}
+ {:module gamma :namespace metabase.gamma.core :deps []}
+ {:module enterprise/zeta :namespace metabase-enterprise.zeta.core
+ :deps [{:module alpha :namespace metabase.alpha.core}
+ {:module beta :namespace metabase.beta.core}]}]
+
+ ;; {:model/X => owning module}
+ :model-ownership
+ {:model/Alpha alpha
+ :model/Beta beta
+ :model/Gamma gamma
+ :model/Zeta enterprise/zeta}
+
+ ;; {module => #{models it references}}
+ :model-references
+ {alpha #{:model/Alpha :model/Beta :model/Gamma}
+ beta #{:model/Beta}
+ gamma #{:model/Gamma :model/Alpha}
+ enterprise/zeta #{:model/Zeta :model/Alpha}}}
diff --git a/dev/src/dev/model_boundary_config.clj b/dev/src/dev/model_boundary_config.clj
index ec9528e98dc..f74e6c6cdbd 100644
--- a/dev/src/dev/model_boundary_config.clj
+++ b/dev/src/dev/model_boundary_config.clj
@@ -29,50 +29,54 @@
`:model-imports` for module M = models owned by other modules that M references.
Modules with `:model-imports :bypass` are excluded: they don't need computed imports, and their
- references don't drive exports (models used only by bypass modules need not be exported)."
- []
- (let [config (deps-graph/kondo-config)
- ownership (deps-graph/model-ownership)
- module-refs (deps-graph/model-references-by-module)
- all-modules (set (keys config))
- bypass-modules (into #{}
- (keep (fn [[mod mod-config]]
- (when (= (:model-imports mod-config) :bypass)
- mod)))
- config)
- ;; {:model/X => #{modules that reference it}} — inverted index for fast export lookups
- ;; Only non-bypass modules drive exports.
- model->referencing-modules
- (reduce-kv (fn [acc mod models]
- (if (contains? bypass-modules mod)
- acc
- (reduce (fn [acc model]
- (update acc model (fnil conj #{}) mod))
- acc
- models)))
- {}
- module-refs)]
- {:model-exports
- (into (sorted-map)
- (for [mod all-modules
- :let [owned (into #{} (comp (filter (fn [[_ owner]] (= owner mod))) (map key)) ownership)]
- :when (seq owned)]
- [mod (into (sorted-set)
- (for [model owned
- :let [refs (get model->referencing-modules model)]
- :when (some #(not= % mod) refs)]
- model))]))
- :model-imports
- (into (sorted-map)
- (for [mod all-modules
- :when (not (contains? bypass-modules mod))
- :let [imported (into (sorted-set)
- (for [model (get module-refs mod)
- :let [defining-mod (get ownership model)]
- :when (and defining-mod (not= defining-mod mod))]
- model))]
- :when (seq imported)]
- [mod imported]))}))
+ references don't drive exports (models used only by bypass modules need not be exported).
+
+ The 3-arity variant takes precomputed `config`, `ownership`, and `module-refs` so callers that already
+ have them (e.g. a single shared parse pass) can avoid re-parsing every source file."
+ ([]
+ (compute-model-boundaries (deps-graph/kondo-config)
+ (deps-graph/model-ownership)
+ (deps-graph/model-references-by-module)))
+ ([config ownership module-refs]
+ (let [all-modules (set (keys config))
+ bypass-modules (into #{}
+ (keep (fn [[mod mod-config]]
+ (when (= (:model-imports mod-config) :bypass)
+ mod)))
+ config)
+ ;; {:model/X => #{modules that reference it}} — inverted index for fast export lookups
+ ;; Only non-bypass modules drive exports.
+ model->referencing-modules
+ (reduce-kv (fn [acc mod models]
+ (if (contains? bypass-modules mod)
+ acc
+ (reduce (fn [acc model]
+ (update acc model (fnil conj #{}) mod))
+ acc
+ models)))
+ {}
+ module-refs)]
+ {:model-exports
+ (into (sorted-map)
+ (for [mod all-modules
+ :let [owned (into #{} (comp (filter (fn [[_ owner]] (= owner mod))) (map key)) ownership)]
+ :when (seq owned)]
+ [mod (into (sorted-set)
+ (for [model owned
+ :let [refs (get model->referencing-modules model)]
+ :when (some #(not= % mod) refs)]
+ model))]))
+ :model-imports
+ (into (sorted-map)
+ (for [mod all-modules
+ :when (not (contains? bypass-modules mod))
+ :let [imported (into (sorted-set)
+ (for [model (get module-refs mod)
+ :let [defining-mod (get ownership model)]
+ :when (and defining-mod (not= defining-mod mod))]
+ model))]
+ :when (seq imported)]
+ [mod imported]))})))
(def ^:private config-path ".clj-kondo/config/modules/config.edn")
diff --git a/dev/src/dev/modules_config.clj b/dev/src/dev/modules_config.clj
new file mode 100644
index 00000000000..567f602ca17
--- /dev/null
+++ b/dev/src/dev/modules_config.clj
@@ -0,0 +1,308 @@
+(ns dev.modules-config
+ "Generate/repair `.clj-kondo/config/modules/config.edn` so it passes `metabase.core.modules-test`.
+
+ The heavy lifting already lives in [[dev.deps-graph]] (which computes the correct `:api` and `:uses`
+ for every module) and [[dev.model-boundary-config]] (which computes `:model-exports`/`:model-imports`).
+ This namespace ties them together into a single writer that rewrites the four *generated* keys in place,
+ sorted, while preserving everything a human owns: `:team`, `:friends`, header/footer comments, inline
+ `;;` annotations on set elements, the `:ignored-namespace-patterns` key, and the sentinel values
+ `:any`/`:bypass`.
+
+ Usage:
+
+ (dev.modules-config/update-config!) ; from a warm REPL — fast
+
+ Or from a cold JVM (see `./bin/mage fix-modules-config`):
+
+ clojure -X:dev dev.modules-config/-main
+
+ Scope: this auto-fixes the *content and ordering* of `:api`, `:uses`, `:model-exports`, and
+ `:model-imports` for modules that already exist in the config. Structural changes — adding a brand new
+ module (which needs a human-assigned `:team`), removing a stale one, or reordering modules — are only
+ *reported* as warnings, never performed, since they require judgement this tool doesn't have."
+ (:require
+ [clojure.string :as str]
+ [dev.deps-graph :as deps-graph]
+ [dev.model-boundary-config :as mbc]
+ [rewrite-clj.node :as n]
+ [rewrite-clj.parser :as r.parser]
+ [rewrite-clj.zip :as z]))
+
+(set! *warn-on-reflection* true)
+
+(def ^:private config-path ".clj-kondo/config/modules/config.edn")
+
+(def ^:private generated-keys
+ "The keys this tool owns and rewrites. Order matters only for where appended keys land."
+ [:api :uses :model-exports :model-imports])
+
+;;;; ---------------------------------------------------------------------------
+;;;; Ordering — must match `metabase.core.modules-test`
+;;;; ---------------------------------------------------------------------------
+
+(defn- enterprise? [module-name]
+ (str/starts-with? (str module-name) "enterprise/"))
+
+(defn- sort-module-names
+ "Sort module symbols with `enterprise/` modules last. Mirrors the test's `sort-module-names`."
+ [module-names]
+ (sort-by (juxt #(if (enterprise? %) 1 0) str) module-names))
+
+(defn- sort-for-key
+ "Return the sorted element vector for a generated key. `:uses` sorts modules enterprise-last; everything
+ else (namespace symbols for `:api`, `:model/X` keywords for the model keys) sorts naturally."
+ [k elements]
+ (vec (if (= k :uses)
+ (sort-module-names elements)
+ (sort elements))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Set nodes
+;;;; ---------------------------------------------------------------------------
+
+(defn- set-elements
+ "Ordered vector of element sexprs in a `:set` node, in file order. (`z/sexpr` would return an unordered
+ set, useless for the sortedness comparison.)"
+ [set-node]
+ (into [] (comp (filter #(= :token (n/tag %))) (map n/sexpr))
+ (n/children set-node)))
+
+(defn- build-set-node
+ "rewrite-clj `:set` node for `sorted-elements`. Small sets (<=3) stay inline; larger ones go one element
+ per line, continuation lines indented `indent` spaces to align under the first element. Matches the
+ hand-formatting convention used throughout config.edn (and by [[dev.model-boundary-config]])."
+ [sorted-elements indent]
+ (r.parser/parse-string
+ (if (<= (count sorted-elements) 3)
+ (str "#{" (str/join " " sorted-elements) "}")
+ (let [pad (apply str (repeat indent \space))]
+ (str "#{" (str/join (str "\n" pad) sorted-elements) "}")))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Zipper navigation
+;;;; ---------------------------------------------------------------------------
+
+(defn- modules-map-zloc
+ "Zipper at the `:metabase/modules` map value."
+ [root-node]
+ (-> (z/of-node root-node)
+ (z/find-value z/next :metabase/modules)
+ z/right))
+
+(defn- module-config-zloc
+ "Zipper at the config map for `module-sym`, or nil."
+ [root-node module-sym]
+ (some-> (z/find-value (z/down (modules-map-zloc root-node)) z/right module-sym)
+ z/right))
+
+(defn- key-value-zloc
+ "Zipper at the value of `k` inside a map zipper, or nil."
+ [map-zloc k]
+ (some-> (z/find-value (z/down map-zloc) z/right k)
+ z/right))
+
+(defn- module-syms-in-order
+ "Vector of module symbols in file order (skips non-symbol keys like `:ignored-namespace-patterns`)."
+ [root-node]
+ (loop [zloc (z/down (modules-map-zloc root-node)), acc []]
+ (if-not zloc
+ acc
+ (let [k (z/sexpr zloc)]
+ (recur (some-> zloc z/right z/right)
+ (cond-> acc (symbol? k) (conj k)))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Editing
+;;;; ---------------------------------------------------------------------------
+
+(defn- append-indent
+ "Continuation indent for a freshly appended `:key #{...}` line: aligns elements under the first one."
+ [k]
+ ;; " :key " + "#{" -> element starts two columns past `#`
+ (+ 3 (count (str k)) 1 2))
+
+(defn- update-module-key
+ "Rewrite one generated `key` of one module in `root-node`, returning the new root node.
+ `computed` is the desired set (possibly empty). `indent` is the continuation indent for an existing set
+ (from its `#{` column) or nil to fall back to an appended-key indent."
+ [root-node module-sym k computed indent]
+ (let [map-zloc (module-config-zloc root-node module-sym)
+ val-zloc (when map-zloc (key-value-zloc map-zloc k))]
+ (cond
+ ;; No such module (structural — handled/warned elsewhere).
+ (nil? map-zloc) root-node
+
+ ;; Sentinel value (`:any`/`:bypass`) — human-owned, leave untouched.
+ (and val-zloc (not (set? (z/sexpr val-zloc)))) root-node
+
+ ;; Existing set value.
+ val-zloc
+ (let [elements (set-elements (z/node val-zloc))
+ target (sort-for-key k computed)]
+ (cond
+ (empty? computed)
+ (if (#{:model-exports :model-imports} k)
+ ;; Model keys default to absent; drop the now-empty key entirely.
+ (-> val-zloc z/remove z/remove z/root)
+ ;; :api/:uses stay explicit; normalize to `#{}` only if not already empty.
+ (if (empty? elements)
+ root-node
+ (z/root (z/replace val-zloc (n/set-node [])))))
+
+ ;; Already correct and sorted — leave the node (and its comments) byte-for-byte.
+ (= elements target) root-node
+
+ :else
+ (z/root (z/replace val-zloc (build-set-node target (or indent (append-indent k)))))))
+
+ ;; Key missing but needed — append it to the module's config map.
+ (empty? computed) root-node
+
+ :else
+ (let [m-node (z/node map-zloc)
+ set-node (build-set-node (sort-for-key k computed) (append-indent k))
+ children (concat (n/children m-node)
+ [(n/newlines 1) (n/spaces 3)
+ (n/keyword-node k) (n/spaces 1) set-node])]
+ (z/root (z/replace map-zloc (n/replace-children m-node children)))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Indents (captured from the pristine, position-tracked parse)
+;;;; ---------------------------------------------------------------------------
+
+(defn- collect-indents
+ "Map of `[module-sym key]` -> continuation-indent (spaces) for every existing generated set value,
+ read from element `#{` columns so rebuilt sets align exactly like their neighbours. Navigates the
+ position-tracked zipper `pos-root` directly (re-wrapping via `z/of-node` would drop positions)."
+ [pos-root]
+ (let [mods (-> pos-root (z/find-value z/next :metabase/modules) z/right)]
+ (loop [zloc (z/down mods), acc {}]
+ (if-not zloc
+ acc
+ (let [k (z/sexpr zloc)
+ vmap (z/right zloc)]
+ (recur (some-> vmap z/right)
+ (if (symbol? k)
+ (reduce
+ (fn [acc gk]
+ (let [vz (some-> (z/find-value (z/down vmap) z/right gk) z/right)]
+ (if (and vz (= :set (z/tag vz)))
+ (assoc acc [k gk] (inc (second (z/position vz))))
+ acc)))
+ acc
+ generated-keys)
+ acc)))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Compute the desired config (single shared parse pass)
+;;;; ---------------------------------------------------------------------------
+
+(defn compute-desired
+ "Compute the desired value of every generated key for every module.
+
+ Returns `{module-sym {:api set, :uses set, :model-exports set, :model-imports set}}`.
+
+ The 4-arity is pure: it derives the answer purely from the four inputs (the parsed Kondo `config`, the
+ `dependencies` file-scan, `model-ownership`, and `model-references`), so it can be exercised in tests
+ with mock data and no file or REPL state. The 0-arity gathers those four inputs from [[dev.deps-graph]]
+ for real, running the independent file-scanning passes concurrently so a warm REPL finishes in a few
+ seconds."
+ ([]
+ (let [config (deps-graph/kondo-config)
+ f-deps (future (deps-graph/dependencies))
+ f-own (future (deps-graph/model-ownership))
+ f-refs (future (deps-graph/model-references-by-module))]
+ (compute-desired config @f-deps @f-own @f-refs)))
+ ([config dependencies model-ownership model-references]
+ (let [api+uses (deps-graph/generate-config dependencies config)
+ boundaries (mbc/compute-model-boundaries config model-ownership model-references)
+ modules (into #{} (concat (keys api+uses)
+ (keys (:model-exports boundaries))
+ (keys (:model-imports boundaries))))]
+ (into {}
+ (map (fn [module]
+ [module {:api (get-in api+uses [module :api] #{})
+ :uses (get-in api+uses [module :uses] #{})
+ :model-exports (get-in boundaries [:model-exports module] #{})
+ :model-imports (get-in boundaries [:model-imports module] #{})}]))
+ modules))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Structural warnings (things this tool won't auto-fix)
+;;;; ---------------------------------------------------------------------------
+
+(defn- structural-warnings
+ "Sequence of human-readable warnings for structural mismatches this tool won't touch."
+ [file-modules desired]
+ (let [file-set (set file-modules)
+ desired-set (set (keys desired))
+ to-add (sort (remove file-set desired-set))
+ ;; a desired module absent from the file needs adding; but connection-pool etc. legitimately live
+ ;; only in the file, so only flag file modules that produced *no* generated data as candidates.
+ sorted? (= file-modules (sort-module-names file-modules))]
+ (cond-> []
+ (seq to-add)
+ (conj (str "New module(s) not in config — add them by hand (they need a :team): "
+ (str/join ", " to-add)))
+
+ (not sorted?)
+ (conj "Modules are not sorted by name (enterprise/ last) — reorder them by hand."))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Entry point
+;;;; ---------------------------------------------------------------------------
+
+(defn rewrite-config
+ "Pure core of the tool. Given the current config.edn `original` text and the `desired` per-module
+ generated-key map (see [[compute-desired]]), return
+
+ {:text the rewritten config text
+ :warnings [human-readable strings for structural mismatches this tool won't touch]}
+
+ Idempotent: a config already matching `desired` yields `:text` byte-for-byte equal to `original`. Only
+ modules present in `desired` are touched, so library-sourced entries like `connection-pool` (dissoc'd
+ from the deps graph) are left alone."
+ [original desired]
+ (let [pos-root (z/of-string original {:track-position? true})
+ indents (collect-indents pos-root)
+ ;; full `:forms` node: keeps the header/footer comment blocks that surround the top-level map.
+ full-root (z/root pos-root)
+ file-modules (module-syms-in-order full-root)
+ edited (reduce
+ (fn [root-node module-sym]
+ (reduce
+ (fn [root-node k]
+ (update-module-key root-node module-sym k
+ (get-in desired [module-sym k] #{})
+ (get indents [module-sym k])))
+ root-node
+ generated-keys))
+ full-root
+ (filter desired file-modules))]
+ {:text (n/string edited)
+ :warnings (structural-warnings file-modules desired)}))
+
+(defn update-config!
+ "Rewrite `.clj-kondo/config/modules/config.edn` so the generated keys are correct and sorted.
+ Idempotent: a clean config is left byte-for-byte unchanged. Returns `:updated` or `:unchanged`."
+ []
+ (let [original (slurp config-path)
+ {:keys [text warnings]} (rewrite-config original (compute-desired))]
+ #_{:clj-kondo/ignore [:discouraged-var]}
+ (doseq [w warnings]
+ (println (str "WARNING: " w)))
+ (if (= text original)
+ (do #_{:clj-kondo/ignore [:discouraged-var]}
+ (println "config.edn already up to date.")
+ :unchanged)
+ (do (spit config-path text)
+ #_{:clj-kondo/ignore [:discouraged-var]}
+ (println "Updated" config-path)
+ :updated))))
+
+(defn fix-config!
+ "Cold-JVM entry point for `clojure -X:dev dev.modules-config/fix-config!`. Takes (and ignores) the
+ `-X` exec-args map."
+ [_]
+ (update-config!)
+ (shutdown-agents))
diff --git a/dev/test/dev/modules_config_test.clj b/dev/test/dev/modules_config_test.clj
new file mode 100644
index 00000000000..de7465e489a
--- /dev/null
+++ b/dev/test/dev/modules_config_test.clj
@@ -0,0 +1,147 @@
+(ns dev.modules-config-test
+ (:require
+ [clojure.edn :as edn]
+ [clojure.java.io :as io]
+ [clojure.string :as str]
+ [clojure.test :refer :all]
+ [dev.modules-config :as mc]
+ [rewrite-clj.node :as n]
+ [rewrite-clj.parser :as r.parser]))
+
+(defn- resource [path]
+ (slurp (io/resource (str "dev/modules_config_test/" path))))
+
+(defn- rewrite
+ "Rewrite `input` text against `desired`, returning just the text."
+ [input desired]
+ (:text (mc/rewrite-config input desired)))
+
+;;;; ---------------------------------------------------------------------------
+;;;; End-to-end golden test: full config + mock inputs -> expected config
+;;;; ---------------------------------------------------------------------------
+
+(deftest ^:parallel full-config-rewrite-test
+ (testing "a full config.edn rewritten against mock deps-graph inputs matches the expected resource"
+ (let [{:keys [config dependencies model-ownership model-references]}
+ (edn/read-string (resource "mocks.edn"))
+ desired (mc/compute-desired config dependencies model-ownership model-references)
+ {:keys [text warnings]} (mc/rewrite-config (resource "input-config.edn") desired)]
+ (is (= (resource "expected-config.edn") text))
+ (is (= [] warnings)
+ "the scenario is structurally sound (no new/unsorted modules)")
+ (testing "and rewriting is idempotent — re-running on the output is a no-op"
+ (is (= text (rewrite text desired)))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; compute-desired is pure: value in, value out
+;;;; ---------------------------------------------------------------------------
+
+(deftest ^:parallel compute-desired-test
+ (let [{:keys [config dependencies model-ownership model-references]}
+ (edn/read-string (resource "mocks.edn"))
+ desired (mc/compute-desired config dependencies model-ownership model-references)]
+ (testing ":api = a module's namespaces used by *other* modules; :uses = modules it depends on"
+ (is (= #{'metabase.beta.core} (get-in desired ['beta :api])))
+ (is (= #{'beta 'gamma} (get-in desired ['alpha :uses])))
+ (is (= #{} (get-in desired ['enterprise/zeta :api]))
+ "zeta is used by nobody"))
+ (testing ":model-exports = owned models referenced elsewhere; a model only used at home is not exported"
+ (is (= #{:model/Alpha} (get-in desired ['alpha :model-exports])))
+ (is (= #{} (get-in desired ['enterprise/zeta :model-exports]))
+ "zeta owns :model/Zeta but only references it itself"))
+ (testing ":model-imports = referenced models owned by other modules"
+ (is (= #{:model/Beta :model/Gamma} (get-in desired ['alpha :model-imports]))))
+ (testing "a :bypass module drives no imports and its references don't force exports"
+ (is (= #{} (get-in desired ['gamma :model-imports]))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Sorting (must match metabase.core.modules-test)
+;;;; ---------------------------------------------------------------------------
+
+(deftest ^:parallel sort-module-names-test
+ (testing "enterprise/ modules sort last, others alphabetically"
+ (is (= '[alpha beta enterprise/aaa enterprise/zzz]
+ (#'mc/sort-module-names '[enterprise/zzz beta enterprise/aaa alpha])))))
+
+(deftest ^:parallel sort-for-key-test
+ (testing ":uses sorts modules enterprise-last; other keys sort naturally"
+ (is (= '[alpha enterprise/zeta] (#'mc/sort-for-key :uses '[enterprise/zeta alpha])))
+ (is (= '[a.b a.c a.d] (#'mc/sort-for-key :api '[a.d a.b a.c])))
+ (is (= [:model/A :model/B] (#'mc/sort-for-key :model-imports [:model/B :model/A])))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Set nodes
+;;;; ---------------------------------------------------------------------------
+
+(deftest ^:parallel set-elements-test
+ (testing "elements come back in file order (not the unordered order z/sexpr would give)"
+ (is (= '[gamma alpha beta]
+ (#'mc/set-elements (r.parser/parse-string "#{gamma alpha beta}"))))
+ (is (= '[a b]
+ (#'mc/set-elements (r.parser/parse-string "#{a ; trailing\n b}")))
+ "comments and whitespace are skipped")))
+
+(deftest ^:parallel build-set-node-test
+ (testing "<=3 elements stay inline"
+ (is (= "#{a b c}" (n/string (#'mc/build-set-node '[a b c] 10)))))
+ (testing ">3 elements go one per line, continuation lines indented to `indent`"
+ (is (= "#{a\n b\n c\n d}"
+ (n/string (#'mc/build-set-node '[a b c d] 4)))))
+ (testing "empty set"
+ (is (= "#{}" (n/string (#'mc/build-set-node [] 4))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; rewrite-config branches (via tiny hand-written configs)
+;;;; ---------------------------------------------------------------------------
+
+(defn- parse-modules [text]
+ (:metabase/modules (edn/read-string text)))
+
+(deftest ^:parallel sentinel-values-are-preserved-test
+ (testing ":any and :bypass survive even when a concrete value is computed"
+ (let [input "{:metabase/modules {m {:api :any :uses :any :model-imports :bypass}}}"]
+ (is (= input (rewrite input '{m {:api #{metabase.m.core}
+ :uses #{other}
+ :model-imports #{:model/X}}}))))))
+
+(deftest ^:parallel missing-key-is-appended-test
+ (let [input "{:metabase/modules {m {:team \"T\"}}}"
+ output (rewrite input '{m {:api #{metabase.m.core}}})]
+ (is (= #{'metabase.m.core} (get-in (parse-modules output) ['m :api])))))
+
+(deftest ^:parallel unsorted-set-is-resorted-test
+ (let [input "{:metabase/modules {m {:uses #{gamma beta alpha}}}}"]
+ (is (str/includes? (rewrite input '{m {:uses #{alpha beta gamma}}})
+ "#{alpha beta gamma}"))))
+
+(deftest ^:parallel empty-computed-values-test
+ (testing "an emptied model key is removed entirely"
+ (let [input "{:metabase/modules {m {:model-exports #{:model/X}}}}"
+ output (rewrite input '{m {:model-exports #{}}})]
+ (is (not (contains? (get (parse-modules output) 'm) :model-exports)))))
+ (testing "an emptied :api is normalized to #{} rather than removed"
+ (let [input "{:metabase/modules {m {:api #{metabase.m.core}}}}"
+ output (rewrite input '{m {:api #{}}})]
+ (is (= #{} (get-in (parse-modules output) ['m :api]))))))
+
+(deftest ^:parallel comment-preservation-test
+ (testing "a comment inside a set that does NOT change is preserved"
+ (let [input "{:metabase/modules {m {:uses #{alpha ; keep me\n beta}}}}"]
+ (is (str/includes? (rewrite input '{m {:uses #{alpha beta}}}) "; keep me"))))
+ (testing "a comment inside a set that IS resorted is dropped"
+ (let [input "{:metabase/modules {m {:uses #{beta ; drop me\n alpha}}}}"]
+ (is (not (str/includes? (rewrite input '{m {:uses #{alpha beta}}}) "; drop me"))))))
+
+;;;; ---------------------------------------------------------------------------
+;;;; Structural warnings (things the tool won't auto-fix, only reports)
+;;;; ---------------------------------------------------------------------------
+
+(deftest ^:parallel structural-warnings-test
+ (testing "a desired module absent from the file is reported (needs a human-assigned :team)"
+ (let [{:keys [warnings]} (mc/rewrite-config "{:metabase/modules {a {:uses #{}}}}"
+ '{a {:uses #{}} b {:uses #{}}})]
+ (is (some #(str/includes? % "b") warnings))))
+ (testing "out-of-order modules are reported"
+ (let [{:keys [warnings]} (mc/rewrite-config "{:metabase/modules {beta {:uses #{}} alpha {:uses #{}}}}"
+ '{alpha {:uses #{}} beta {:uses #{}}})]
+ (is (some #(str/includes? % "sorted") warnings)))))
diff --git a/mage/src/mage/be_dev.clj b/mage/src/mage/be_dev.clj
index 7438cf00713..c3ab4d57d20 100644
--- a/mage/src/mage/be_dev.clj
+++ b/mage/src/mage/be_dev.clj
@@ -7,9 +7,11 @@
[clojure.string :as str]
[clojure.walk :as walk]
[mage.color :as c]
+ [mage.shell :as shell]
[mage.util :as u])
(:import
- [java.io InputStream]))
+ [java.io InputStream PushbackInputStream]
+ (java.net ConnectException Socket)))
(set! *warn-on-reflection* true)
@@ -36,21 +38,21 @@
"Acquire a Java socket or die."
[host port]
(try
- (java.net.Socket. host port)
- (catch java.net.ConnectException _
+ (Socket. ^String host ^int port)
+ (catch ConnectException _
(println
(str "Could not connect to the REPL server on port: " (c/red port) " (found port number in .nrepl-port).\n"
"Is the Metabase backend running?\n\n"
"To start it, run:\n"
(c/green " clj -M:dev:ee:ee-dev:drivers:drivers-dev:dev-start\n\n")
"If you prefer a different way to start the backend, please use that instead."))
- (System/exit 1))
+ (u/exit 1))
(catch Exception e
(if (= "No matching ctor found for class java.net.Socket" (ex-message e))
(do
(println (format "There seems to be an error specifying port: '%s'."
(c/yellow (pr-str port))))
- (System/exit 1))
+ (u/exit 1))
(throw e)))))
(defn ^:private consume [v]
@@ -83,7 +85,7 @@
(let [port (nrepl-port port)
s (socket! "localhost" port)
out (.getOutputStream s)
- in (java.io.PushbackInputStream. (.getInputStream s))
+ in (PushbackInputStream. (.getInputStream s))
code-str (str (eval-in-ns nns code))
_ (u/debug "Code:\n-----\n" code-str "\n-----")
_ (bencode/write-bencode out {:op "eval" :code code-str})
@@ -102,7 +104,6 @@
;; Flush to ensure output appears immediately
(flush)
-
(if (some #{"done"} (get response "status"))
(some->> @final-value
(safe-print "\n=>\n"))
@@ -120,6 +121,48 @@
(= "2" o))
(catch Exception _ false))))
+(defn- resolve-port
+ "Best-effort nREPL port: explicit `port`, else `.nrepl-port`, else nil. Never throws."
+ [port]
+ (or (safe-parse-int port)
+ (try (safe-parse-int (slurp ".nrepl-port"))
+ (catch Exception _ nil))))
+
+(defn nrepl-reachable?
+ "Whether something is accepting TCP connections on the dev nREPL port (explicit `port`, else `.nrepl-port`).
+
+ A raw-socket probe: it never evaluates code and never exits the process, so it is safe to branch on when
+ choosing between a running REPL and a cold-JVM fallback. (Contrast [[nrepl-open?]], which evaluates
+ through [[socket!]] and will `System/exit` if the port file is stale.)"
+ ([] (nrepl-reachable? nil))
+ ([port]
+ (boolean
+ (when-let [p (resolve-port port)]
+ (try
+ (with-open [_ (Socket. "localhost" (int p))] true)
+ (catch Exception _ false))))))
+
+(defn eval-or-spawn
+ "Run backend work that needs a live JVM, preferring a running dev nREPL for speed and spawning a cold JVM
+ when none is reachable. Returns the exit code (0 for the nREPL path, else the subprocess exit code).
+
+ The reusable form of the \"piggyback on the dev REPL, else boot a JVM\" pattern any mage command can use
+ when its work depends on a running backend.
+
+ :nrepl-ns namespace to evaluate `:nrepl-code` in when a REPL is reachable
+ :nrepl-code Clojure code string to evaluate in the running REPL
+ :jvm-args args passed to `clojure` for the cold-JVM fallback, e.g. [\"-X:dev\" \"my.ns/entry\"]
+ :port explicit nREPL port (defaults to `.nrepl-port`)
+ :nrepl-msg status line printed when taking the REPL path
+ :jvm-msg status line printed when taking the JVM path"
+ [{:keys [nrepl-ns nrepl-code jvm-args port nrepl-msg jvm-msg]}]
+ (if (nrepl-reachable? port)
+ (do (when nrepl-msg (println nrepl-msg))
+ (nrepl-eval nrepl-ns nrepl-code (resolve-port port))
+ 0)
+ (do (when jvm-msg (println jvm-msg))
+ (:exit (apply shell/sh* "clojure" jvm-args)))))
+
(defn nrepl-type
"Returns :bb :cljs or :clj to indicate what type of nrepl server is running on the given port (or the port in .nrepl-port if none given)."
([] (nrepl-type nil))
diff --git a/mage/src/mage/modules.clj b/mage/src/mage/modules.clj
index 21bef96eb25..25bb9db39a4 100644
--- a/mage/src/mage/modules.clj
+++ b/mage/src/mage/modules.clj
@@ -5,6 +5,7 @@
[clojure.edn :as edn]
[clojure.set :as set]
[clojure.string :as str]
+ [mage.be-dev :as be-dev]
[mage.color :as c]
[mage.util :as u]))
@@ -254,6 +255,28 @@
drivers-affected? 1
:else 0))))
+;;;; =============================================================================
+;;;; Fix modules config
+;;;; =============================================================================
+(defn cli-fix-config
+ "Regenerate `.clj-kondo/config/modules/config.edn` so it passes `metabase.core.modules-test`.
+
+ Fast path: evaluate in the running dev nREPL (a few seconds). Fallback: spawn a cold JVM (~25s) when no
+ dev REPL is running."
+ [{:keys [options] :as _parsed}]
+ (let [port (some-> (:port options) str str/trim parse-long)
+ timer (u/start-timer)
+ exit (be-dev/eval-or-spawn
+ {:port port
+ :nrepl-ns "dev.modules-config"
+ :nrepl-code "(update-config!)"
+ :jvm-args ["-X:dev" "dev.modules-config/fix-config!"]
+ :nrepl-msg (c/green "Regenerating modules config via the running dev REPL...")
+ :jvm-msg (c/yellow "No dev REPL found — starting a JVM (slower; start your dev REPL for ~5s runs)...")})]
+ (printf "\nFinished in %.1fs\n" (/ (u/since-ms timer) 1000.0))
+ (flush)
+ (u/exit (or exit 0))))
+
;;;; =============================================================================
;;;; Driver test decisions - consolidated logic for which drivers to run
;;;; =============================================================================