The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Auto-fix modules config.edn (#77541)
Ben Grabowba83271710df
diff --git a/.claude/skills/clojure-write/SKILL.md b/.claude/skills/clojure-write/SKILL.mdindex 350cca74658..e08ab1f37c1 100644--- a/.claude/skills/clojure-write/SKILL.md+++ b/.claude/skills/clojure-write/SKILL.md@@ -102,3 +102,7 @@ breakage keeps happening, *that* is the signal a comment was warranted.- End all files with a newline- When editing tabular code, where the columns line up, try to keep them aligned- Spaces on a line with nothing after it is not allowed+- After changing module boundaries (adding/removing/renaming a `src` namespace, a cross-module `require`+ or `:model/X` reference, or a new module), run `./bin/mage fix-modules-config` to regenerate+ `.clj-kondo/config/modules/config.edn` and keep `metabase.core.modules-test` green. No-op when nothing+ drifted; see "Module Boundaries" in the project `CLAUDE.md`.diff --git a/CLAUDE.md b/CLAUDE.mdindex 22cdee4d79e..ca3924fc0a2 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -58,6 +58,25 @@ For module-scoped runs — useful when validating a branch's blast radius — paOnce again, do not use `clj -X:dev:test` directly — its progress-bar output is hard to parse.+## Module Boundaries++The linter config at `.clj-kondo/config/modules/config.edn` records each module's `:api`, `:uses`,+`:model-exports`, and `:model-imports`. `metabase.core.modules-test` fails when it drifts from the source.++After **any** backend change that could shift module boundaries, regenerate it:++```bash+./bin/mage fix-modules-config+```++Changes that shift boundaries include: adding/removing/renaming a `src` namespace, adding or dropping a+cross-module `require` or `:model/X` reference, or creating a new module. When unsure, just run it — it is+a no-op (exits `unchanged`) when nothing drifted.++It piggybacks on a running dev nREPL (~5s) and auto-spawns a JVM if none is running (~15s). It only edits+the four generated keys; structural changes it can't safely make (a new module needs a human `:team`, or+modules need reordering) are printed as `WARNING:` lines for you to resolve by hand.+## Tool PreferencesIf `clojure-mcp` tools are available, prefer them over shell-based alternatives for Clojure development.\ No newline at end of filediff --git a/bb.edn b/bb.ednindex 3ccd336c5ab..b97953d6076 100644--- a/bb.edn+++ b/bb.edn@@ -265,6 +265,15 @@:requires [[mage.modules]]:task (task! (mage.modules/cli-can-skip-driver-tests (:arguments parsed)))}+ fix-modules-config+ {:doc "Regenerate .clj-kondo/config/modules/config.edn so it passes metabase.core.modules-test."+ :examples [["./bin/mage fix-modules-config" "Fix the modules config (uses the running dev REPL if available, else a JVM)"]+ ["./bin/mage fix-modules-config --port 59498" "Use the dev REPL on a specific nREPL port"]]+ ;; Cold-JVM fallback invokes `clojure -X:dev dev.modules-config/fix-config!`.+ :options [["-p" "--port PORT" "nREPL port to use, defaults to value in .nrepl-port"]]+ :requires [[mage.modules]]+ :task (task! (mage.modules/cli-fix-config (cli/parse! (current-task))))}+ls{:doc "List all mage public and private tasks":examples [["./bin/mage ls" "List all tasks"]]diff --git a/dev/resources/dev/modules_config_test/expected-config.edn b/dev/resources/dev/modules_config_test/expected-config.ednnew file mode 100644index 00000000000..f0a70083777--- /dev/null+++ b/dev/resources/dev/modules_config_test/expected-config.edn@@ -0,0 +1,44 @@+;;+;; Synthetic modules config used by dev.modules-config-test. Small, but structurally a full config:+;; a header comment block plus a `:metabase/modules` map. Each module here is shaped to exercise a+;; different branch of the rewriter (sentinel preservation, resorting, append, removal, comment+;; preservation, and the library-module skip). See the test for the intent behind each.+;;+{:metabase/modules+ {alpha+ {:team "Test Team"+ ;; :api is a sentinel — must survive even though a real :api is computed for alpha.+ :api :any+ ;; unsorted + carries an element comment; rewriting to sorted order drops the comment.+ :uses #{beta gamma}+ ;; stale: missing :model/Gamma, must be rewritten.+ :model-imports #{:model/Beta :model/Gamma}+ :model-exports #{:model/Alpha}}++ beta+ {:team "Test Team"+ ;; :api missing entirely — must be appended.+ :uses #{}+ :model-exports #{:model/Beta}+ :api #{metabase.beta.core}}++ connection-pool+ {:team "Test Team"+ ;; Library-sourced module absent from the deps graph; the rewriter must leave it untouched.+ :api #{metabase.connection-pool}}++ gamma+ {:team "Test Team"+ :api :any+ :uses :any+ ;; bypass sentinel — preserved; also means gamma drives no model-imports.+ :model-imports :bypass+ :model-exports #{:model/Gamma}}++ enterprise/zeta+ {:team "Test Team"+ :api #{}+ ;; already correct + sorted; the element comment must be preserved untouched.+ :uses #{alpha ; core dependency+ beta}+ :model-imports #{:model/Alpha}}}}diff --git a/dev/resources/dev/modules_config_test/input-config.edn b/dev/resources/dev/modules_config_test/input-config.ednnew file mode 100644index 00000000000..65807f60f1b--- /dev/null+++ b/dev/resources/dev/modules_config_test/input-config.edn@@ -0,0 +1,42 @@+;;+;; Synthetic modules config used by dev.modules-config-test. Small, but structurally a full config:+;; a header comment block plus a `:metabase/modules` map. Each module here is shaped to exercise a+;; different branch of the rewriter (sentinel preservation, resorting, append, removal, comment+;; preservation, and the library-module skip). See the test for the intent behind each.+;;+{:metabase/modules+ {alpha+ {:team "Test Team"+ ;; :api is a sentinel — must survive even though a real :api is computed for alpha.+ :api :any+ ;; unsorted + carries an element comment; rewriting to sorted order drops the comment.+ :uses #{gamma ; used heavily+ beta}+ ;; stale: missing :model/Gamma, must be rewritten.+ :model-imports #{:model/Beta}}++ beta+ {:team "Test Team"+ ;; :api missing entirely — must be appended.+ :uses #{}+ :model-exports #{:model/Beta}}++ connection-pool+ {:team "Test Team"+ ;; Library-sourced module absent from the deps graph; the rewriter must leave it untouched.+ :api #{metabase.connection-pool}}++ gamma+ {:team "Test Team"+ :api :any+ :uses :any+ ;; bypass sentinel — preserved; also means gamma drives no model-imports.+ :model-imports :bypass}++ enterprise/zeta+ {:team "Test Team"+ :api #{}+ ;; already correct + sorted; the element comment must be preserved untouched.+ :uses #{alpha ; core dependency+ beta}+ :model-exports #{:model/Zeta}}}}diff --git a/dev/resources/dev/modules_config_test/mocks.edn b/dev/resources/dev/modules_config_test/mocks.ednnew file mode 100644index 00000000000..a5ea7f96214--- /dev/null+++ b/dev/resources/dev/modules_config_test/mocks.edn@@ -0,0 +1,36 @@+;;+;; Mock inputs for dev.modules-config-test — the four values dev.modules-config/compute-desired takes.+;; Shaped like the real dev.deps-graph outputs, but tiny and hand-authored so the expected output is+;; predictable. `:config` mirrors (kondo-config): connection-pool is intentionally absent (the deps graph+;; dissocs it), which is what makes the rewriter leave that module untouched.+;;+{:config+ {alpha {}+ beta {}+ gamma {:model-imports :bypass}+ enterprise/zeta {}}++ ;; one entry per source file: {:module M :namespace NS :deps [{:module M :namespace NS} ...]}+ :dependencies+ [{:module alpha :namespace metabase.alpha.core+ :deps [{:module beta :namespace metabase.beta.core}+ {:module gamma :namespace metabase.gamma.core}]}+ {:module beta :namespace metabase.beta.core :deps []}+ {:module gamma :namespace metabase.gamma.core :deps []}+ {:module enterprise/zeta :namespace metabase-enterprise.zeta.core+ :deps [{:module alpha :namespace metabase.alpha.core}+ {:module beta :namespace metabase.beta.core}]}]++ ;; {:model/X => owning module}+ :model-ownership+ {:model/Alpha alpha+ :model/Beta beta+ :model/Gamma gamma+ :model/Zeta enterprise/zeta}++ ;; {module => #{models it references}}+ :model-references+ {alpha #{:model/Alpha :model/Beta :model/Gamma}+ beta #{:model/Beta}+ gamma #{:model/Gamma :model/Alpha}+ enterprise/zeta #{:model/Zeta :model/Alpha}}}diff --git a/dev/src/dev/model_boundary_config.clj b/dev/src/dev/model_boundary_config.cljindex ec9528e98dc..f74e6c6cdbd 100644--- a/dev/src/dev/model_boundary_config.clj+++ b/dev/src/dev/model_boundary_config.clj@@ -29,50 +29,54 @@`:model-imports` for module M = models owned by other modules that M references.Modules with `:model-imports :bypass` are excluded: they don't need computed imports, and their- references don't drive exports (models used only by bypass modules need not be exported)."- []- (let [config (deps-graph/kondo-config)- ownership (deps-graph/model-ownership)- module-refs (deps-graph/model-references-by-module)- all-modules (set (keys config))- bypass-modules (into #{}- (keep (fn [[mod mod-config]]- (when (= (:model-imports mod-config) :bypass)- mod)))- config)- ;; {:model/X => #{modules that reference it}} — inverted index for fast export lookups- ;; Only non-bypass modules drive exports.- model->referencing-modules- (reduce-kv (fn [acc mod models]- (if (contains? bypass-modules mod)- acc- (reduce (fn [acc model]- (update acc model (fnil conj #{}) mod))- acc- models)))- {}- module-refs)]- {:model-exports- (into (sorted-map)- (for [mod all-modules- :let [owned (into #{} (comp (filter (fn [[_ owner]] (= owner mod))) (map key)) ownership)]- :when (seq owned)]- [mod (into (sorted-set)- (for [model owned- :let [refs (get model->referencing-modules model)]- :when (some #(not= % mod) refs)]- model))]))- :model-imports- (into (sorted-map)- (for [mod all-modules- :when (not (contains? bypass-modules mod))- :let [imported (into (sorted-set)- (for [model (get module-refs mod)- :let [defining-mod (get ownership model)]- :when (and defining-mod (not= defining-mod mod))]- model))]- :when (seq imported)]- [mod imported]))}))+ references don't drive exports (models used only by bypass modules need not be exported).++ The 3-arity variant takes precomputed `config`, `ownership`, and `module-refs` so callers that already+ have them (e.g. a single shared parse pass) can avoid re-parsing every source file."+ ([]+ (compute-model-boundaries (deps-graph/kondo-config)+ (deps-graph/model-ownership)+ (deps-graph/model-references-by-module)))+ ([config ownership module-refs]+ (let [all-modules (set (keys config))+ bypass-modules (into #{}+ (keep (fn [[mod mod-config]]+ (when (= (:model-imports mod-config) :bypass)+ mod)))+ config)+ ;; {:model/X => #{modules that reference it}} — inverted index for fast export lookups+ ;; Only non-bypass modules drive exports.+ model->referencing-modules+ (reduce-kv (fn [acc mod models]+ (if (contains? bypass-modules mod)+ acc+ (reduce (fn [acc model]+ (update acc model (fnil conj #{}) mod))+ acc+ models)))+ {}+ module-refs)]+ {:model-exports+ (into (sorted-map)+ (for [mod all-modules+ :let [owned (into #{} (comp (filter (fn [[_ owner]] (= owner mod))) (map key)) ownership)]+ :when (seq owned)]+ [mod (into (sorted-set)+ (for [model owned+ :let [refs (get model->referencing-modules model)]+ :when (some #(not= % mod) refs)]+ model))]))+ :model-imports+ (into (sorted-map)+ (for [mod all-modules+ :when (not (contains? bypass-modules mod))+ :let [imported (into (sorted-set)+ (for [model (get module-refs mod)+ :let [defining-mod (get ownership model)]+ :when (and defining-mod (not= defining-mod mod))]+ model))]+ :when (seq imported)]+ [mod imported]))})))(def ^:private config-path ".clj-kondo/config/modules/config.edn")diff --git a/dev/src/dev/modules_config.clj b/dev/src/dev/modules_config.cljnew file mode 100644index 00000000000..567f602ca17--- /dev/null+++ b/dev/src/dev/modules_config.clj@@ -0,0 +1,308 @@+(ns dev.modules-config+ "Generate/repair `.clj-kondo/config/modules/config.edn` so it passes `metabase.core.modules-test`.++ The heavy lifting already lives in [[dev.deps-graph]] (which computes the correct `:api` and `:uses`+ for every module) and [[dev.model-boundary-config]] (which computes `:model-exports`/`:model-imports`).+ This namespace ties them together into a single writer that rewrites the four *generated* keys in place,+ sorted, while preserving everything a human owns: `:team`, `:friends`, header/footer comments, inline+ `;;` annotations on set elements, the `:ignored-namespace-patterns` key, and the sentinel values+ `:any`/`:bypass`.++ Usage:++ (dev.modules-config/update-config!) ; from a warm REPL — fast++ Or from a cold JVM (see `./bin/mage fix-modules-config`):++ clojure -X:dev dev.modules-config/-main++ Scope: this auto-fixes the *content and ordering* of `:api`, `:uses`, `:model-exports`, and+ `:model-imports` for modules that already exist in the config. Structural changes — adding a brand new+ module (which needs a human-assigned `:team`), removing a stale one, or reordering modules — are only+ *reported* as warnings, never performed, since they require judgement this tool doesn't have."+ (:require+ [clojure.string :as str]+ [dev.deps-graph :as deps-graph]+ [dev.model-boundary-config :as mbc]+ [rewrite-clj.node :as n]+ [rewrite-clj.parser :as r.parser]+ [rewrite-clj.zip :as z]))++(set! *warn-on-reflection* true)++(def ^:private config-path ".clj-kondo/config/modules/config.edn")++(def ^:private generated-keys+ "The keys this tool owns and rewrites. Order matters only for where appended keys land."+ [:api :uses :model-exports :model-imports])++;;;; ---------------------------------------------------------------------------+;;;; Ordering — must match `metabase.core.modules-test`+;;;; ---------------------------------------------------------------------------++(defn- enterprise? [module-name]+ (str/starts-with? (str module-name) "enterprise/"))++(defn- sort-module-names+ "Sort module symbols with `enterprise/` modules last. Mirrors the test's `sort-module-names`."+ [module-names]+ (sort-by (juxt #(if (enterprise? %) 1 0) str) module-names))++(defn- sort-for-key+ "Return the sorted element vector for a generated key. `:uses` sorts modules enterprise-last; everything+ else (namespace symbols for `:api`, `:model/X` keywords for the model keys) sorts naturally."+ [k elements]+ (vec (if (= k :uses)+ (sort-module-names elements)+ (sort elements))))++;;;; ---------------------------------------------------------------------------+;;;; Set nodes+;;;; ---------------------------------------------------------------------------++(defn- set-elements+ "Ordered vector of element sexprs in a `:set` node, in file order. (`z/sexpr` would return an unordered+ set, useless for the sortedness comparison.)"+ [set-node]+ (into [] (comp (filter #(= :token (n/tag %))) (map n/sexpr))+ (n/children set-node)))++(defn- build-set-node+ "rewrite-clj `:set` node for `sorted-elements`. Small sets (<=3) stay inline; larger ones go one element+ per line, continuation lines indented `indent` spaces to align under the first element. Matches the+ hand-formatting convention used throughout config.edn (and by [[dev.model-boundary-config]])."+ [sorted-elements indent]+ (r.parser/parse-string+ (if (<= (count sorted-elements) 3)+ (str "#{" (str/join " " sorted-elements) "}")+ (let [pad (apply str (repeat indent \space))]+ (str "#{" (str/join (str "\n" pad) sorted-elements) "}")))))++;;;; ---------------------------------------------------------------------------+;;;; Zipper navigation+;;;; ---------------------------------------------------------------------------++(defn- modules-map-zloc+ "Zipper at the `:metabase/modules` map value."+ [root-node]+ (-> (z/of-node root-node)+ (z/find-value z/next :metabase/modules)+ z/right))++(defn- module-config-zloc+ "Zipper at the config map for `module-sym`, or nil."+ [root-node module-sym]+ (some-> (z/find-value (z/down (modules-map-zloc root-node)) z/right module-sym)+ z/right))++(defn- key-value-zloc+ "Zipper at the value of `k` inside a map zipper, or nil."+ [map-zloc k]+ (some-> (z/find-value (z/down map-zloc) z/right k)+ z/right))++(defn- module-syms-in-order+ "Vector of module symbols in file order (skips non-symbol keys like `:ignored-namespace-patterns`)."+ [root-node]+ (loop [zloc (z/down (modules-map-zloc root-node)), acc []]+ (if-not zloc+ acc+ (let [k (z/sexpr zloc)]+ (recur (some-> zloc z/right z/right)+ (cond-> acc (symbol? k) (conj k)))))))++;;;; ---------------------------------------------------------------------------+;;;; Editing+;;;; ---------------------------------------------------------------------------++(defn- append-indent+ "Continuation indent for a freshly appended `:key #{...}` line: aligns elements under the first one."+ [k]+ ;; " :key " + "#{" -> element starts two columns past `#`+ (+ 3 (count (str k)) 1 2))++(defn- update-module-key+ "Rewrite one generated `key` of one module in `root-node`, returning the new root node.+ `computed` is the desired set (possibly empty). `indent` is the continuation indent for an existing set+ (from its `#{` column) or nil to fall back to an appended-key indent."+ [root-node module-sym k computed indent]+ (let [map-zloc (module-config-zloc root-node module-sym)+ val-zloc (when map-zloc (key-value-zloc map-zloc k))]+ (cond+ ;; No such module (structural — handled/warned elsewhere).+ (nil? map-zloc) root-node++ ;; Sentinel value (`:any`/`:bypass`) — human-owned, leave untouched.+ (and val-zloc (not (set? (z/sexpr val-zloc)))) root-node++ ;; Existing set value.+ val-zloc+ (let [elements (set-elements (z/node val-zloc))+ target (sort-for-key k computed)]+ (cond+ (empty? computed)+ (if (#{:model-exports :model-imports} k)+ ;; Model keys default to absent; drop the now-empty key entirely.+ (-> val-zloc z/remove z/remove z/root)+ ;; :api/:uses stay explicit; normalize to `#{}` only if not already empty.+ (if (empty? elements)+ root-node+ (z/root (z/replace val-zloc (n/set-node [])))))++ ;; Already correct and sorted — leave the node (and its comments) byte-for-byte.+ (= elements target) root-node++ :else+ (z/root (z/replace val-zloc (build-set-node target (or indent (append-indent k)))))))++ ;; Key missing but needed — append it to the module's config map.+ (empty? computed) root-node++ :else+ (let [m-node (z/node map-zloc)+ set-node (build-set-node (sort-for-key k computed) (append-indent k))+ children (concat (n/children m-node)+ [(n/newlines 1) (n/spaces 3)+ (n/keyword-node k) (n/spaces 1) set-node])]+ (z/root (z/replace map-zloc (n/replace-children m-node children)))))))++;;;; ---------------------------------------------------------------------------+;;;; Indents (captured from the pristine, position-tracked parse)+;;;; ---------------------------------------------------------------------------++(defn- collect-indents+ "Map of `[module-sym key]` -> continuation-indent (spaces) for every existing generated set value,+ read from element `#{` columns so rebuilt sets align exactly like their neighbours. Navigates the+ position-tracked zipper `pos-root` directly (re-wrapping via `z/of-node` would drop positions)."+ [pos-root]+ (let [mods (-> pos-root (z/find-value z/next :metabase/modules) z/right)]+ (loop [zloc (z/down mods), acc {}]+ (if-not zloc+ acc+ (let [k (z/sexpr zloc)+ vmap (z/right zloc)]+ (recur (some-> vmap z/right)+ (if (symbol? k)+ (reduce+ (fn [acc gk]+ (let [vz (some-> (z/find-value (z/down vmap) z/right gk) z/right)]+ (if (and vz (= :set (z/tag vz)))+ (assoc acc [k gk] (inc (second (z/position vz))))+ acc)))+ acc+ generated-keys)+ acc)))))))++;;;; ---------------------------------------------------------------------------+;;;; Compute the desired config (single shared parse pass)+;;;; ---------------------------------------------------------------------------++(defn compute-desired+ "Compute the desired value of every generated key for every module.++ Returns `{module-sym {:api set, :uses set, :model-exports set, :model-imports set}}`.++ The 4-arity is pure: it derives the answer purely from the four inputs (the parsed Kondo `config`, the+ `dependencies` file-scan, `model-ownership`, and `model-references`), so it can be exercised in tests+ with mock data and no file or REPL state. The 0-arity gathers those four inputs from [[dev.deps-graph]]+ for real, running the independent file-scanning passes concurrently so a warm REPL finishes in a few+ seconds."+ ([]+ (let [config (deps-graph/kondo-config)+ f-deps (future (deps-graph/dependencies))+ f-own (future (deps-graph/model-ownership))+ f-refs (future (deps-graph/model-references-by-module))]+ (compute-desired config @f-deps @f-own @f-refs)))+ ([config dependencies model-ownership model-references]+ (let [api+uses (deps-graph/generate-config dependencies config)+ boundaries (mbc/compute-model-boundaries config model-ownership model-references)+ modules (into #{} (concat (keys api+uses)+ (keys (:model-exports boundaries))+ (keys (:model-imports boundaries))))]+ (into {}+ (map (fn [module]+ [module {:api (get-in api+uses [module :api] #{})+ :uses (get-in api+uses [module :uses] #{})+ :model-exports (get-in boundaries [:model-exports module] #{})+ :model-imports (get-in boundaries [:model-imports module] #{})}]))+ modules))))++;;;; ---------------------------------------------------------------------------+;;;; Structural warnings (things this tool won't auto-fix)+;;;; ---------------------------------------------------------------------------++(defn- structural-warnings+ "Sequence of human-readable warnings for structural mismatches this tool won't touch."+ [file-modules desired]+ (let [file-set (set file-modules)+ desired-set (set (keys desired))+ to-add (sort (remove file-set desired-set))+ ;; a desired module absent from the file needs adding; but connection-pool etc. legitimately live+ ;; only in the file, so only flag file modules that produced *no* generated data as candidates.+ sorted? (= file-modules (sort-module-names file-modules))]+ (cond-> []+ (seq to-add)+ (conj (str "New module(s) not in config — add them by hand (they need a :team): "+ (str/join ", " to-add)))++ (not sorted?)+ (conj "Modules are not sorted by name (enterprise/ last) — reorder them by hand."))))++;;;; ---------------------------------------------------------------------------+;;;; Entry point+;;;; ---------------------------------------------------------------------------++(defn rewrite-config+ "Pure core of the tool. Given the current config.edn `original` text and the `desired` per-module+ generated-key map (see [[compute-desired]]), return++ {:text the rewritten config text+ :warnings [human-readable strings for structural mismatches this tool won't touch]}++ Idempotent: a config already matching `desired` yields `:text` byte-for-byte equal to `original`. Only+ modules present in `desired` are touched, so library-sourced entries like `connection-pool` (dissoc'd+ from the deps graph) are left alone."+ [original desired]+ (let [pos-root (z/of-string original {:track-position? true})+ indents (collect-indents pos-root)+ ;; full `:forms` node: keeps the header/footer comment blocks that surround the top-level map.+ full-root (z/root pos-root)+ file-modules (module-syms-in-order full-root)+ edited (reduce+ (fn [root-node module-sym]+ (reduce+ (fn [root-node k]+ (update-module-key root-node module-sym k+ (get-in desired [module-sym k] #{})+ (get indents [module-sym k])))+ root-node+ generated-keys))+ full-root+ (filter desired file-modules))]+ {:text (n/string edited)+ :warnings (structural-warnings file-modules desired)}))++(defn update-config!+ "Rewrite `.clj-kondo/config/modules/config.edn` so the generated keys are correct and sorted.+ Idempotent: a clean config is left byte-for-byte unchanged. Returns `:updated` or `:unchanged`."+ []+ (let [original (slurp config-path)+ {:keys [text warnings]} (rewrite-config original (compute-desired))]+ #_{:clj-kondo/ignore [:discouraged-var]}+ (doseq [w warnings]+ (println (str "WARNING: " w)))+ (if (= text original)+ (do #_{:clj-kondo/ignore [:discouraged-var]}+ (println "config.edn already up to date.")+ :unchanged)+ (do (spit config-path text)+ #_{:clj-kondo/ignore [:discouraged-var]}+ (println "Updated" config-path)+ :updated))))++(defn fix-config!+ "Cold-JVM entry point for `clojure -X:dev dev.modules-config/fix-config!`. Takes (and ignores) the+ `-X` exec-args map."+ [_]+ (update-config!)+ (shutdown-agents))diff --git a/dev/test/dev/modules_config_test.clj b/dev/test/dev/modules_config_test.cljnew file mode 100644index 00000000000..de7465e489a--- /dev/null+++ b/dev/test/dev/modules_config_test.clj@@ -0,0 +1,147 @@+(ns dev.modules-config-test+ (:require+ [clojure.edn :as edn]+ [clojure.java.io :as io]+ [clojure.string :as str]+ [clojure.test :refer :all]+ [dev.modules-config :as mc]+ [rewrite-clj.node :as n]+ [rewrite-clj.parser :as r.parser]))++(defn- resource [path]+ (slurp (io/resource (str "dev/modules_config_test/" path))))++(defn- rewrite+ "Rewrite `input` text against `desired`, returning just the text."+ [input desired]+ (:text (mc/rewrite-config input desired)))++;;;; ---------------------------------------------------------------------------+;;;; End-to-end golden test: full config + mock inputs -> expected config+;;;; ---------------------------------------------------------------------------++(deftest ^:parallel full-config-rewrite-test+ (testing "a full config.edn rewritten against mock deps-graph inputs matches the expected resource"+ (let [{:keys [config dependencies model-ownership model-references]}+ (edn/read-string (resource "mocks.edn"))+ desired (mc/compute-desired config dependencies model-ownership model-references)+ {:keys [text warnings]} (mc/rewrite-config (resource "input-config.edn") desired)]+ (is (= (resource "expected-config.edn") text))+ (is (= [] warnings)+ "the scenario is structurally sound (no new/unsorted modules)")+ (testing "and rewriting is idempotent — re-running on the output is a no-op"+ (is (= text (rewrite text desired)))))))++;;;; ---------------------------------------------------------------------------+;;;; compute-desired is pure: value in, value out+;;;; ---------------------------------------------------------------------------++(deftest ^:parallel compute-desired-test+ (let [{:keys [config dependencies model-ownership model-references]}+ (edn/read-string (resource "mocks.edn"))+ desired (mc/compute-desired config dependencies model-ownership model-references)]+ (testing ":api = a module's namespaces used by *other* modules; :uses = modules it depends on"+ (is (= #{'metabase.beta.core} (get-in desired ['beta :api])))+ (is (= #{'beta 'gamma} (get-in desired ['alpha :uses])))+ (is (= #{} (get-in desired ['enterprise/zeta :api]))+ "zeta is used by nobody"))+ (testing ":model-exports = owned models referenced elsewhere; a model only used at home is not exported"+ (is (= #{:model/Alpha} (get-in desired ['alpha :model-exports])))+ (is (= #{} (get-in desired ['enterprise/zeta :model-exports]))+ "zeta owns :model/Zeta but only references it itself"))+ (testing ":model-imports = referenced models owned by other modules"+ (is (= #{:model/Beta :model/Gamma} (get-in desired ['alpha :model-imports]))))+ (testing "a :bypass module drives no imports and its references don't force exports"+ (is (= #{} (get-in desired ['gamma :model-imports]))))))++;;;; ---------------------------------------------------------------------------+;;;; Sorting (must match metabase.core.modules-test)+;;;; ---------------------------------------------------------------------------++(deftest ^:parallel sort-module-names-test+ (testing "enterprise/ modules sort last, others alphabetically"+ (is (= '[alpha beta enterprise/aaa enterprise/zzz]+ (#'mc/sort-module-names '[enterprise/zzz beta enterprise/aaa alpha])))))++(deftest ^:parallel sort-for-key-test+ (testing ":uses sorts modules enterprise-last; other keys sort naturally"+ (is (= '[alpha enterprise/zeta] (#'mc/sort-for-key :uses '[enterprise/zeta alpha])))+ (is (= '[a.b a.c a.d] (#'mc/sort-for-key :api '[a.d a.b a.c])))+ (is (= [:model/A :model/B] (#'mc/sort-for-key :model-imports [:model/B :model/A])))))++;;;; ---------------------------------------------------------------------------+;;;; Set nodes+;;;; ---------------------------------------------------------------------------++(deftest ^:parallel set-elements-test+ (testing "elements come back in file order (not the unordered order z/sexpr would give)"+ (is (= '[gamma alpha beta]+ (#'mc/set-elements (r.parser/parse-string "#{gamma alpha beta}"))))+ (is (= '[a b]+ (#'mc/set-elements (r.parser/parse-string "#{a ; trailing\n b}")))+ "comments and whitespace are skipped")))++(deftest ^:parallel build-set-node-test+ (testing "<=3 elements stay inline"+ (is (= "#{a b c}" (n/string (#'mc/build-set-node '[a b c] 10)))))+ (testing ">3 elements go one per line, continuation lines indented to `indent`"+ (is (= "#{a\n b\n c\n d}"+ (n/string (#'mc/build-set-node '[a b c d] 4)))))+ (testing "empty set"+ (is (= "#{}" (n/string (#'mc/build-set-node [] 4))))))++;;;; ---------------------------------------------------------------------------+;;;; rewrite-config branches (via tiny hand-written configs)+;;;; ---------------------------------------------------------------------------++(defn- parse-modules [text]+ (:metabase/modules (edn/read-string text)))++(deftest ^:parallel sentinel-values-are-preserved-test+ (testing ":any and :bypass survive even when a concrete value is computed"+ (let [input "{:metabase/modules {m {:api :any :uses :any :model-imports :bypass}}}"]+ (is (= input (rewrite input '{m {:api #{metabase.m.core}+ :uses #{other}+ :model-imports #{:model/X}}}))))))++(deftest ^:parallel missing-key-is-appended-test+ (let [input "{:metabase/modules {m {:team \"T\"}}}"+ output (rewrite input '{m {:api #{metabase.m.core}}})]+ (is (= #{'metabase.m.core} (get-in (parse-modules output) ['m :api])))))++(deftest ^:parallel unsorted-set-is-resorted-test+ (let [input "{:metabase/modules {m {:uses #{gamma beta alpha}}}}"]+ (is (str/includes? (rewrite input '{m {:uses #{alpha beta gamma}}})+ "#{alpha beta gamma}"))))++(deftest ^:parallel empty-computed-values-test+ (testing "an emptied model key is removed entirely"+ (let [input "{:metabase/modules {m {:model-exports #{:model/X}}}}"+ output (rewrite input '{m {:model-exports #{}}})]+ (is (not (contains? (get (parse-modules output) 'm) :model-exports)))))+ (testing "an emptied :api is normalized to #{} rather than removed"+ (let [input "{:metabase/modules {m {:api #{metabase.m.core}}}}"+ output (rewrite input '{m {:api #{}}})]+ (is (= #{} (get-in (parse-modules output) ['m :api]))))))++(deftest ^:parallel comment-preservation-test+ (testing "a comment inside a set that does NOT change is preserved"+ (let [input "{:metabase/modules {m {:uses #{alpha ; keep me\n beta}}}}"]+ (is (str/includes? (rewrite input '{m {:uses #{alpha beta}}}) "; keep me"))))+ (testing "a comment inside a set that IS resorted is dropped"+ (let [input "{:metabase/modules {m {:uses #{beta ; drop me\n alpha}}}}"]+ (is (not (str/includes? (rewrite input '{m {:uses #{alpha beta}}}) "; drop me"))))))++;;;; ---------------------------------------------------------------------------+;;;; Structural warnings (things the tool won't auto-fix, only reports)+;;;; ---------------------------------------------------------------------------++(deftest ^:parallel structural-warnings-test+ (testing "a desired module absent from the file is reported (needs a human-assigned :team)"+ (let [{:keys [warnings]} (mc/rewrite-config "{:metabase/modules {a {:uses #{}}}}"+ '{a {:uses #{}} b {:uses #{}}})]+ (is (some #(str/includes? % "b") warnings))))+ (testing "out-of-order modules are reported"+ (let [{:keys [warnings]} (mc/rewrite-config "{:metabase/modules {beta {:uses #{}} alpha {:uses #{}}}}"+ '{alpha {:uses #{}} beta {:uses #{}}})]+ (is (some #(str/includes? % "sorted") warnings)))))diff --git a/mage/src/mage/be_dev.clj b/mage/src/mage/be_dev.cljindex 7438cf00713..c3ab4d57d20 100644--- a/mage/src/mage/be_dev.clj+++ b/mage/src/mage/be_dev.clj@@ -7,9 +7,11 @@[clojure.string :as str][clojure.walk :as walk][mage.color :as c]+ [mage.shell :as shell][mage.util :as u])(:import- [java.io InputStream]))+ [java.io InputStream PushbackInputStream]+ (java.net ConnectException Socket)))(set! *warn-on-reflection* true)@@ -36,21 +38,21 @@"Acquire a Java socket or die."[host port](try- (java.net.Socket. host port)- (catch java.net.ConnectException _+ (Socket. ^String host ^int port)+ (catch ConnectException _(println(str "Could not connect to the REPL server on port: " (c/red port) " (found port number in .nrepl-port).\n""Is the Metabase backend running?\n\n""To start it, run:\n"(c/green " clj -M:dev:ee:ee-dev:drivers:drivers-dev:dev-start\n\n")"If you prefer a different way to start the backend, please use that instead."))- (System/exit 1))+ (u/exit 1))(catch Exception e(if (= "No matching ctor found for class java.net.Socket" (ex-message e))(do(println (format "There seems to be an error specifying port: '%s'."(c/yellow (pr-str port))))- (System/exit 1))+ (u/exit 1))(throw e)))))(defn ^:private consume [v]@@ -83,7 +85,7 @@(let [port (nrepl-port port)s (socket! "localhost" port)out (.getOutputStream s)- in (java.io.PushbackInputStream. (.getInputStream s))+ in (PushbackInputStream. (.getInputStream s))code-str (str (eval-in-ns nns code))_ (u/debug "Code:\n-----\n" code-str "\n-----")_ (bencode/write-bencode out {:op "eval" :code code-str})@@ -102,7 +104,6 @@;; Flush to ensure output appears immediately(flush)-(if (some #{"done"} (get response "status"))(some->> @final-value(safe-print "\n=>\n"))@@ -120,6 +121,48 @@(= "2" o))(catch Exception _ false))))+(defn- resolve-port+ "Best-effort nREPL port: explicit `port`, else `.nrepl-port`, else nil. Never throws."+ [port]+ (or (safe-parse-int port)+ (try (safe-parse-int (slurp ".nrepl-port"))+ (catch Exception _ nil))))++(defn nrepl-reachable?+ "Whether something is accepting TCP connections on the dev nREPL port (explicit `port`, else `.nrepl-port`).++ A raw-socket probe: it never evaluates code and never exits the process, so it is safe to branch on when+ choosing between a running REPL and a cold-JVM fallback. (Contrast [[nrepl-open?]], which evaluates+ through [[socket!]] and will `System/exit` if the port file is stale.)"+ ([] (nrepl-reachable? nil))+ ([port]+ (boolean+ (when-let [p (resolve-port port)]+ (try+ (with-open [_ (Socket. "localhost" (int p))] true)+ (catch Exception _ false))))))++(defn eval-or-spawn+ "Run backend work that needs a live JVM, preferring a running dev nREPL for speed and spawning a cold JVM+ when none is reachable. Returns the exit code (0 for the nREPL path, else the subprocess exit code).++ The reusable form of the \"piggyback on the dev REPL, else boot a JVM\" pattern any mage command can use+ when its work depends on a running backend.++ :nrepl-ns namespace to evaluate `:nrepl-code` in when a REPL is reachable+ :nrepl-code Clojure code string to evaluate in the running REPL+ :jvm-args args passed to `clojure` for the cold-JVM fallback, e.g. [\"-X:dev\" \"my.ns/entry\"]+ :port explicit nREPL port (defaults to `.nrepl-port`)+ :nrepl-msg status line printed when taking the REPL path+ :jvm-msg status line printed when taking the JVM path"+ [{:keys [nrepl-ns nrepl-code jvm-args port nrepl-msg jvm-msg]}]+ (if (nrepl-reachable? port)+ (do (when nrepl-msg (println nrepl-msg))+ (nrepl-eval nrepl-ns nrepl-code (resolve-port port))+ 0)+ (do (when jvm-msg (println jvm-msg))+ (:exit (apply shell/sh* "clojure" jvm-args)))))+(defn nrepl-type"Returns :bb :cljs or :clj to indicate what type of nrepl server is running on the given port (or the port in .nrepl-port if none given)."([] (nrepl-type nil))diff --git a/mage/src/mage/modules.clj b/mage/src/mage/modules.cljindex 21bef96eb25..25bb9db39a4 100644--- a/mage/src/mage/modules.clj+++ b/mage/src/mage/modules.clj@@ -5,6 +5,7 @@[clojure.edn :as edn][clojure.set :as set][clojure.string :as str]+ [mage.be-dev :as be-dev][mage.color :as c][mage.util :as u]))@@ -254,6 +255,28 @@drivers-affected? 1:else 0))))+;;;; =============================================================================+;;;; Fix modules config+;;;; =============================================================================+(defn cli-fix-config+ "Regenerate `.clj-kondo/config/modules/config.edn` so it passes `metabase.core.modules-test`.++ Fast path: evaluate in the running dev nREPL (a few seconds). Fallback: spawn a cold JVM (~25s) when no+ dev REPL is running."+ [{:keys [options] :as _parsed}]+ (let [port (some-> (:port options) str str/trim parse-long)+ timer (u/start-timer)+ exit (be-dev/eval-or-spawn+ {:port port+ :nrepl-ns "dev.modules-config"+ :nrepl-code "(update-config!)"+ :jvm-args ["-X:dev" "dev.modules-config/fix-config!"]+ :nrepl-msg (c/green "Regenerating modules config via the running dev REPL...")+ :jvm-msg (c/yellow "No dev REPL found — starting a JVM (slower; start your dev REPL for ~5s runs)...")})]+ (printf "\nFinished in %.1fs\n" (/ (u/since-ms timer) 1000.0))+ (flush)+ (u/exit (or exit 0))))+;;;; =============================================================================;;;; Driver test decisions - consolidated logic for which drivers to run;;;; =============================================================================