Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

feat: support transforms in remote sync (#68255)

Edward Pagetd26bf4e454e1
diff --git a/.claude-env-files.edn b/.claude-env-files.edn
new file mode 100644
index 00000000000..4ada47930bb
--- /dev/null
+++ b/.claude-env-files.edn
@@ -0,0 +1 @@
+{"edpaget" ["PROJECT_SUMMARY.md" "LLM_CODE_STYLE.md" "CLAUDE.md" ".claude/agents/clojure-project-executor.md" ".claude/agents/web-assumption-validator.md" ".claude/agents/software-architect-planner.md" ".claude/agents/codebase-behavior-researcher.md" ".claude/commands/orchestrate-clojure.md" ".clojure-mcp/config.edn"]}
\ No newline at end of file
diff --git a/.claude-env-state.edn b/.claude-env-state.edn
new file mode 100644
index 00000000000..2982c31ecfa
--- /dev/null
+++ b/.claude-env-state.edn
@@ -0,0 +1 @@
+{:current-environment "edpaget", :last-updated #inst "2025-08-20T16:01:36.621-00:00"}
\ No newline at end of file
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.edn
index 42e213d2adb..efef3b82a95 100644
--- a/.clj-kondo/config/modules/config.edn
+++ b/.clj-kondo/config/modules/config.edn
@@ -2332,6 +2332,7 @@
premium-features
queries
query-processor
+ remote-sync
request
revisions
search
diff --git a/DECOUPLING_INTERFACES_BROAD_SURVEY.md b/DECOUPLING_INTERFACES_BROAD_SURVEY.md
new file mode 100644
index 00000000000..9b476ca0696
--- /dev/null
+++ b/DECOUPLING_INTERFACES_BROAD_SURVEY.md
@@ -0,0 +1,451 @@
+# Decoupling via Defined Interfaces in Functional Programming
+
+## A Cross-Language Survey of Best Practices
+
+This document surveys interface and decoupling mechanisms across functional programming languages, examining how different languages approach polymorphism, dependency injection, and architectural boundaries.
+
+---
+
+## Table of Contents
+
+1. [Overview](#overview)
+2. [Haskell](#haskell)
+3. [ML Family (OCaml, Standard ML) & Scala](#ml-family-and-scala)
+4. [Elixir](#elixir)
+5. [F#](#f)
+6. [Cross-Cutting Patterns](#cross-cutting-patterns)
+7. [Key Papers and Talks](#key-papers-and-talks)
+8. [Summary Comparison](#summary-comparison)
+
+---
+
+## Overview
+
+Functional programming languages approach decoupling differently than object-oriented languages. Instead of interface-based inheritance hierarchies, FP languages typically use:
+
+- **Type classes / Protocols** - Ad-hoc polymorphism
+- **Module systems** - Encapsulation and abstraction
+- **Effect systems** - Separating what from how
+- **Higher-order functions** - Functions as first-class dependencies
+
+Each approach has trade-offs in terms of expressiveness, performance, and ease of use.
+
+---
+
+## Haskell
+
+Haskell provides the most sophisticated type-level interface mechanisms among mainstream FP languages.
+
+### Type Classes
+
+Type classes are Haskell's primary interface mechanism, providing ad-hoc polymorphism:
+
+```haskell
+class Comparable a where
+ compare :: a -> a -> Ordering
+
+ -- Default implementation
+ lessThan :: a -> a -> Bool
+ lessThan x y = compare x y == LT
+
+instance Comparable Temperature where
+ compare (Celsius x) (Celsius y) = Prelude.compare x y
+```
+
+**Key advantages over OOP interfaces:**
+- Instances can be defined separately from type definitions
+- Type safety enforced at compile time
+- Laws provide semantic contracts
+
+### Type Class Laws
+
+Laws are semantic contracts that implementations must satisfy:
+
+```haskell
+-- Functor Laws
+-- 1. Identity: fmap id == id
+-- 2. Composition: fmap (f . g) == fmap f . fmap g
+
+-- Monad Laws
+-- 1. Left identity: return a >>= k == k a
+-- 2. Right identity: m >>= return == m
+-- 3. Associativity: (m >>= k) >>= h == m >>= (\x -> k x >>= h)
+```
+
+### Module System
+
+Haskell uses explicit export lists for encapsulation:
+
+```haskell
+module Stack
+ ( Stack -- Export type only, not constructors
+ , empty
+ , push
+ , pop
+ ) where
+
+-- Implementation hidden
+data Stack a = Stack [a]
+
+empty :: Stack a
+empty = Stack []
+```
+
+**Smart constructors** enforce invariants:
+
+```haskell
+module Email (Email, mkEmail, emailText) where
+
+newtype Email = Email Text
+
+mkEmail :: Text -> Either String Email
+mkEmail t
+ | "@" `T.isInfixOf` t = Right (Email t)
+ | otherwise = Left "Invalid email"
+```
+
+### Effect Systems
+
+Haskell has evolved multiple approaches to decoupling effects:
+
+#### MTL (Monad Transformer Library)
+
+```haskell
+processUser :: (MonadReader Config m, MonadState AppState m, MonadError AppError m)
+ => UserId -> m User
+```
+
+#### ReaderT Pattern
+
+```haskell
+data Env = Env
+ { envConfig :: Config
+ , envLogger :: LogLevel -> Text -> IO ()
+ , envDatabase :: Connection
+ }
+
+newtype AppM a = AppM { unAppM :: ReaderT Env IO a }
+```
+
+#### Modern Effect Systems
+
+| Library | Performance | Ergonomics | Use Case |
+|---------|-------------|------------|----------|
+| effectful | Excellent | Good | Production |
+| fused-effects | Excellent | Medium | Performance-critical |
+| polysemy | Poor | Excellent | Prototyping |
+
+### Key Resources
+
+**Papers:**
+- ["How to Make Ad-Hoc Polymorphism Less Ad Hoc"](https://dl.acm.org/doi/pdf/10.1145/75277.75283) - Wadler & Blott (1989)
+- ["Data Types à la Carte"](https://www.cambridge.org/core/journals/journal-of-functional-programming/article/data-types-a-la-carte/14416CB20C4637164EA9F77097909409) - Swierstra (2008)
+- ["Freer Monads, More Extensible Effects"](https://okmij.org/ftp/Haskell/extensible/more.pdf) - Kiselyov & Ishii (2015)
+
+**Blogs:**
+- [Three Layer Haskell Cake](https://www.parsonsmatt.org/2018/03/22/three_layer_haskell_cake.html) - Matt Parsons
+- [The ReaderT Design Pattern](https://www.fpcomplete.com/blog/readert-design-pattern/) - FP Complete
+
+---
+
+## ML Family and Scala
+
+### OCaml/Standard ML Module System
+
+ML languages have the most powerful module systems with signatures and functors:
+
+```ocaml
+(* Signature defines interface *)
+module type STACK = sig
+ type 'a t
+ val empty : 'a t
+ val push : 'a -> 'a t -> 'a t
+ val pop : 'a t -> ('a * 'a t) option
+end
+
+(* Module implements signature *)
+module ListStack : STACK = struct
+ type 'a t = 'a list
+ let empty = []
+ let push x s = x :: s
+ let pop = function
+ | [] -> None
+ | x::xs -> Some (x, xs)
+end
+```
+
+**Functors** enable parameterized modules (dependency injection at the module level):
+
+```ocaml
+module MakeCache (S : STORAGE) : CACHE = struct
+ let get key = S.read key
+ let set key value = S.write key value
+end
+
+module RedisCache = MakeCache(RedisStorage)
+module MemCache = MakeCache(InMemoryStorage)
+```
+
+### Scala Type Classes and Tagless Final
+
+Scala encodes type classes via implicit parameters:
+
+```scala
+trait Serializable[A] {
+ def serialize(a: A): String
+}
+
+implicit val stringSerializable: Serializable[String] =
+ (s: String) => s
+
+def save[A: Serializable](a: A): Unit =
+ println(implicitly[Serializable[A]].serialize(a))
+```
+
+**Tagless Final** provides effect abstraction:
+
+```scala
+trait UserRepository[F[_]] {
+ def getUser(id: UserId): F[Option[User]]
+ def saveUser(user: User): F[Unit]
+}
+
+// Production interpreter
+class PostgresUserRepo extends UserRepository[IO] { ... }
+
+// Test interpreter
+class InMemoryUserRepo extends UserRepository[Id] { ... }
+```
+
+### Key Resources
+
+**Papers:**
+- ["Applicative Programming with Effects"](https://www.cambridge.org/core/journals/journal-of-functional-programming/article/applicative-programming-with-effects/9A3F61D50B56E6D40E779F9FE91C2021) - McBride & Paterson
+- ["Finally Tagless, Partially Evaluated"](https://okmij.org/ftp/tagless-final/JFP.pdf) - Carette, Kiselyov, Shan
+
+**Talks:**
+- Odersky's talks on Scala type system
+- "Constraints Liberate, Liberties Constrain" - Runar Bjarnason
+
+---
+
+## Elixir
+
+Elixir provides two main mechanisms for polymorphism: behaviours and protocols.
+
+### Behaviours
+
+Behaviours define contracts that modules must implement:
+
+```elixir
+defmodule MyApp.Notifier do
+ @callback send_notification(user :: term(), message :: String.t()) ::
+ {:ok, reference()} | {:error, reason :: term()}
+end
+
+defmodule MyApp.EmailNotifier do
+ @behaviour MyApp.Notifier
+
+ @impl MyApp.Notifier
+ def send_notification(user, message) do
+ # Implementation
+ {:ok, make_ref()}
+ end
+end
+```
+
+**Best practice:** Always use `@impl ModuleName` rather than `@impl true`.
+
+### Protocols
+
+Protocols enable polymorphism based on data type:
+
+```elixir
+defprotocol Size do
+ def size(data)
+end
+
+defimpl Size, for: BitString do
+ def size(string), do: byte_size(string)
+end
+
+defimpl Size, for: Map do
+ def size(map), do: map_size(map)
+end
+```
+
+### OTP Patterns
+
+GenServer separates client API from server implementation:
+
+```elixir
+defmodule MyApp.Cache do
+ use GenServer
+
+ # Client API
+ def get(key), do: GenServer.call(__MODULE__, {:get, key})
+ def put(key, value), do: GenServer.cast(__MODULE__, {:put, key, value})
+
+ # Server callbacks
+ @impl GenServer
+ def handle_call({:get, key}, _from, state) do
+ {:reply, Map.get(state, key), state}
+ end
+end
+```
+
+### Key Resources
+
+**Talks:**
+- Jose Valim's ElixirConf Keynotes
+- "Gang of None? Design Patterns in Elixir" - ElixirConf EU 2024
+
+**Documentation:**
+- [OTP Design Principles](https://www.erlang.org/doc/system/design_principles.html)
+- [Elixir Behaviours](https://hexdocs.pm/elixir/behaviours.html)
+- [Elixir Protocols](https://hexdocs.pm/elixir/protocols.html)
+
+---
+
+## F#
+
+F# combines functional programming with .NET's object system.
+
+### Object Expressions
+
+Anonymous interface implementations:
+
+```fsharp
+type ILogger =
+ abstract member Log : string -> unit
+
+let createConsoleLogger prefix =
+ { new ILogger with
+ member _.Log(message) =
+ printfn "[%s] %s" prefix message }
+```
+
+### Type Providers
+
+Compile-time type generation from external schemas:
+
+```fsharp
+type MoviesProvider = JsonProvider<"movies.json">
+
+let movies = MoviesProvider.Load("https://api.example.com/movies")
+for movie in movies do
+ printfn "%s (%d)" movie.Title movie.Year
+```
+
+### Computation Expressions
+
+Uniform syntax for monadic computations:
+
+```fsharp
+let validateUser name email = result {
+ let! validName = validateName name
+ let! validEmail = validateEmail email
+ return { Name = validName; Email = validEmail }
+}
+```
+
+### Signature Files (.fsi)
+
+Define public interfaces separately:
+
+```fsharp
+// Users.fsi
+module Users =
+ type User
+ val create : name:string -> email:string -> Result<User, string>
+ val findById : id:int -> User option
+```
+
+### Key Resources
+
+**Talks:**
+- ["F# Code I Love"](https://www.microsoft.com/en-us/research/video/f-code-i-love-don-syme/) - Don Syme
+- ["The F# Path to Relaxation"](https://www.microsoft.com/en-us/research/video/the-f-path-to-relaxation-don-syme/) - Don Syme
+
+**Articles:**
+- [F# for Fun and Profit - Dependency Injection](https://fsharpforfunandprofit.com/posts/dependencies/)
+- [Designing with Capabilities](https://fsharpforfunandprofit.com/cap/)
+
+---
+
+## Cross-Cutting Patterns
+
+### Dependency Injection in FP
+
+| Pattern | Description | Example |
+|---------|-------------|---------|
+| **Partial Application** | Pre-bind dependencies | `(partial save-user db)` |
+| **Reader Monad** | Environment passing | `ReaderT Env IO a` |
+| **Higher-Order Functions** | Functions as dependencies | `(defn process [get-data] ...)` |
+| **Records of Functions** | Service objects | `{:get-user fn :save-user fn}` |
+
+### Interface Design Principles
+
+1. **Minimal Surface Area** - Small, focused interfaces
+2. **Composition over Inheritance** - Combine small pieces
+3. **Data-Oriented Boundaries** - Pass data, not behavior
+4. **Test at the Seams** - Mock at interface boundaries
+
+### Anti-Patterns to Avoid
+
+- **Over-abstraction** - Don't abstract prematurely
+- **Leaky abstractions** - Implementation details bleeding through
+- **Premature protocol creation** - Start with functions, graduate to protocols
+- **Ignoring data-driven alternatives** - Maps are often better than protocols
+
+---
+
+## Key Papers and Talks
+
+### Foundational Papers
+
+| Paper | Authors | Year | Significance |
+|-------|---------|------|--------------|
+| [How to Make Ad-Hoc Polymorphism Less Ad Hoc](https://dl.acm.org/doi/pdf/10.1145/75277.75283) | Wadler & Blott | 1989 | Introduced type classes |
+| [Data Types à la Carte](https://www.cambridge.org/core/journals/journal-of-functional-programming/article/data-types-a-la-carte/14416CB20C4637164EA9F77097909409) | Swierstra | 2008 | Composable data types |
+| [Freer Monads, More Extensible Effects](https://okmij.org/ftp/Haskell/extensible/more.pdf) | Kiselyov & Ishii | 2015 | Efficient effect systems |
+
+### Conference Talks
+
+- **"The Evolution of Effects"** - Haskell Symposium 2023 Keynote
+- **"Components Just Enough Structure"** - Stuart Sierra, Clojure/West 2014
+- **Jose Valim's ElixirConf Keynotes** - Annual Elixir design updates
+
+---
+
+## Summary Comparison
+
+| Language | Primary Mechanism | Dispatch | Key Strength |
+|----------|-------------------|----------|--------------|
+| **Haskell** | Type classes | Compile-time | Type safety + laws |
+| **OCaml** | Module signatures | Compile-time | Powerful functors |
+| **Scala** | Implicits/type classes | Compile-time | JVM integration |
+| **Elixir** | Behaviours/Protocols | Runtime | OTP patterns |
+| **F#** | Object expressions | Runtime | .NET integration |
+| **Clojure** | Protocols/Multimethods | Runtime | Maximum flexibility |
+
+### When to Use Each Approach
+
+- **Type classes** - When you need compile-time safety and laws
+- **Module systems** - When you need strong encapsulation
+- **Effect systems** - When you need to mock/swap implementations
+- **Protocols** - When you need open extension
+- **Higher-order functions** - When you need maximum simplicity
+
+---
+
+## Next Steps
+
+For Clojure-specific patterns and practices, see:
+- [DECOUPLING_INTERFACES_CLOJURE.md](./DECOUPLING_INTERFACES_CLOJURE.md)
+
+This companion document covers:
+- Protocols and Multimethods in depth
+- Clojure Spec and Malli
+- Component, Integrant, and Mount
+- Testing and mocking patterns
diff --git a/DECOUPLING_INTERFACES_CLOJURE.md b/DECOUPLING_INTERFACES_CLOJURE.md
new file mode 100644
index 00000000000..3e558ce374b
--- /dev/null
+++ b/DECOUPLING_INTERFACES_CLOJURE.md
@@ -0,0 +1,1119 @@
+# Decoupling via Defined Interfaces in Clojure
+
+## A Deep Dive into Clojure's Interface Mechanisms
+
+This document provides comprehensive coverage of Clojure's approaches to decoupling and defining interfaces, including protocols, multimethods, spec, Malli, and component systems.
+
+---
+
+## Table of Contents
+
+1. [Protocols](#protocols)
+2. [Multimethods](#multimethods)
+3. [Protocols vs Multimethods Decision Matrix](#protocols-vs-multimethods-decision-matrix)
+4. [Clojure Spec](#clojure-spec)
+5. [Malli](#malli)
+6. [Component Systems](#component-systems)
+7. [Dependency Injection Patterns](#dependency-injection-patterns)
+8. [Best Practices](#best-practices)
+9. [Resources and References](#resources-and-references)
+
+---
+
+## Protocols
+
+Protocols provide high-performance, type-based polymorphism that solves the expression problem.
+
+### Definition and Basic Usage
+
+```clojure
+(defprotocol Greetable
+ "A protocol for things that can greet"
+ (greet [this] "Return a greeting")
+ (greet-with-name [this name] "Return a personalized greeting"))
+```
+
+### Implementation with defrecord
+
+```clojure
+(defprotocol WereCreature
+ (full-moon-behavior [creature]))
+
+(defrecord WereWolf [name title]
+ WereCreature
+ (full-moon-behavior [this]
+ (str name " will howl and murder")))
+
+;; Usage
+(def jacob (->WereWolf "Jacob" "Alpha"))
+(full-moon-behavior jacob) ; => "Jacob will howl and murder"
+```
+
+### Extension Mechanisms
+
+**extend-type** - One type, multiple protocols:
+
+```clojure
+(extend-type java.lang.String
+ Greetable
+ (greet [this] (str "Hello, I am " this))
+ (greet-with-name [this name] (str this " says hello to " name)))
+```
+
+**extend-protocol** - One protocol, multiple types:
+
+```clojure
+(extend-protocol Greetable
+ java.lang.String
+ (greet [s] (str "String: " s))
+
+ clojure.lang.PersistentVector
+ (greet [v] (str "Vector with " (count v) " items"))
+
+ java.lang.Object
+ (greet [o] "Generic greeting"))
+```
+
+**Reification** - Anonymous implementations:
+
+```clojure
+(defn open-resource [path]
+ (let [resource (load-resource path)]
+ (reify
+ Closeable
+ (close [this] (cleanup resource))
+
+ Object
+ (toString [this] (str "Resource: " path)))))
+```
+
+### When to Use Protocols
+
+- Single dispatch on first argument's type
+- Need maximum performance (~5x faster than multimethods)
+- Defining coherent abstractions (group of related functions)
+- Building libraries for others to extend
+
+---
+
+## Multimethods
+
+Multimethods provide maximum flexibility with dispatch on any function of any arguments.
+
+### Basic Pattern
+
+```clojure
+(defmulti encounter
+ (fn [creature1 creature2]
+ [(:species creature1) (:species creature2)]))
+
+(defmethod encounter [:human :dog]
+ [h d]
+ (str (:name h) " pets " (:name d)))
+
+(defmethod encounter :default
+ [c1 c2]
+ (str (:name c1) " ignores " (:name c2)))
+```
+
+### Hierarchies
+
+```clojure
+(derive ::square ::rect)
+(derive ::rect ::shape)
+
+(defmulti area :shape-type)
+
+(defmethod area ::rect
+ [{:keys [width height]}]
+ (* width height))
+
+;; Square inherits rect implementation
+(area {:shape-type ::square :width 5 :height 5}) ; => 25
+```
+
+### When to Use Multimethods
+
+- Dispatch on multiple arguments
+- Dispatch on values (not types)
+- Need custom hierarchies
+- Dispatch logic is complex or arbitrary
+
+---
+
+## Protocols vs Multimethods Decision Matrix
+
+| Scenario | Recommendation |
+|----------|----------------|
+| Dispatch on first argument's type only | **Protocols** |
+| Performance critical | **Protocols** (~5x faster) |
+| Dispatch on multiple arguments | **Multimethods** |
+| Dispatch on values/keywords | **Multimethods** |
+| Custom hierarchies | **Multimethods** |
+| Maximum REPL flexibility | **Multimethods** |
+
+### Key References
+
+**Talks:**
+- [Rich Hickey on Protocols (InfoQ 2010)](https://www.infoq.com/interviews/hickey-clojure-protocols/)
+- [Clojure's Solutions to the Expression Problem (Strange Loop)](https://www.infoq.com/presentations/Clojure-Expression-Problem/)
+
+**Documentation:**
+- [Clojure Protocols Reference](https://clojure.org/reference/protocols)
+- [Clojure Multimethods Reference](https://clojure.org/reference/multimethods)
+
+**Books:**
+- ["The Joy of Clojure" Chapter 9](https://www.manning.com/books/the-joy-of-clojure-second-edition)
+- ["Clojure for the Brave and True"](https://www.braveclojure.com/multimethods-records-protocols/)
+
+---
+
+## Clojure Spec
+
+Spec provides specifications for data and function contracts with generative testing support.
+
+### Defining Specifications
+
+```clojure
+(require '[clojure.spec.alpha :as s])
+
+;; Simple predicates
+(s/def ::email string?)
+(s/def ::age pos-int?)
+
+;; Map specs
+(s/def ::person
+ (s/keys :req [::name ::age]
+ :opt [::email]))
+
+;; Using specs
+(s/valid? ::person {::name "Alice" ::age 30}) ; => true
+```
+
+### Function Contracts
+
+```clojure
+(defn ranged-rand [start end]
+ (+ start (rand-int (- end start))))
+
+(s/fdef ranged-rand
+ :args (s/and (s/cat :start int? :end int?)
+ #(< (:start %) (:end %)))
+ :ret int?
+ :fn (s/and #(>= (:ret %) (-> % :args :start))
+ #(< (:ret %) (-> % :args :end))))
+```
+
+### Instrumentation
+
+```clojure
+(require '[clojure.spec.test.alpha :as stest])
+
+;; Enable validation during development
+(stest/instrument)
+
+;; Run generative tests
+(stest/check `ranged-rand)
+```
+
+### Key Resources
+
+**Talks:**
+- [Rich Hickey - "Spec-ulation" (Clojure/conj 2016)](https://www.youtube.com/watch?v=oyLBGkS5ICk)
+- [Rich Hickey - "Maybe Not" (Clojure/conj 2018)](https://www.youtube.com/watch?v=YR5WdGrpoug)
+- [Alex Miller - "clojure.spec" (Strange Loop 2016)](https://www.youtube.com/watch?v=VNTQ-M_uSo8)
+
+**Documentation:**
+- [Clojure Spec Guide](https://clojure.org/guides/spec)
+- [Spec 2 Alpha Repository](https://github.com/clojure/spec-alpha2)
+
+---
+
+## Malli
+
+Malli is a high-performance, data-driven schema library from Metosin.
+
+### Schema Definition
+
+```clojure
+(require '[malli.core :as m])
+
+(def Person
+ [:map
+ [:name :string]
+ [:age :int]
+ [:email {:optional true} :string]])
+
+(m/validate Person {:name "Alice" :age 30}) ; => true
+
+;; Compiled validator for performance
+(def valid-person? (m/validator Person))
+```
+
+### Function Schemas
+
+```clojure
+(m/=> plus [:=> [:cat :int :int] :int])
+
+(defn plus [x y] (+ x y))
+
+;; Instrumentation
+(require '[malli.instrument :as mi])
+(mi/instrument!)
+```
+
+### Transformations and Coercion
+
+```clojure
+(require '[malli.transform :as mt])
+
+(m/decode :int "42" (mt/string-transformer))
+; => 42
+
+(m/decode
+ [:map [:x :boolean] [:y :int]]
+ {:x "true" :y "1"}
+ (mt/string-transformer))
+; => {:x true, :y 1}
+```
+
+### Error Messages
+
+```clojure
+(require '[malli.error :as me])
+
+(-> Person
+ (m/explain {:name 123})
+ (me/humanize))
+; => {:name ["should be a string"]}
+```
+
+### Spec vs Malli Comparison
+
+| Feature | Clojure Spec | Malli |
+|---------|--------------|-------|
+| **Syntax** | Macro-based | Data-driven |
+| **Performance** | Good | Excellent (10-100x) |
+| **Transformation** | Not supported | Built-in |
+| **Error Messages** | Basic | Rich with spell-check |
+| **JSON Schema** | Third-party | Built-in |
+| **Global Registry** | Required | Optional |
+
+### When to Choose Malli
+
+- Runtime schema creation
+- Data transformation/coercion
+- Better error messages
+- JSON Schema generation
+- Performance-critical validation
+
+### Key Resources
+
+**Documentation:**
+- [Malli GitHub](https://github.com/metosin/malli)
+- [Function Schemas](https://github.com/metosin/malli/blob/master/docs/function-schemas.md)
+- [Value Transformation](https://github.com/metosin/malli/blob/master/docs/value-transformation.md)
+
+---
+
+## Component Systems
+
+Three major libraries for managing application state and dependencies.
+
+### Component (Stuart Sierra)
+
+**Philosophy:** Explicit dependency injection with lifecycle management.
+
+```clojure
+(require '[com.stuartsierra.component :as component])
+
+(defrecord Database [host port connection]
+ component/Lifecycle
+ (start [this]
+ (assoc this :connection (connect host port)))
+ (stop [this]
+ (.close connection)
+ (assoc this :connection nil)))
+
+(defn create-system [config]
+ (component/system-map
+ :database (new-database (:db-host config) (:db-port config))
+ :app (component/using
+ (new-application)
+ [:database])))
+
+(def system (component/start (create-system config)))
+```
+
+**GitHub:** https://github.com/stuartsierra/component
+
+### Integrant (James Reeves)
+
+**Philosophy:** Data-driven configuration with multimethod lifecycle.
+
+**config.edn:**
+```clojure
+{:adapter/jetty {:port 8080
+ :handler #ig/ref :handler/api}
+ :handler/api {:database #ig/ref :database/postgres}
+ :database/postgres {:host "localhost" :port 5432}}
+```
+
+```clojure
+(require '[integrant.core :as ig])
+
+(defmethod ig/init-key :database/postgres [_ {:keys [host port]}]
+ (create-connection host port))
+
+(defmethod ig/halt-key! :database/postgres [_ conn]
+ (.close conn))
+
+(def system (ig/init config))
+```
+
+**GitHub:** https://github.com/weavejester/integrant
+
+### Mount (Anatoly Polinsky)
+
+**Philosophy:** Namespace-based states with compiler-driven ordering.
+
+```clojure
+(require '[mount.core :refer [defstate]])
+
+(defstate conn
+ :start (create-connection config)
+ :stop (close-connection conn))
+
+;; Start all states
+(mount/start)
+
+;; Testing with substitutes
+(mount/start-with {#'myapp.db/conn mock-conn})
+```
+
+**GitHub:** https://github.com/tolitius/mount
+
+### Comparison Table
+
+| Feature | Component | Integrant | Mount |
+|---------|-----------|-----------|-------|
+| **Configuration** | Code | EDN | In defstate |
+| **Dependencies** | Explicit | Refs | Implicit |
+| **Multiple instances** | Native | Composite keys | Limited |
+| **Learning curve** | Moderate | Moderate | Easy |
+| **Boilerplate** | High | Medium | Low |
+
+### When to Use Each
+
+- **Component** - Large apps, explicit dependencies, DI familiar teams
+- **Integrant** - Data-driven config, Duct framework
+- **Mount** - Incremental adoption, minimal ceremony, REPL-first
+
+### Avoiding the "God Object" Problem
+
+A common criticism of Component/Integrant is that they lead to functions requiring large system objects with all dependencies. Here's how to avoid this anti-pattern:
+
+#### 1. Pass Only What's Needed
+
+```clojure
+;; Bad - god object
+(defn create-user [app-component user-data]
+ (let [db (:database app-component)
+ email (:email-service app-component)]
+ ...))
+
+;; Good - explicit minimal dependencies
+(defn create-user [db email-service user-data]
+ ...)
+```
+
+#### 2. Functional Core, Imperative Shell
+
+Keep business logic pure - it doesn't need dependencies:
+
+```clojure
+;; Pure core - no dependencies needed
+(defn calculate-order-total [order discounts tax-rate]
+ (let [subtotal (reduce + (map :price (:items order)))
+ discount (apply-discounts subtotal discounts)]
+ (* discount (+ 1 tax-rate))))
+
+;; Thin shell - only this needs dependencies
+(defn process-order! [db email-service order]
+ (let [discounts (db/get-discounts db (:user-id order))
+ tax-rate (db/get-tax-rate db (:region order))
+ total (calculate-order-total order discounts tax-rate)]
+ (db/save-order! db (assoc order :total total))
+ (email/send-receipt! email-service order)))
+```
+
+#### 3. Partial Application / Closures
+
+Pre-bind dependencies at system startup:
+
+```clojure
+(defn make-user-creator [db email-service]
+ (fn [user-data]
+ (let [user (db/insert! db :users user-data)]
+ (email/send-welcome! email-service user)
+ user)))
+
+(defrecord UserService [database email-service create-user]
+ component/Lifecycle
+ (start [this]
+ (assoc this :create-user
+ (make-user-creator database email-service)))
+ (stop [this]
+ (assoc this :create-user nil)))
+
+;; Usage - clean single-purpose function
+((:create-user user-service) {:name "Alice" :email "[email protected]"})
+```
+
+#### 4. Protocol Segregation
+
+Define narrow, focused protocols instead of one big component:
+
+```clojure
+;; Good - segregated protocols
+(defprotocol UserRepository
+ (find-user [this id])
+ (save-user [this user]))
+
+(defprotocol EmailSender
+ (send-email [this to subject body]))
+
+;; Functions declare exactly what they need
+(defn notify-user [email-sender user message]
+ (send-email email-sender (:email user) "Notification" message))
+```
+
+#### 5. The "Has" Pattern
+
+Define capability protocols that document requirements:
+
+```clojure
+(defprotocol HasDatabase
+ (get-database [this]))
+
+(defprotocol HasEmailService
+ (get-email-service [this]))
+
+(defrecord AppComponent [database email-service]
+ HasDatabase
+ (get-database [this] database)
+ HasEmailService
+ (get-email-service [this] email-service))
+
+;; Function documents exactly what it needs
+(defn create-user [component user-data]
+ {:pre [(satisfies? HasDatabase component)
+ (satisfies? HasEmailService component)]}
+ (let [db (get-database component)
+ email (get-email-service component)]
+ ...))
+```
+
+#### 6. Layered Architecture
+
+Organize components so each layer has limited scope:
+
+```clojure
+;; Layer 1: Infrastructure
+:database (new-database config)
+:cache (new-cache config)
+
+;; Layer 2: Repositories (only infrastructure deps)
+:user-repo (component/using (new-user-repo) [:database :cache])
+
+;; Layer 3: Services (repos + services)
+:user-service (component/using (new-user-service)
+ [:user-repo :email-client])
+
+;; Layer 4: Handlers (only services they need)
+:user-handler (component/using (new-user-handler) [:user-service])
+```
+
+#### Key Principle
+
+The god object problem comes from conflating "what I depend on" with "what I need for this operation."
+
+| Anti-Pattern | Better Approach |
+|--------------|-----------------|
+| Pass whole system | Pass specific dependencies |
+| One big component | Many small, focused components |
+| All logic in components | Pure functions + thin component shell |
+| Components do everything | Components just wire things together |
+
+### REPL-Friendly Patterns
+
+A common pain point with component systems is needing to start the entire system just to run a quick query. Here's how to maintain REPL productivity:
+
+#### 1. Standalone Resource Constructors
+
+Design components so resources can be created independently:
+
+```clojure
+;; This should work without any system
+(def conn (db/create-connection {:host "localhost" :port 5432}))
+(jdbc/execute! conn ["SELECT * FROM users LIMIT 5"])
+(.close conn)
+
+;; The component just automates lifecycle
+(defrecord Database [config conn]
+ component/Lifecycle
+ (start [this] (assoc this :conn (db/create-connection config)))
+ (stop [this] (.close conn) (assoc this :conn nil)))
+```
+
+#### 2. Dev Namespace with Helpers
+
+```clojure
+(ns user
+ (:require [integrant.repl :refer [go halt reset]]
+ [integrant.repl.state :refer [system]]))
+
+;; Convenience accessors
+(defn db [] (:database/postgres system))
+(defn cache [] (:cache/redis system))
+
+;; Quick connection without system
+(defn dev-db []
+ (db/create-connection (load-dev-config)))
+
+(defmacro with-dev-db [binding & body]
+ `(let [~binding (dev-db)]
+ (try ~@body (finally (.close ~binding)))))
+
+;; At REPL:
+(with-dev-db [conn]
+ (jdbc/execute! conn ["SELECT count(*) FROM users"]))
+```
+
+#### 3. Partial System Startup
+
+Start only what you need:
+
+```clojure
+;; Integrant - start specific keys
+(def db-only (ig/init config [:database/postgres]))
+(jdbc/execute! (:database/postgres db-only) ["SELECT 1"])
+(ig/halt! db-only)
+
+;; Mount - start specific states
+(mount/start #'myapp.db/conn)
+(jdbc/execute! myapp.db/conn ["SELECT 1"])
+```
+
+#### 4. The "Bag of Functions" Pattern
+
+Return pre-bound functions that manage their own resources:
+
+```clojure
+(defn make-user-operations [db-config]
+ (let [get-conn #(db/create-connection db-config)]
+ {:get-user (fn [id]
+ (with-open [conn (get-conn)]
+ (jdbc/get-by-id conn :users id)))
+ :list-users (fn []
+ (with-open [conn (get-conn)]
+ (jdbc/execute! conn ["SELECT * FROM users"])))}))
+
+;; At REPL - no system needed
+(def ops (make-user-operations dev-config))
+((:list-users ops))
+```
+
+#### Key Insight
+
+**Separate resource creation from lifecycle management.** Your constructors should work standalone; the component system just manages them.
+
+### Combining Multimethods with Component Systems
+
+Multimethods and component systems solve different problems and work well together:
+
+| Concern | Solution |
+|---------|----------|
+| **Polymorphism** - Different behavior by type/value | Multimethods / Protocols |
+| **Lifecycle & Dependencies** - Managing stateful resources | Component / Integrant / Mount |
+
+#### When Multimethods Are Enough
+
+If your implementations are **stateless** (use global resources like database pools), pure multimethods work great:
+
+```clojure
+;; No components needed - uses global database access
+(defmulti authenticate (fn [provider _request] provider))
+
+(defmethod authenticate :provider/password
+ [_provider request]
+ ;; t2/select uses global connection pool
+ (let [user (t2/select-one :model/User :email (:email request))]
+ (when (check-password user (:password request))
+ {:success? true :user-id (:id user)})))
+```
+
+#### When to Add Components
+
+Add component management when providers need:
+- **Connection pools** - LDAP, HTTP clients
+- **Caching** - Provider-specific caches
+- **Background processes** - Token refresh
+- **Expensive initialization** - SAML metadata, keys
+
+#### Pattern 1: Pass Dependencies Through Context
+
+Multimethod receives dependencies via context map:
+
+```clojure
+(defmulti authenticate
+ (fn [provider _context _request] provider))
+
+;; Simple providers ignore context
+(defmethod authenticate :provider/password
+ [_provider _context request]
+ (password-authenticate request))
+
+;; Complex providers use context
+(defmethod authenticate :provider/ldap
+ [_provider {:keys [ldap-client]} request]
+ (ldap-authenticate ldap-client request))
+
+;; Component creates context
+(defrecord AuthService [ldap-client saml-config database]
+ component/Lifecycle
+ (start [this]
+ (assoc this :context {:ldap-client ldap-client
+ :saml-config saml-config
+ :database database}))
+ (stop [this] this))
+
+;; Public API with default context
+(defn authenticate-user
+ ([provider request]
+ (authenticate provider default-context request))
+ ([provider context request]
+ (authenticate provider context request)))
+```
+
+#### Pattern 2: Component Registry of Providers
+
+Each provider is a component implementing a protocol:
+
+```clojure
+(defprotocol AuthProvider
+ (authenticate [this request]))
+
+(defrecord LdapProvider [ldap-client config]
+ component/Lifecycle
+ (start [this]
+ (assoc this :ldap-client (ldap/connect config)))
+ (stop [this]
+ (ldap/disconnect ldap-client)
+ this)
+
+ AuthProvider
+ (authenticate [this request]
+ (ldap-authenticate ldap-client request)))
+
+;; Registry holds all providers
+(defrecord AuthRegistry [ldap-provider saml-provider]
+ component/Lifecycle
+ (start [this]
+ (assoc this :providers
+ {:provider/ldap ldap-provider
+ :provider/saml saml-provider}))
+ (stop [this] this))
+
+;; Lookup and delegate
+(defn authenticate-user [registry provider-key request]
+ (if-let [provider (get-in registry [:providers provider-key])]
+ (authenticate provider request)
+ (throw (ex-info "Unknown provider" {:provider provider-key}))))
+```
+
+#### Recommended Hybrid Approach
+
+Use multimethods for dispatch with optional context for dependencies:
+
+```clojure
+(defmulti authenticate
+ (fn [provider _context _request] provider))
+
+;; Stateless - ignores context
+(defmethod authenticate :provider/password
+ [_provider _context request]
+ (password-authenticate request))
+
+;; Stateful - uses context
+(defmethod authenticate :provider/ldap
+ [_provider {:keys [ldap-pool]} request]
+ (ldap-authenticate ldap-pool request))
+
+;; Default context for production
+(def default-auth-context
+ {:ldap-pool @ldap-connection-pool
+ :http-client @http-client})
+
+;; Testable public API
+(defn authenticate-user
+ ([provider request]
+ (authenticate-user provider default-auth-context request))
+ ([provider context request]
+ (authenticate provider context request)))
+```
+
+This gives you:
+- Backward compatibility with stateless providers
+- Dependency injection for providers that need it
+- Testability (pass mock context)
+- No forced component system overhead
+
+### Key Resources
+
+**Talks:**
+- [Stuart Sierra - Components Just Enough Structure](https://www.youtube.com/watch?v=av9Xi6CNqq4) (Clojure/West 2014)
+
+**Books:**
+- [Clojure Applied](https://pragprog.com/titles/vmclojeco/clojure-applied/) - Chapters 6 & 7
+
+**Comparisons:**
+- [Yogthos - Contrasting Component and Mount](https://yogthos.net/posts/2016-01-19-ContrastingComponentAndMount.html)
+- [Mount - Differences from Component](https://github.com/tolitius/mount/blob/master/doc/differences-from-component.md)
+
+---
+
+## Dependency Injection Patterns
+
+### Pattern 1: Higher-Order Functions
+
+```clojure
+(defn calculate-total [get-price items]
+ (reduce + (map get-price items)))
+
+;; Production
+(calculate-total db/get-price items)
+
+;; Test
+(calculate-total (constantly 10.0) items)
+```
+
+### Pattern 2: Configuration Maps
+
+```clojure
+(defn process-order [{:keys [db-conn email-service]} order]
+ (-> order
+ (save-to-db db-conn)
+ (send-confirmation email-service)))
+```
+
+### Pattern 3: Protocols and Records
+
+```clojure
+(defprotocol EmailService
+ (send-email [this to subject body]))
+
+(defrecord SmtpService [config]
+ EmailService
+ (send-email [this to subject body]
+ (smtp/send config to subject body)))
+
+(defrecord MockEmailService [sent-emails]
+ EmailService
+ (send-email [this to subject body]
+ (swap! sent-emails conj {:to to :subject subject})))
+```
+
+### Pattern 4: Functional Core, Imperative Shell
+
+```clojure
+;; Pure core
+(defn calculate-discount [user order]
+ (cond
+ (:premium user) (* 0.8 (:total order))
+ (:member user) (* 0.9 (:total order))
+ :else (:total order)))
+
+;; Impure shell
+(defn process-order! [db user-id order-id]
+ (let [user (db/get-user db user-id)
+ order (db/get-order db order-id)
+ discounted (calculate-discount user order)] ; Pure!
+ (db/update-order! db order-id {:total discounted})))
+```
+
+---
+
+## Toucan Lifecycle Hooks and Model Coupling
+
+Toucan2 lifecycle hooks (`define-before-insert`, `define-after-update`, etc.) are a form of implicit coupling that deserves special consideration.
+
+### The Trade-off
+
+Lifecycle hooks provide **automatic enforcement** - you can't forget to call them - but create **hidden dependencies** that are hard to trace and test.
+
+### When Hooks Work Well
+
+**Use hooks for single-model concerns:**
+
+```clojure
+;; Good - validation and normalization
+(t2/define-before-insert :model/User
+ [user]
+ (-> user
+ (update :email u/lower-case-en)
+ (assoc :date_joined (t/offset-date-time))))
+
+;; Good - simple defaults
+(t2/define-before-insert :model/Card
+ [card]
+ (assoc card :creator_id api/*current-user-id*))
+```
+
+### When Hooks Cause Problems
+
+**Avoid hooks for complex multi-model operations:**
+
+```clojure
+;; Problematic - hidden cross-model coupling
+(t2/define-after-insert :model/User
+ [user]
+ ;; Caller has no idea this happens
+ (t2/insert! :model/AuthIdentity {:user_id (:id user) ...})
+ (t2/insert! :model/PermissionsGroupMembership {:user_id (:id user) ...})
+ (events/publish-event! :event/user-joined {:user user})
+ user)
+```
+
+**Problems with this pattern:**
+- Callers don't know what else happens on insert
+- Hard to create a "bare" user for testing
+- Transaction boundaries are unclear
+- Can cause N+1 queries or circular dependencies
+
+### Better Alternatives
+
+#### 1. Service Functions for Complex Operations
+
+```clojure
+;; Explicit orchestration - caller knows what happens
+(defn create-user!
+ "Create a user with all related entities."
+ [user-data]
+ (t2/with-transaction [_conn]
+ (let [user (t2/insert-returning-instance! :model/User user-data)]
+ (t2/insert! :model/AuthIdentity {:user_id (:id user) ...})
+ (add-to-default-group! user)
+ (events/publish-event! :event/user-joined {:user user})
+ user)))
+
+;; Keep the hook simple
+(t2/define-before-insert :model/User
+ [user]
+ (-> user
+ (update :email u/lower-case-en)
+ (assoc :date_joined (t/offset-date-time))))
+```
+
+#### 2. Events for Async Side Effects
+
+```clojure
+;; Loose coupling via events
+(t2/define-after-insert :model/Card
+ [card]
+ (events/publish-event! :event/card-create {:object card})
+ card)
+
+;; Handlers are separate and testable
+(methodical/defmethod events/publish-event! :event/card-create
+ [_topic {:keys [object]}]
+ (search/update-index! object)
+ (audit/log-create! object))
+```
+
+#### 3. Derived Hooks for Cross-Cutting Concerns
+
+```clojure
+;; Metabase pattern - declare capabilities
+(methodical/defmethod t2/model-hooks :model/Card
+ [_model]
+ [:hook/timestamped? :hook/entity-id :hook/search-index])
+
+;; Single implementation for all models with this hook
+(t2/define-before-insert :hook/timestamped?
+ [instance]
+ (assoc instance
+ :created_at (t/offset-date-time)
+ :updated_at (t/offset-date-time)))
+```
+
+### Coupling Analysis: When to Use What
+
+| Pattern | Use For | Coupling | Testability |
+|---------|---------|----------|-------------|
+| **Lifecycle Hooks** | Validation, defaults, single-model transforms | Tight | Hard |
+| **Service Functions** | Complex multi-model operations | Medium | Good |
+| **Events** | Async side effects (indexing, notifications) | Loose | Good |
+| **Derived Hooks** | Cross-cutting concerns (timestamps, IDs) | Medium | Good |
+
+### Problem Patterns to Avoid
+
+1. **Hidden N+1 queries** - Hook that iterates over related entities
+2. **Circular dependencies** - Model A hook calls Model B which calls Model A
+3. **Transaction ambiguity** - Unclear if hook runs in same transaction
+4. **Requiring-resolve hacks** - Sign of circular dependency
+
+### Recommendations
+
+1. **Keep hooks simple** - Validation and single-model transforms only
+2. **Use service functions** - For anything that touches multiple models
+3. **Use events** - For side effects that don't need to be transactional
+4. **Document what happens** - If a hook does something surprising, document it
+
+```clojure
+;; Good documentation
+(t2/define-after-insert :model/Database
+ [database]
+ ;; NOTE: This creates DataPermissions entries for ALL groups
+ ;; and schedules sync tasks. See `create-database!` for the
+ ;; full orchestration.
+ (create-default-permissions! database)
+ (schedule-sync-tasks! database)
+ database)
+```
+
+The goal is **explicit over implicit** - callers should be able to understand what happens when they call `t2/insert!` vs `create-database!`.
+
+---
+
+## Best Practices
+
+### Interface Design
+
+1. **Keep interfaces small** - Single-method protocols are easier to extend
+2. **Start with maps** - Graduate to records when needed
+3. **Spec at boundaries** - Not everywhere, just system edges
+4. **Program to abstractions** - Not implementations
+
+### Protocol Usage
+
+```clojure
+;; Good - small, focused
+(defprotocol Nameable (get-name [x]))
+(defprotocol Describable (get-description [x]))
+
+;; Avoid - too broad
+(defprotocol Entity
+ (get-name [x])
+ (get-description [x])
+ (get-id [x])
+ (serialize [x]))
+```
+
+### Multimethod Usage
+
+```clojure
+;; Good - simple dispatch
+(defmulti process :type)
+
+;; Avoid - expensive dispatch function
+(defmulti process (fn [x] (expensive-calculation x)))
+
+;; Always provide default
+(defmethod process :default [x]
+ (throw (ex-info "Unknown type" {:input x})))
+```
+
+### Spec/Malli Usage
+
+```clojure
+;; Good - at API boundary
+(s/fdef api-handler :args (s/cat :request ::api-request))
+
+;; Avoid - internal implementation
+;; (s/fdef private-helper ...)
+
+;; Don't use spec for coercion (use Malli instead)
+```
+
+### Testing
+
+```clojure
+;; Test with protocol implementations
+(deftest service-test
+ (let [mock-db (->MockDatabase (atom {}))]
+ (is (valid? (process-data mock-db input)))))
+
+;; Test with spec/malli generators
+(defspec round-trip 100
+ (prop/for-all [person (mg/generator Person)]
+ (= person (-> person serialize deserialize))))
+```
+
+---
+
+## Resources and References
+
+### Official Documentation
+
+- [Clojure Protocols](https://clojure.org/reference/protocols)
+- [Clojure Multimethods](https://clojure.org/reference/multimethods)
+- [Clojure Spec Guide](https://clojure.org/guides/spec)
+
+### Video Talks
+
+| Talk | Speaker | Event | Link |
+|------|---------|-------|------|
+| Protocols and Clojure 1.3 | Rich Hickey | InfoQ 2010 | [Link](https://www.infoq.com/interviews/hickey-clojure-protocols/) |
+| Spec-ulation | Rich Hickey | Clojure/conj 2016 | [Link](https://www.youtube.com/watch?v=oyLBGkS5ICk) |
+| Maybe Not | Rich Hickey | Clojure/conj 2018 | [Link](https://www.youtube.com/watch?v=YR5WdGrpoug) |
+| clojure.spec | Alex Miller | Strange Loop 2016 | [Link](https://www.youtube.com/watch?v=VNTQ-M_uSo8) |
+| Components | Stuart Sierra | Clojure/West 2014 | [Link](https://www.youtube.com/watch?v=av9Xi6CNqq4) |
+
+### Books
+
+- [The Joy of Clojure](https://www.manning.com/books/the-joy-of-clojure-second-edition) - Chapter 9
+- [Clojure for the Brave and True](https://www.braveclojure.com/multimethods-records-protocols/)
+- [Clojure Applied](https://pragprog.com/titles/vmclojeco/clojure-applied/) - Chapters 6 & 7
+
+### Libraries
+
+| Library | Purpose | Link |
+|---------|---------|------|
+| Component | Dependency injection | [GitHub](https://github.com/stuartsierra/component) |
+| Integrant | Data-driven systems | [GitHub](https://github.com/weavejester/integrant) |
+| Mount | Namespace states | [GitHub](https://github.com/tolitius/mount) |
+| Malli | Data schemas | [GitHub](https://github.com/metosin/malli) |
+| Spec-tools | Spec utilities | [GitHub](https://github.com/metosin/spec-tools) |
+| Expound | Better spec errors | [GitHub](https://github.com/bhb/expound) |
+
+### Blog Posts
+
+- [Polymorphic Performance](https://insideclojure.org/2015/04/27/poly-perf/) - Alex Miller
+- [Contrasting Component and Mount](https://yogthos.net/posts/2016-01-19-ContrastingComponentAndMount.html) - Yogthos
+- [Data Validation in Clojure](https://ostash.dev/posts/2021-08-22-data-validation-in-clojure/)
+
+---
+
+## Summary
+
+Clojure provides multiple mechanisms for decoupling:
+
+| Mechanism | Use Case | Performance |
+|-----------|----------|-------------|
+| **Protocols** | Type-based dispatch | Excellent |
+| **Multimethods** | Flexible dispatch | Good |
+| **Spec** | Contracts + testing | Good |
+| **Malli** | Schemas + transforms | Excellent |
+| **Component/Integrant/Mount** | System management | N/A |
+
+**Recommendations:**
+
+1. Start with **higher-order functions** for simple DI
+2. Use **protocols** for type-based polymorphism
+3. Use **multimethods** for complex dispatch logic
+4. Use **Malli** for validation with transformation
+5. Use **Spec** for function contracts and generative testing
+6. Choose **Component/Integrant/Mount** based on team preference and app complexity
+
+The key principle: **start simple, add abstraction when needed**. Maps and functions go a long way before you need protocols or multimethods.
diff --git a/OIDC_SECURITY_ANALYSIS.md b/OIDC_SECURITY_ANALYSIS.md
new file mode 100644
index 00000000000..2639aa48277
--- /dev/null
+++ b/OIDC_SECURITY_ANALYSIS.md
@@ -0,0 +1,420 @@
+# OIDC Implementation Security Analysis
+
+**Date:** 2026-01-19
+**Scope:** `metabase.sso.oidc.*` and `metabase.sso.providers.oidc`
+
+## Executive Summary
+
+This document provides a security analysis of the OIDC (OpenID Connect) authentication implementation in Metabase. The implementation includes several strong security measures but has areas that require attention, particularly around redirect URL validation and caching mechanisms.
+
+---
+
+## High Severity Issues
+
+### 1. Open Redirect Vulnerability - FIXED
+
+**Locations:**
+- `src/metabase/sso/oidc/state.clj:70-111` (validation functions)
+- `src/metabase/sso/oidc/state.clj:136` (validation call in create-oidc-state)
+
+**Description:**
+
+~~The redirect URL stored in the encrypted state cookie is used directly after authentication completes without validation.~~
+
+**FIXED:** The `create-oidc-state` function now validates redirect URLs before storing them. The validation:
+- Accepts relative URLs starting with `/` (but rejects protocol-relative URLs like `//evil.com`)
+- Accepts absolute URLs only if they have the same origin as the configured `site-url`
+- Rejects all other URLs including `javascript:`, `data:`, and external domains
+
+**Implementation:**
+
+```clojure
+;; New validation functions added to state.clj:
+(defn valid-redirect-url?
+ "Validates that a redirect URL is safe for use after OIDC authentication."
+ ([redirect-url] (valid-redirect-url? redirect-url (system.settings/site-url)))
+ ([redirect-url site-url]
+ (cond
+ (or (nil? redirect-url) (str/blank? redirect-url)) false
+ (relative-url? redirect-url) true ; /path is OK, //evil.com is not
+ :else (same-origin? redirect-url site-url))))
+
+;; Validation is called in create-oidc-state:
+(validate-redirect-url! redirect) ; throws on invalid URL
+```
+
+**Tests:** See `test/metabase/sso/oidc/state_test.clj`:
+- `valid-redirect-url-test` - comprehensive URL validation tests
+- `create-oidc-state-rejects-invalid-redirects-test` - integration tests for rejection
+- `wrap-oidc-redirect-test` - tests open redirect protection in high-level API
+
+---
+
+## Medium Severity Issues
+
+### 2. JWKS Cache Has No Expiration - FIXED
+
+**Location:** `src/metabase/sso/oidc/tokens.clj:11-45`
+
+**Description:**
+
+~~The JWKS (JSON Web Key Set) cache is implemented as a simple atom with no TTL or size limits.~~
+
+**FIXED:** The JWKS cache now includes TTL-based expiration:
+
+- Cache entries expire after 1 hour (3600000 ms)
+- Expired entries trigger automatic re-fetch from the IdP
+- New `invalidate-jwks-cache!` function allows manual cache invalidation for specific URIs
+
+**Implementation:**
+
+```clojure
+(def ^:private jwks-cache
+ "Cache of JWKS with TTL. Map of JWKS URI string -> {:jwks map :fetched-at timestamp}."
+ (atom {}))
+
+(def ^:private jwks-cache-ttl-ms
+ "JWKS cache TTL: 1 hour."
+ 3600000)
+
+(defn- cache-expired? [cache-entry]
+ (or (nil? cache-entry)
+ (nil? (:fetched-at cache-entry))
+ (> (- (System/currentTimeMillis) (:fetched-at cache-entry))
+ jwks-cache-ttl-ms)))
+
+(defn invalidate-jwks-cache! [jwks-uri]
+ "Invalidate cached JWKS for a specific URI."
+ (swap! jwks-cache dissoc jwks-uri))
+```
+
+**Tests:** See `test/metabase/sso/oidc/tokens_test.clj`:
+- `jwks-cache-uses-cached-entry-when-fresh-test`
+- `jwks-cache-refetches-when-expired-test`
+- `invalidate-jwks-cache-test`
+
+---
+
+### 3. Discovery Document Cache Has No Expiration - FIXED
+
+**Location:** `src/metabase/sso/oidc/discovery.clj:11-14`
+
+**Description:**
+
+~~Similar to the JWKS cache, the discovery document cache has no TTL.~~
+
+**FIXED:** The discovery document cache now includes TTL-based expiration:
+
+- Cache entries expire after 24 hours (86400000 ms)
+- Expired entries trigger automatic re-fetch from the IdP
+- New `invalidate-cache!` function allows manual cache invalidation for specific issuers
+- Cache refresh events are logged for audit purposes
+
+**Implementation:**
+
+```clojure
+(def ^:private discovery-cache
+ "Cache of discovery documents with TTL.
+ Map of issuer URL string -> {:document discovery-map :fetched-at timestamp}."
+ (atom {}))
+
+(def ^:private discovery-cache-ttl-ms
+ "Discovery document cache TTL: 24 hours."
+ 86400000)
+
+(defn- cache-expired? [cache-entry]
+ (or (nil? cache-entry)
+ (nil? (:fetched-at cache-entry))
+ (> (- (System/currentTimeMillis) (:fetched-at cache-entry))
+ discovery-cache-ttl-ms)))
+
+(defn invalidate-cache! [issuer]
+ "Invalidate cached discovery document for a specific issuer."
+ (let [normalized (normalize-issuer issuer)]
+ (swap! discovery-cache dissoc normalized)
+ (log/infof "Invalidated discovery cache for issuer %s" normalized)))
+```
+
+**Tests:** See `test/metabase/sso/oidc/discovery_test.clj`:
+- `discovery-cache-uses-cached-entry-when-fresh-test`
+- `discovery-cache-refetches-when-expired-test`
+- `invalidate-discovery-cache-test`
+- `invalidate-discovery-cache-with-trailing-slash-test`
+
+---
+
+### 4. Missing SSRF Protection on External URL Fetches
+
+**Locations:**
+- `src/metabase/sso/oidc/discovery.clj:27-46`
+- `src/metabase/sso/oidc/tokens.clj:20-33`
+
+**Description:**
+
+When fetching discovery documents and JWKS, there's no validation preventing requests to internal network addresses:
+
+```clojure
+;; discovery.clj
+(http/get url {:as :json ...})
+
+;; tokens.clj
+(http/get jwks-uri {:as :json ...})
+```
+
+**Security Impact:**
+
+A malicious or compromised issuer configuration could:
+- Make requests to internal services (SSRF)
+- Probe internal network topology
+- Access cloud metadata endpoints (e.g., `169.254.169.254`)
+
+**Recommendation:**
+
+Use the existing `metabase.util.http/valid-host?` function:
+
+```clojure
+(require '[metabase.util.http :as u.http])
+
+(defn- fetch-discovery-document [issuer]
+ (let [url (discovery-url issuer)]
+ (when-not (u.http/valid-host? :external-only url)
+ (throw (ex-info "Invalid issuer URL: internal addresses not allowed" {:url url})))
+ ;; ... existing fetch logic
+ ))
+```
+
+---
+
+### 5. No PKCE Support
+
+**Locations:**
+- `src/metabase/sso/providers/oidc.clj:54-83`
+- `src/metabase/sso/oidc/common.clj:29-49`
+
+**Description:**
+
+PKCE (Proof Key for Code Exchange) is not implemented. The authorization flow uses only the basic authorization code flow without code challenge/verifier.
+
+**Security Impact:**
+
+- Authorization code interception attacks are possible if an attacker can observe the callback URL
+- OAuth 2.1 and current OIDC best practices recommend PKCE for all clients
+- Some IdPs may require PKCE
+
+**Recommendation:**
+
+Add PKCE support:
+
+```clojure
+;; In common.clj
+(defn generate-code-verifier
+ "Generate a cryptographically random code verifier for PKCE."
+ []
+ (-> (nonce/random-bytes 32) codecs/bytes->b64-str
+ (str/replace #"\+" "-")
+ (str/replace #"/" "_")
+ (str/replace #"=" "")))
+
+(defn generate-code-challenge
+ "Generate S256 code challenge from verifier."
+ [verifier]
+ (-> verifier
+ codecs/str->bytes
+ (buddy.core.hash/sha256)
+ codecs/bytes->b64-str
+ (str/replace #"\+" "-")
+ (str/replace #"/" "_")
+ (str/replace #"=" "")))
+
+;; Add to authorization URL params:
+{:code_challenge (generate-code-challenge code-verifier)
+ :code_challenge_method "S256"}
+
+;; Store code_verifier in state cookie, include in token exchange
+```
+
+---
+
+### 6. Hardcoded RS256 Algorithm
+
+**Location:** `src/metabase/sso/oidc/tokens.clj:74`
+
+**Description:**
+
+The JWT verification algorithm is hardcoded to RS256:
+
+```clojure
+(jwt/unsign token public-key {:alg :rs256})
+```
+
+**Security Impact:**
+
+- Some IdPs use different algorithms (RS384, RS512, ES256)
+- The JWK's `alg` field is not validated against the token's algorithm
+- Could lead to interoperability issues or security misconfigurations
+
+**Recommendation:**
+
+```clojure
+(def ^:private allowed-algorithms
+ "Allowlist of acceptable JWT signing algorithms."
+ #{:rs256 :rs384 :rs512 :es256 :es384 :es512})
+
+(defn- get-algorithm [key-data]
+ (let [alg (-> key-data :alg str/lower-case keyword)]
+ (when-not (contains? allowed-algorithms alg)
+ (throw (ex-info "Unsupported JWT algorithm" {:algorithm alg})))
+ alg))
+
+;; In verify-signature:
+(let [alg (get-algorithm key-data)
+ public-key (keys/jwk->public-key key-data)]
+ (jwt/unsign token public-key {:alg alg}))
+```
+
+---
+
+## Low Severity Issues
+
+### 7. Missing Token Time Validations
+
+**Location:** `src/metabase/sso/oidc/tokens.clj:128-166`
+
+**Description:**
+
+Token validation checks `exp` (expiry) but not:
+- `iat` (issued at) - tokens issued far in the past could indicate replay
+- `nbf` (not before) - respecting intended activation time
+- `auth_time` - for session freshness requirements
+
+**Recommendation:**
+
+```clojure
+(defn validate-issued-at
+ "Validate that the token was issued recently (within acceptable window)."
+ [claims max-age-seconds]
+ (when-let [iat (:iat claims)]
+ (let [now (quot (System/currentTimeMillis) 1000)
+ age (- now iat)]
+ (and (>= age 0) (<= age max-age-seconds)))))
+
+(defn validate-not-before
+ "Validate that the token is active (current time >= nbf)."
+ [claims]
+ (if-let [nbf (:nbf claims)]
+ (let [now (quot (System/currentTimeMillis) 1000)]
+ (>= now nbf))
+ true)) ; nbf is optional
+```
+
+---
+
+### 8. Missing `azp` Claim Validation
+
+**Location:** `src/metabase/sso/oidc/tokens.clj:102-115`
+
+**Description:**
+
+When the audience (`aud`) claim is an array with multiple values, the OIDC spec recommends validating the `azp` (authorized party) claim equals the client_id.
+
+**Recommendation:**
+
+```clojure
+(defn validate-authorized-party
+ "Validate azp claim when aud is multi-valued."
+ [claims expected-client-id]
+ (let [aud (:aud claims)
+ azp (:azp claims)]
+ (if (and (sequential? aud) (> (count aud) 1))
+ (= azp expected-client-id)
+ true))) ; azp validation only required for multi-valued aud
+```
+
+---
+
+### 9. Potential Sensitive Data in Error Logs
+
+**Location:** `src/metabase/sso/providers/oidc.clj:79`
+
+**Description:**
+
+Token exchange error responses are logged in their entirety:
+
+```clojure
+(log/errorf "Token exchange failed: %s" (:body response))
+```
+
+Some IdP implementations may include sensitive information in error responses.
+
+**Recommendation:**
+
+Log only sanitized error information:
+
+```clojure
+(log/errorf "Token exchange failed: error=%s description=%s"
+ (get-in response [:body :error])
+ (get-in response [:body :error_description]))
+```
+
+---
+
+### 10. SameSite=Lax Cookie Consideration
+
+**Location:** `src/metabase/sso/oidc/state.clj:141-146`
+
+**Description:**
+
+The state cookie uses `SameSite=Lax`, which is required for OIDC flows (the cookie must be sent on the redirect back from the IdP). However, this means the cookie will be sent on top-level GET navigations from other sites.
+
+**Impact:**
+
+This is acceptable and necessary for OIDC but should be documented. The 10-minute TTL and cryptographic state validation mitigate most risks.
+
+**Recommendation:**
+
+- Document this trade-off in security documentation
+- Ensure state TTL remains short (current 10 minutes is appropriate)
+- Consider reducing TTL further if possible (e.g., 5 minutes)
+
+---
+
+## Positive Security Findings
+
+The implementation includes several strong security measures:
+
+| Feature | Implementation | Assessment |
+|---------|---------------|------------|
+| State Encryption | AES256-CBC-HMAC-SHA512 | Strong |
+| Key Derivation | PBKDF2+SHA512 with 100k iterations | Strong |
+| CSRF Protection | Cryptographically random state parameter | Correct |
+| Nonce Validation | Prevents token replay | Correct |
+| State TTL | 10-minute expiration | Appropriate |
+| Issuer Validation | Exact match required | Correct |
+| Audience Validation | Supports string and array formats | Correct |
+| Cookie Security | HttpOnly, Secure (on HTTPS), path-restricted | Correct |
+| Encryption Requirement | Enforces `MB_ENCRYPTION_SECRET_KEY` | Good |
+
+---
+
+## Remediation Priority
+
+| Priority | Issue | Effort | Status |
+|----------|-------|--------|--------|
+| 1 | Open Redirect Vulnerability | Low | **FIXED** |
+| 2 | JWKS Cache Expiration | Medium | **FIXED** |
+| 3 | SSRF Protection | Low | Open |
+| 4 | Discovery Cache Expiration | Medium | **FIXED** |
+| 5 | PKCE Support | Medium | Open |
+| 6 | Algorithm Flexibility | Low | Open |
+| 7 | Token Time Validations | Low | Open |
+| 8 | azp Claim Validation | Low | Open |
+| 9 | Log Sanitization | Low | Open |
+
+---
+
+## References
+
+- [OpenID Connect Core 1.0](https://openid.net/specs/openid-connect-core-1_0.html)
+- [OAuth 2.0 Security Best Current Practice](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-security-topics)
+- [PKCE RFC 7636](https://datatracker.ietf.org/doc/html/rfc7636)
+- [OWASP Authentication Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)
diff --git a/PLAN_frontend_snippet_dirty_state.md b/PLAN_frontend_snippet_dirty_state.md
new file mode 100644
index 00000000000..9a5fe608bc7
--- /dev/null
+++ b/PLAN_frontend_snippet_dirty_state.md
@@ -0,0 +1,109 @@
+# Plan: Frontend Snippet Dirty State Refresh
+
+## Overview
+
+Implement declarative support for refreshing dirty status when snippets are created/updated/deleted in the frontend. This ensures snippet dirty state is properly reflected as library dirty state.
+
+## Background
+
+### Current State
+- Backend support for snippet remote sync was added in commit `d7bfe1a05af`
+- Snippets live in collections with `namespace: "snippets"` - NOT within the Library collection hierarchy
+- The Library section in the UI displays snippets, so snippet dirty state should show as Library dirty
+
+### What Was Missing
+1. **Snippets not in `MODEL_MUTATION_CONFIGS`** - Snippet mutations didn't trigger dirty state invalidation
+2. **`useHasLibraryDirtyChanges` didn't check snippets** - It only checked the Library collection tree, not snippets-namespace collections
+
+## Implementation
+
+### Change 1: Add Snippet to Model Mutation Configs
+
+**File**: `enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/model-configs.ts`
+
+Added import for `snippetApi` and snippet config to `MODEL_MUTATION_CONFIGS`:
+
+```typescript
+import { snippetApi } from "metabase/api/snippet";
+
+// In MODEL_MUTATION_CONFIGS array:
+{
+ modelType: "snippet",
+ createEndpoints: [snippetApi.endpoints.createSnippet.matchFulfilled],
+ updateEndpoints: [snippetApi.endpoints.updateSnippet.matchFulfilled],
+ invalidation: { type: InvalidationType.Always },
+},
+```
+
+This ensures:
+- When `createSnippet` or `updateSnippet` mutations complete, dirty state tags are invalidated
+- Snippet archiving (deletion) uses `updateSnippet` with `archived: true`, so it's covered
+
+### Change 2: Update Library Dirty Changes Hook
+
+**File**: `enterprise/frontend/src/metabase-enterprise/remote_sync/hooks/use-has-library-dirty-changes.ts`
+
+Updated to also check for dirty snippets and snippets-namespace collections:
+
+1. Added fetch for snippets-namespace collections:
+ ```typescript
+ const { data: snippetsCollections = [] } = useListCollectionsTreeQuery(
+ { namespace: "snippets" },
+ { skip: !isGitSyncVisible },
+ );
+ ```
+
+2. Added check for dirty snippets before checking Library tree:
+ ```typescript
+ const hasSnippetDirtyChanges = dirty.some((entity) => {
+ if (entity.model === "snippet") {
+ return true;
+ }
+ if (
+ entity.model === "collection" &&
+ numericSnippetCollectionIds.has(entity.id)
+ ) {
+ return true;
+ }
+ return false;
+ });
+
+ if (hasSnippetDirtyChanges) {
+ return true;
+ }
+ ```
+
+## Files Changed
+
+| File | Change |
+|------|--------|
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/model-configs.ts` | Add snippetApi import and snippet config |
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/hooks/use-has-library-dirty-changes.ts` | Add snippet dirty state detection |
+
+## How It Works
+
+1. User creates/updates/archives a snippet
+2. RTK Query mutation completes (`createSnippet.matchFulfilled` or `updateSnippet.matchFulfilled`)
+3. Listener middleware detects the mutation via `MODEL_MUTATION_CONFIGS`
+4. Middleware invalidates `REMOTE_SYNC_INVALIDATION_TAGS` (including `collection-dirty-entities`)
+5. `useRemoteSyncDirtyState` refetches dirty entities from `/api/ee/remote-sync/dirty`
+6. `useHasLibraryDirtyChanges` checks if any dirty entity is:
+ - A snippet (`model === "snippet"`)
+ - A collection in the snippets namespace
+ - An entity in the Library collection tree
+7. If any match, Library shows as dirty in the UI
+
+## Testing
+
+### Manual Testing Steps
+1. Enable remote sync on Library collection
+2. Create a new snippet in the SQL editor or Data Studio
+3. Verify Library shows dirty indicator
+4. Update an existing snippet
+5. Verify Library shows dirty indicator
+6. Archive a snippet
+7. Verify Library shows dirty indicator
+8. Export changes
+9. Verify dirty indicators clear
+
+## Status: IMPLEMENTED
diff --git a/PLAN_memoize_default_secret_key.md b/PLAN_memoize_default_secret_key.md
new file mode 100644
index 00000000000..001954c505e
--- /dev/null
+++ b/PLAN_memoize_default_secret_key.md
@@ -0,0 +1,108 @@
+# Plan: Convert `default-secret-key` to Memoized Function
+
+## Goal
+Convert `default-secret-key` from a `defonce` binding to a memoized function so the value can be recalculated when `env/env` changes (e.g., during testing).
+
+## File to Modify
+- `/Users/edwardpaget/Projects/metabase/src/metabase/util/encryption.clj`
+
+## Implementation Steps
+
+### Step 1: Change `defonce` to a memoized function
+
+Replace:
+```clojure
+(def ^:private ^{:tag 'bytes} default-secret-key
+ (validate-and-hash-secret-key (env/env :mb-encryption-secret-key)))
+```
+
+With:
+```clojure
+(def ^:private default-secret-key*
+ "Memoized function that hashes the secret key. Takes the raw key as argument
+ so memoization correctly detects when the env value changes."
+ (memoize
+ (fn [secret-key-from-env]
+ (validate-and-hash-secret-key secret-key-from-env))))
+
+(defn ^:private default-secret-key
+ "Returns the hashed default encryption secret key, memoized for performance.
+ Automatically recalculates if the env var value changes."
+ ^bytes []
+ (default-secret-key* (env/env :mb-encryption-secret-key)))
+```
+
+**Key insight**: By passing `(env/env :mb-encryption-secret-key)` as an argument to the memoized function, the memoization will automatically return the cached value when the env value is the same, but recompute when it changes. No manual cache clearing needed for env changes.
+
+### Step 2: (Optional) Add a function to reset the memoization cache
+
+Since the memoization now keys on the env value itself, the cache will automatically recompute when the env changes. However, we may still want a reset function for testing edge cases (e.g., clearing all cached entries):
+
+```clojure
+(defn reset-default-secret-key!
+ "Clears all memoized secret key hashes. Normally not needed since the cache
+ automatically handles env changes, but useful for testing."
+ []
+ (memoize/memo-clear! default-secret-key*))
+```
+
+This requires adding `[clojure.core.memoize :as memoize]` to the requires. **This step is optional** - we can skip it if tests don't require clearing the entire cache.
+
+### Step 3: Update all usages to call it as a function
+
+Replace all occurrences of `default-secret-key` with `(default-secret-key)`:
+
+| Line | Current | Updated |
+|------|---------|---------|
+| 62 | `(boolean default-secret-key)` | `(boolean (default-secret-key))` |
+| 67 | `(if default-secret-key` | `(if (default-secret-key)` |
+| 68-70 | `(if default-secret-key "..." "...")` | `(if (default-secret-key) "..." "...")` |
+| 79 | `(encrypt-bytes default-secret-key b)` | `(encrypt-bytes (default-secret-key) b)` |
+| 93 | `(encrypt default-secret-key s)` | `(encrypt (default-secret-key) s)` |
+| 104 | `(decrypt-bytes default-secret-key b)` | `(decrypt-bytes (default-secret-key) b)` |
+| 117 | `(encrypt-stream default-secret-key input-stream)` | `(encrypt-stream (default-secret-key) input-stream)` |
+| 130 | `(encrypt-for-stream default-secret-key input)` | `(encrypt-for-stream (default-secret-key) input)` |
+| 139 | `(maybe-decrypt-stream default-secret-key input-stream)` | `(maybe-decrypt-stream (default-secret-key) input-stream)` |
+| 164 | `(decrypt default-secret-key s)` | `(decrypt (default-secret-key) s)` |
+| 171 | `(maybe-encrypt default-secret-key s)` | `(maybe-encrypt (default-secret-key) s)` |
+| 183 | `(maybe-encrypt-bytes default-secret-key b)` | `(maybe-encrypt-bytes (default-secret-key) b)` |
+| 193 | `(maybe-encrypt-for-stream default-secret-key s)` | `(maybe-encrypt-for-stream (default-secret-key) s)` |
+| 235 | `(cons default-secret-key args)` | `(cons (default-secret-key) args)` |
+
+### Step 4: (Only if Step 2 is included) Add require for clojure.core.memoize
+
+If adding the `reset-default-secret-key!` function, add to the `:require` block:
+```clojure
+[clojure.core.memoize :as memoize]
+```
+
+Note: `clojure.core/memoize` (used in Step 1) is built-in and requires no import. Only `clojure.core.memoize` (the contrib library) is needed for `memo-clear!`.
+
+## Verification
+
+1. Run the encryption tests:
+ ```
+ clojure -X:dev:test :only metabase.util.encryption-test
+ ```
+
+2. Test that memoization automatically detects env changes:
+ ```clojure
+ ;; In REPL
+ (require '[metabase.util.encryption :as encryption])
+ (require '[environ.core :as env])
+
+ ;; Check initial state (probably false if no key set)
+ (encryption/default-encryption-enabled?)
+
+ ;; Set a new key
+ (alter-var-root #'env/env assoc :mb-encryption-secret-key "some-test-key-16chars")
+
+ ;; Should now return true - memoization detects the env change automatically
+ (encryption/default-encryption-enabled?)
+
+ ;; Remove the key
+ (alter-var-root #'env/env dissoc :mb-encryption-secret-key)
+
+ ;; Should return false again
+ (encryption/default-encryption-enabled?)
+ ```
diff --git a/PLAN_optimize_can_run_adhoc_query.md b/PLAN_optimize_can_run_adhoc_query.md
new file mode 100644
index 00000000000..11d7ecc74e1
--- /dev/null
+++ b/PLAN_optimize_can_run_adhoc_query.md
@@ -0,0 +1,231 @@
+# Plan: Optimize `can_run_adhoc_query` Computation
+
+## Summary
+
+Optimize `can_run_adhoc_query` computation for both OSS and EE by avoiding expensive query preprocessing.
+
+**Current cost:** N cards × full query preprocessing (~3-15ms each)
+**Proposed cost:**
+- **OSS:** Extract source card IDs (lightweight) + DB-level permission check
+- **EE:** 1 recursive SQL query against dependency table
+
+## OSS Optimization
+
+On OSS, table-level blocking doesn't exist. The permission model simplifies to:
+
+| Check | What's Required |
+|-------|-----------------|
+| Source cards | Collection read permission |
+| Database | `create-queries` ≠ `:no` (`:query-builder` for MBQL, `:query-builder-and-native` for native) |
+
+**Key insight:** We can use `lib/all-source-card-ids` to extract card references without full `preprocess-query`. This walks the query AST but doesn't expand source cards or resolve tables.
+
+```clojure
+;; Lightweight extraction (no preprocessing)
+(lib/all-source-card-ids query) ;; → #{card-id ...}
+(lib/any-native-stage? query) ;; → boolean
+```
+
+## EE Optimization (with Dependencies Feature)
+
+| Dependency Path | Permission Required |
+|-----------------|---------------------|
+| Card → Table (direct) | view-data + create-queries |
+| Card → Card | collection read |
+| Card → Card → Table (transitive) | view-data only |
+
+The `is_direct` flag in the recursive CTE distinguishes direct vs transitive table dependencies.
+
+## Files to Create
+
+### 1. `enterprise/backend/src/metabase_enterprise/queries/adhoc_query_perms.clj`
+
+EE implementation using recursive CTE against dependency table.
+
+### 2. `enterprise/backend/test/metabase_enterprise/queries/adhoc_query_perms_test.clj`
+
+Test coverage for EE-specific behavior (recursive CTE, staleness fallback).
+
+## Files to Modify
+
+### 1. `src/metabase/queries/models/card.clj`
+
+**Change:** Replace expensive `query-perms/can-run-query?` with lightweight check
+
+```clojure
+;; NEW: Lightweight OSS permission check (no query preprocessing)
+(defn- can-run-adhoc-query-oss?
+ "Check if current user can run ad-hoc query for card.
+ Lightweight: extracts source cards without full query preprocessing."
+ [{:keys [database_id dataset_query]}]
+ (let [query (lib/query (lib-be/application-database-metadata-provider database_id)
+ dataset_query)
+ source-card-ids (lib/all-source-card-ids query)
+ native? (lib/any-native-stage? query)
+ required-perm (if native? :query-builder-and-native :query-builder)]
+ (and
+ ;; User has query permission on the database
+ (not= :no (perms/full-db-permission-for-user
+ api/*current-user-id* :perms/create-queries database_id))
+ (or (not native?)
+ (= required-perm (perms/full-db-permission-for-user
+ api/*current-user-id* :perms/create-queries database_id)))
+ ;; User can read all source cards
+ (every? mi/can-read?
+ (when (seq source-card-ids)
+ (t2/select :model/Card :id [:in source-card-ids]))))))
+
+;; defenterprise wrapper for EE override
+(defenterprise compute-can-run-adhoc-query-batch
+ "Compute can_run_adhoc_query for a batch of cards."
+ metabase-enterprise.queries.adhoc-query-perms
+ [cards]
+ ;; OSS: use lightweight check
+ (into {}
+ (map (fn [{card-id :id :as card}]
+ [card-id (can-run-adhoc-query-oss? card)]))
+ (filter (comp seq :dataset_query) cards)))
+
+;; Simplify with-can-run-adhoc-query - remove expensive prefetching
+(mu/defn with-can-run-adhoc-query
+ [cards :- [:maybe [:sequential ::queries.schema/card]]]
+ (perms/prime-db-cache (into #{} (map :database_id cards)))
+ (mi/instances-with-hydrated-data
+ cards :can_run_adhoc_query
+ #(compute-can-run-adhoc-query-batch (filter (comp seq :dataset_query) cards))
+ :id
+ {:default false}))
+```
+
+## Recursive CTE Query
+
+```clojure
+(defn transitive-deps-for-cards-query
+ "Build HoneySQL query for transitive dependencies with directness tracking."
+ [card-ids]
+ {:with-recursive
+ [[[:transitive_deps {:columns [:start_card_id :to_entity_type :to_entity_id :is_direct_table]}]
+ {:union-all
+ [;; Base case: direct dependencies
+ {:select [[:from_entity_id :start_card_id]
+ :to_entity_type
+ :to_entity_id
+ [[:case [:= :to_entity_type [:inline "table"]]
+ [:inline 1]
+ :else [:inline 0]]
+ :is_direct_table]]
+ :from [:dependency]
+ :where [:and
+ [:= :from_entity_type [:inline "card"]]
+ [:in :from_entity_id card-ids]]}
+ ;; Recursive case: follow card->X edges
+ {:select [:td.start_card_id
+ :d.to_entity_type
+ :d.to_entity_id
+ [[:inline 0] :is_direct_table]]
+ :from [[:dependency :d]]
+ :join [[:transitive_deps :td]
+ [:and
+ [:= :d.from_entity_type :td.to_entity_type]
+ [:= :d.from_entity_id :td.to_entity_id]]]
+ :where [:= :td.to_entity_type [:inline "card"]]}]}]]
+ :select [:start_card_id :to_entity_type :to_entity_id
+ [[:max :is_direct_table] :is_direct]]
+ :from [:transitive_deps]
+ :group-by [:start_card_id :to_entity_type :to_entity_id]})
+```
+
+## Staleness Handling
+
+Check `dependency_analysis_version` and fall back to OSS for stale cards:
+
+```clojure
+(defn- partition-by-staleness
+ "Split card IDs into fresh (use deps) and stale (use fallback)."
+ [card-ids]
+ (let [current-version models.dependency/current-dependency-analysis-version
+ stale-ids (t2/select-fn-set :id :model/Card
+ :id [:in card-ids]
+ :dependency_analysis_version [:< current-version])]
+ {:fresh (remove stale-ids card-ids)
+ :stale stale-ids}))
+```
+
+## Native Query Handling
+
+Native queries are handled by checking `dataset_query` type:
+
+```clojure
+(defn- native-query? [card]
+ (= :native (-> card :dataset_query :type)))
+
+(defn- has-native-perms? [db-id]
+ (= :query-builder-and-native
+ (perms/full-db-permission-for-user
+ api/*current-user-id* :perms/create-queries db-id)))
+```
+
+Native queries require `create-queries: :query-builder-and-native` on the database regardless of table dependencies.
+
+## Integration with Block Permissions
+
+The existing `check-block-permissions` middleware (EE) checks view-data on ALL tables in the preprocessed query. Our optimization handles the same tables via the transitive dependency graph, so block permissions will be respected.
+
+## Feature Gating
+
+```clojure
+(defenterprise compute-can-run-adhoc-query-batch
+ "..."
+ metabase-enterprise.queries.adhoc-query-perms
+ :feature :dependencies ;; Requires dependencies premium feature
+ [cards]
+ ;; OSS fallback
+ ...)
+```
+
+## Implementation Steps
+
+### Phase 1: OSS Optimization
+1. **Add `can-run-adhoc-query-oss?`** - lightweight check using `lib/all-source-card-ids`
+2. **Wrap in `defenterprise`** for EE override
+3. **Simplify `with-can-run-adhoc-query`** - remove expensive prefetching
+4. **Write tests** for OSS permission scenarios
+5. **Benchmark** OSS improvement
+
+### Phase 2: EE Optimization (requires dependencies feature)
+1. **Create EE namespace** with recursive CTE query builder
+2. **Implement EE version** with:
+ - Transitive deps fetch via CTE
+ - Direct vs transitive table permission checks
+ - Staleness detection and fallback
+3. **Write EE tests** covering dependency graph scenarios
+4. **Benchmark** EE improvement vs OSS
+
+## Verification
+
+### OSS Tests
+1. Card with no source cards → check DB permission only
+2. Card with source cards → check collection read + DB permission
+3. Native query → requires `:query-builder-and-native`
+4. User blocked from DB → returns false
+5. User can't read source card → returns false
+
+### EE Tests
+1. Direct table dependency → needs view-data + create-queries
+2. Transitive table dependency → needs view-data only
+3. Stale dependency_analysis_version → falls back to OSS
+4. Recursive CTE handles deep card chains (A → B → C → table)
+
+### Performance
+```clojure
+;; Benchmark with 500 cards
+(time (t2/hydrate cards :can_run_adhoc_query))
+;; Before: ~2-4 seconds
+;; After OSS: ~200-400ms (no preprocessing)
+;; After EE: ~50-100ms (single CTE)
+```
+
+### Manual Testing
+- GET `/api/collection/:id/items?models=card&include_can_run_adhoc_query=true`
+- Verify correct permissions with large collection (500+ cards)
+- Check SQL logs: should see 1 CTE query (EE) or N lightweight queries (OSS), not N preprocessing calls
diff --git a/PLAN_python_library_remote_sync.md b/PLAN_python_library_remote_sync.md
new file mode 100644
index 00000000000..9c7e7535be9
--- /dev/null
+++ b/PLAN_python_library_remote_sync.md
@@ -0,0 +1,285 @@
+# Plan: Remote-Sync Support for PythonLibrary
+
+## Overview
+
+This plan implements remote-sync support for the `PythonLibrary` entity, enabling Python libraries (reusable code for Python transforms) to be synchronized with a Git repository.
+
+## Background
+
+### Current PythonLibrary State
+- **Model**: `:model/PythonLibrary` (table: `python_library`)
+- **Identity**: Path-based (unique `path` column, currently only supports `common.py`)
+- **Fields**: `id`, `path`, `source`, `created_at`, `updated_at`
+- **Serialization**: Explicitly excluded in `excluded-models`
+- **Access**: Superuser-only
+- **Feature flag**: `transforms-python` premium feature
+
+### Remote-Sync Pattern Requirements
+Based on similar entities (Transforms, NativeQuerySnippets), PythonLibrary needs:
+1. Backend spec definition in `spec.clj`
+2. Serialization support (currently excluded)
+3. Event handling for create/update/delete
+4. Frontend type and config updates
+
+## Design Decisions
+
+### Identity Strategy: `:path`
+Since PythonLibrary uses `path` as its unique identifier (not `entity_id`), use path-based identity similar to Table/Field models.
+
+### Eligibility: Setting-based (tied to transforms)
+PythonLibrary is tightly coupled with Python transforms. Use the existing `remote-sync-transforms` setting to control eligibility - when transforms are synced, so are their associated libraries.
+
+### Export Path Type: `:python-library-path`
+New path type exporting to `python-libraries/{path}.yaml` structure.
+
+### Export Scope: `:all`
+Global scope - export all PythonLibrary instances (similar to TransformTag/NativeQuerySnippet).
+
+---
+
+## Implementation Steps
+
+### Phase 1: Backend - Serialization Support
+
+#### Step 1.1: Add PythonLibrary to serialization models list
+**File**: `enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj`
+
+- Remove `PythonLibrary` from `excluded-models` list
+- Add `:model/PythonLibrary` to the models list (likely in a transforms-related section)
+
+#### Step 1.2: Implement serialization methods for PythonLibrary
+**File**: `enterprise/backend/src/metabase_enterprise/transforms_python/models/python_library.clj`
+
+Add serdes methods:
+```clojure
+(defmethod serdes/hash-fields :model/PythonLibrary [_model] [:path])
+
+(defmethod serdes/make-spec "PythonLibrary" [_model-name _opts]
+ {:copy [:path :source]})
+
+(defmethod serdes/entity-id :model/PythonLibrary [{:keys [path]}] path)
+
+(defmethod serdes/generate-path :model/PythonLibrary [_ {:keys [path]}]
+ [{:model "PythonLibrary" :id path}])
+```
+
+#### Step 1.3: Implement serdes ingestion methods
+**File**: `enterprise/backend/src/metabase_enterprise/transforms_python/models/python_library.clj`
+
+Add ingestion support:
+```clojure
+(defmethod serdes/load-xform "PythonLibrary" [entity]
+ (-> entity
+ (dissoc :serdes/meta)
+ (assoc :path (-> entity :serdes/meta first :id))))
+
+(defmethod serdes/load-find-local "PythonLibrary" [path]
+ (t2/select-one :model/PythonLibrary :path (-> path last :id)))
+
+(defmethod serdes/load-update! "PythonLibrary" [_model-name ingested local]
+ (t2/update! :model/PythonLibrary (:id local) ingested))
+
+(defmethod serdes/load-insert! "PythonLibrary" [_model-name ingested]
+ (t2/insert-returning-instance! :model/PythonLibrary ingested))
+```
+
+---
+
+### Phase 2: Backend - Remote-Sync Spec
+
+#### Step 2.1: Add PythonLibrary spec to remote-sync
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj`
+
+Add spec definition:
+```clojure
+:model/PythonLibrary
+{:model-type "PythonLibrary"
+ :model-key :model/PythonLibrary
+ :identity :path
+ :events {:prefix :event/python-library
+ :types [:create :update :delete]}
+ :eligibility {:type :setting
+ :setting :remote-sync-transforms}
+ :archived-key nil ; PythonLibrary has no archived state
+ :tracking {:select-fields [:path]
+ :field-mappings {:model_name :path}}
+ :removal {:statuses #{"removed"}
+ :scope-key nil} ; global scope
+ :export-path {:type :python-library-path}
+ :export-scope :all
+ :enabled? :remote-sync-transforms}
+```
+
+#### Step 2.2: Implement python-library-path export path type
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj`
+
+Add path generation method:
+```clojure
+(defmethod export-path-segments :python-library-path
+ [_type {:keys [path]}]
+ ["python-libraries" (str (str/replace path #"\.py$" "") ".yaml")])
+```
+
+#### Step 2.3: Add path identity handler if needed
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj`
+
+Verify that the `:path` identity type properly handles PythonLibrary's path-based identification:
+```clojure
+(defmethod get-model-identity :path
+ [spec instance]
+ ;; May need to handle PythonLibrary specifically or ensure generic path handling works
+ ...)
+```
+
+---
+
+### Phase 3: Backend - Event Handling
+
+#### Step 3.1: Add PythonLibrary events to the events system
+**File**: `enterprise/backend/src/metabase_enterprise/transforms_python/models/python_library.clj`
+
+Add event publication on model changes:
+```clojure
+(t2/define-after-insert :model/PythonLibrary [library]
+ (events/publish-event! :event/python-library-create library)
+ library)
+
+(t2/define-after-update :model/PythonLibrary [library]
+ (events/publish-event! :event/python-library-update library)
+ library)
+
+(t2/define-after-delete :model/PythonLibrary [library]
+ (events/publish-event! :event/python-library-delete library))
+```
+
+#### Step 3.2: Verify event registration in remote-sync
+The spec-driven event system should automatically register handlers based on the spec's `:events` configuration. Verify that `register-events-for-spec!` properly handles the new PythonLibrary spec.
+
+---
+
+### Phase 4: Frontend - Types and Configuration
+
+#### Step 4.1: Update RemoteSyncEntityModel type
+**File**: `frontend/src/metabase-types/api/remote-sync.ts`
+
+Add "pythonlibrary" to the entity model type:
+```typescript
+export type RemoteSyncEntityModel =
+ | "card"
+ | "dataset"
+ | "metric"
+ | "dashboard"
+ | "collection"
+ | "document"
+ | "snippet"
+ | "table"
+ | "field"
+ | "segment"
+ | "measure"
+ | "transform"
+ | "transformtag"
+ | "pythonlibrary"; // Add this
+```
+
+#### Step 4.2: Add model mutation configs
+**File**: `enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/model-configs.ts`
+
+Add PythonLibrary configuration:
+```typescript
+{
+ modelType: "pythonlibrary",
+ createEndpoints: [], // Using upsert, no separate create
+ updateEndpoints: [pythonTransformLibraryApi.endpoints.updatePythonLibrary.matchFulfilled],
+ deleteEndpoints: [], // No delete endpoint currently
+ invalidation: { type: InvalidationType.Always },
+}
+```
+
+#### Step 4.3: Update entity display utilities (if needed)
+**File**: `enterprise/frontend/src/metabase-enterprise/remote_sync/utils.ts`
+
+May need to add icon/color mappings for pythonlibrary entity type in the AllChangesView display logic.
+
+---
+
+### Phase 5: Testing
+
+#### Step 5.1: Backend unit tests
+**File**: `enterprise/backend/test/metabase_enterprise/remote_sync/python_library_test.clj` (new)
+
+Test cases:
+- Serialization round-trip (export → import)
+- Event publication on create/update/delete
+- RemoteSyncObject tracking when setting enabled
+- No tracking when setting disabled
+- Export path generation
+
+#### Step 5.2: Backend integration tests
+**File**: `enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj` (add tests)
+
+Test cases:
+- PythonLibrary included in export when transforms setting enabled
+- PythonLibrary excluded from export when setting disabled
+- Import creates/updates PythonLibrary correctly
+
+#### Step 5.3: Frontend unit tests
+**File**: `enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/model-configs.unit.spec.ts` (if exists, add tests)
+
+Test cases:
+- PythonLibrary mutations trigger dirty state invalidation
+- Correct entity type display in AllChangesView
+
+---
+
+## Files to Modify
+
+### Backend (Clojure)
+
+| File | Changes |
+|------|---------|
+| `enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj` | Remove from excluded-models, add to appropriate models list |
+| `enterprise/backend/src/metabase_enterprise/transforms_python/models/python_library.clj` | Add serdes methods, event publication |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj` | Add PythonLibrary spec, export-path-segments method |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/events.clj` | Verify event registration (may be automatic) |
+
+### Frontend (TypeScript)
+
+| File | Changes |
+|------|---------|
+| `frontend/src/metabase-types/api/remote-sync.ts` | Add "pythonlibrary" to RemoteSyncEntityModel |
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/model-configs.ts` | Add PythonLibrary mutation config |
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/utils.ts` | Add icon/display helpers if needed |
+
+### Tests (New)
+
+| File | Purpose |
+|------|---------|
+| `enterprise/backend/test/metabase_enterprise/remote_sync/python_library_test.clj` | New test file for PythonLibrary remote-sync |
+
+---
+
+## Dependencies and Considerations
+
+### Prerequisites
+- Current branch already has transforms remote-sync support (based on recent commits)
+- `remote-sync-transforms` setting already exists and controls transform syncing
+
+### Risks and Mitigations
+1. **Path normalization**: PythonLibrary normalizes paths to `.py` extension - ensure serialization handles this consistently
+2. **Single library constraint**: Currently only `common.py` is supported - serialization should handle this gracefully
+3. **No delete endpoint**: PythonLibrary API only has GET/PUT (upsert) - may need to handle deletion tracking differently
+
+### Future Considerations
+- If more library paths are supported in future, the serialization will already handle them via path-based identity
+- Consider whether library deletion should be supported and how it would affect remote-sync
+
+---
+
+## Success Criteria
+
+1. ✅ PythonLibrary changes are tracked in RemoteSyncObject when `remote-sync-transforms` is enabled
+2. ✅ PythonLibrary exports to `python-libraries/common.yaml` during git sync export
+3. ✅ PythonLibrary imports from YAML correctly during git sync import
+4. ✅ Frontend shows PythonLibrary in dirty changes view
+5. ✅ All existing tests pass
+6. ✅ New tests for PythonLibrary remote-sync functionality pass
diff --git a/PLAN_query_processor_benchmarks.md b/PLAN_query_processor_benchmarks.md
new file mode 100644
index 00000000000..a791247c376
--- /dev/null
+++ b/PLAN_query_processor_benchmarks.md
@@ -0,0 +1,373 @@
+# Query Processor CPU and Memory Benchmarks - Implementation Plan
+
+## Existing Benchmarks
+
+### 1. Query Processor Compilation Benchmark
+**File**: `test/metabase/query_processor/perf_test.clj`
+
+Benchmarks query compilation (MBQL → SQL) using Criterium with mock metadata providers:
+- Small: 5 tables × 10 columns
+- Medium: 5 tables × 100 columns
+- Large: 5 tables × 1000 columns
+- Huge: 5 tables × 10,000 columns
+
+**Coverage**: Only `qp.compile/compile` - does NOT cover preprocessing, execution, or HTTP API.
+
+```clojure
+;; REPL usage
+(compile-time mp-medium trivial-query computed-cache-on)
+```
+
+### 2. Search Performance Benchmark
+**File**: `dev/src/dev/search_perf.clj`
+
+Full load testing framework for search functionality:
+- Creates 10k+ tables, users, permission groups
+- Tracks p50/p95 percentiles
+- Warmup iterations
+- Multiple search patterns
+
+```clojure
+;; REPL usage
+(run-full-benchmark! {:num-tables 10000 :num-groups 10 :num-users 100})
+```
+
+### 3. Memory Utilities
+**File**: `dev/src/dev/memory.clj`
+
+Thread allocation measurement utilities:
+- `measure-thread-allocations` - Bytes allocated by a function
+- `with-memory-logging` - Macro for memory logging
+
+### 4. Available Dependencies (in deps.edn :dev alias)
+- `criterium/criterium {:mvn/version "0.4.6"}` - Statistical benchmarking
+- `com.clojure-goes-fast/clj-async-profiler` - Flame graphs
+- `com.clojure-goes-fast/clj-memory-meter` - Deep object sizing
+- `com.clojure-goes-fast/jvm-alloc-rate-meter` - Heap allocation monitoring
+
+### Gap Analysis
+The existing benchmarks do NOT cover:
+- Full query execution (preprocessing + compilation + execution)
+- HTTP API layer performance
+- Concurrent load testing
+- Memory profiling per query type
+- Native queries, nested queries, pivots, exports
+
+---
+
+## Overview
+
+Add a comprehensive benchmarking suite for Metabase query processing that covers:
+- **Unit benchmarks**: Isolated QP phase testing (preprocess, compile, execute)
+- **Integration benchmarks**: Full query execution through API
+- **Load testing**: 50-200+ concurrent HTTP requests with percentile tracking
+- **Memory profiling**: Heap allocations and object sizing per query
+
+**Target databases**: H2 (default/CI) + Postgres (realistic testing)
+**Execution modes**: CI smoke tests + local REPL-driven profiling
+
+---
+
+## Directory Structure
+
+```
+dev/src/dev/benchmark/
+├── core.clj # Criterium wrappers, config, percentile calculations
+├── memory.clj # Memory profiling (extends dev.memory)
+├── profiler.clj # clj-async-profiler integration (flame graphs)
+├── queries/
+│ └── definitions.clj # Query library: MBQL, native, nested, pivot, exports
+├── qp/
+│ ├── unit.clj # Isolated QP phase benchmarks
+│ └── integration.clj # Full query execution benchmarks
+├── load/
+│ ├── concurrent.clj # HTTP load testing with thread pools
+│ └── data_setup.clj # Large dataset creation for Postgres
+└── report/
+ ├── console.clj # Pretty-printed console output
+ └── json.clj # JSON reports for CI artifacts
+
+test/metabase/benchmark/
+└── smoke_test.clj # Quick CI verification (no perf thresholds)
+```
+
+---
+
+## Implementation Steps
+
+### Step 1: Core Benchmark Infrastructure
+**Files**: `dev/src/dev/benchmark/core.clj`
+
+Create Criterium wrappers with structured result output:
+- `bench` - Run benchmark, return `{:mean-ns :std-dev-ns :percentiles {:p50 :p95 :p99}}`
+- `bench-quick` - Fast mode for smoke tests
+- Configuration via `*benchmark-config*` dynamic var
+
+**Pattern to follow**: `test/metabase/query_processor/perf_test.clj` lines 88-96
+
+```clojure
+(defn bench [thunk & {:keys [quick?]}]
+ (let [results (if quick?
+ (crit/quick-benchmark (thunk) {})
+ (crit/benchmark (thunk) {}))]
+ {:mean-ns (first (:mean results))
+ :percentiles (calculate-percentiles (:samples results))}))
+```
+
+---
+
+### Step 2: Query Definitions Library
+**Files**: `dev/src/dev/benchmark/queries/definitions.clj`
+
+Define representative queries using `metabase.test.data`:
+- `simple-select` - Basic table scan
+- `aggregation-query` - COUNT, SUM with filters
+- `join-query` - Multi-table joins with breakouts
+- `nested-query` - Query with saved question as source
+- `native-query` - Raw SQL execution
+- `pivot-query` - Pivot table generation
+
+Each returns a proper MBQL query map.
+
+---
+
+### Step 3: Unit Benchmarks (QP Phases)
+**Files**: `dev/src/dev/benchmark/qp/unit.clj`
+
+Benchmark isolated QP phases:
+
+```clojure
+(defn benchmark-preprocess [query]
+ (qp.setup/with-qp-setup [query query]
+ (mu/disable-enforcement
+ (bench #(qp.preprocess/preprocess query)))))
+
+(defn benchmark-compile [query]
+ (qp.setup/with-qp-setup [query query]
+ (mu/disable-enforcement
+ (let [preprocessed (qp.preprocess/preprocess query)]
+ (bench #(qp.compile/compile-preprocessed preprocessed))))))
+```
+
+**Key targets**:
+- `qp.preprocess/preprocess` - 42+ middleware steps
+- `qp.compile/compile-preprocessed` - MBQL → SQL
+- Individual middleware: `resolve-fields`, `add-remaps`, `annotate/expected-cols`
+
+---
+
+### Step 4: Integration Benchmarks
+**Files**: `dev/src/dev/benchmark/qp/integration.clj`
+
+Full end-to-end benchmarks:
+
+```clojure
+(defn benchmark-process-query [query]
+ (bench #(qp/process-query (qp/userland-query-with-default-constraints query))))
+
+(defn benchmark-api-query [query user]
+ (bench #(mt/user-http-request user :post 200 "dataset" query)))
+
+(defn benchmark-export [query format user]
+ (bench #(mt/user-http-request user :post 200
+ (str "dataset/" (name format)) {:query query})))
+```
+
+Test all query types + export formats (CSV, JSON, XLSX).
+
+---
+
+### Step 5: Load Testing (Concurrent HTTP)
+**Files**: `dev/src/dev/benchmark/load/concurrent.clj`
+
+Heavy concurrency testing (50-200+ concurrent):
+
+```clojure
+(defn run-load-test [{:keys [concurrent-requests duration-seconds query user]}]
+ (let [pool (cp/threadpool concurrent-requests)
+ results (atom [])]
+ (cp/pfor pool (range concurrent-requests)
+ (while-within-duration
+ (swap! results conj (execute-request query user))))
+ (calculate-stats @results)))
+```
+
+**Output**:
+- Throughput (requests/second)
+- Latency percentiles (p50, p95, p99)
+- Error rate
+- Total requests
+
+**Dependencies**: Add `org.clj-commons/claypoole` to `:dev` alias if not present.
+
+---
+
+### Step 6: Memory Profiling
+**Files**: `dev/src/dev/benchmark/memory.clj`
+
+Extend existing `dev.memory` utilities:
+
+```clojure
+(defn profile-query-memory [query]
+ (let [{:keys [allocations]} (memory/measure-thread-allocations
+ #(qp/process-query query))
+ result-size (mm/measure result)]
+ {:thread-allocations-bytes allocations
+ :result-size-bytes result-size
+ :bytes-per-row (/ result-size row-count)}))
+```
+
+**Uses**:
+- `dev.memory/measure-thread-allocations` (existing)
+- `clj-memory-meter.core/measure` for deep object sizing
+- `jvm-alloc-rate-meter` for allocation rate during load
+
+---
+
+### Step 7: Async Profiler (Flame Graphs)
+**Files**: `dev/src/dev/benchmark/profiler.clj`
+
+Generate CPU and allocation flame graphs:
+
+```clojure
+(defn profile-query-cpu [query & {:keys [iterations output-dir]}]
+ (prof/profile {:event :cpu :output-dir output-dir}
+ (dotimes [_ iterations]
+ (qp/process-query query)))
+ (prof/generate-flamegraph output-dir))
+```
+
+Output HTML flame graphs to `target/profiler/`.
+
+---
+
+### Step 8: Reporting
+**Files**: `dev/src/dev/benchmark/report/{console,json}.clj`
+
+**Console output**: Formatted table with benchmark name, mean, p50, p95, p99
+**JSON output**: Machine-readable for CI artifacts
+
+```clojure
+(defn generate-report [results]
+ {:timestamp (java.time.Instant/now)
+ :git-sha (git-sha)
+ :jvm-version (System/getProperty "java.version")
+ :results results})
+```
+
+Support baseline comparison with configurable regression threshold.
+
+---
+
+### Step 9: CI Integration
+**Files**: `test/metabase/benchmark/smoke_test.clj`, `deps.edn`
+
+Smoke tests verify benchmarks complete without errors (no perf assertions):
+
+```clojure
+(deftest ^:benchmark preprocess-benchmark-smoke-test
+ (testing "Preprocess benchmark completes"
+ (let [result (unit/benchmark-preprocess (queries/simple-select))]
+ (is (pos? (:mean-ns result)))
+ (is (< (:mean-ns result) 1e10))))) ; Sanity: < 10 seconds
+```
+
+**deps.edn alias**:
+```clojure
+:benchmark {:extra-paths ["dev/src"]
+ :exec-fn dev.benchmark.runner/run-benchmarks
+ :exec-args {:output-path "target/benchmark-results.json"}}
+```
+
+---
+
+### Step 10: Database Setup (H2 + Postgres)
+**Files**: `dev/src/dev/benchmark/load/data_setup.clj`
+
+**H2**: Use standard test data (`metabase.test.data/db`)
+**Postgres**: Create larger datasets using `dev.add-load` patterns
+
+```clojure
+(def data-sizes
+ {:small {:tables 5 :rows-per-table 100}
+ :medium {:tables 10 :rows-per-table 1000}
+ :large {:tables 20 :rows-per-table 10000}})
+
+(defn ensure-benchmark-data! [size]
+ (case (driver/the-driver)
+ :h2 (data/db)
+ :postgres (create-postgres-benchmark-data! size)))
+```
+
+---
+
+## Key Files to Reference
+
+| Purpose | Existing File |
+|---------|--------------|
+| Criterium patterns | `test/metabase/query_processor/perf_test.clj` |
+| Load testing patterns | `dev/src/dev/search_perf.clj` |
+| Memory utilities | `dev/src/dev/memory.clj` |
+| Bulk data loading | `dev/src/dev/add_load.clj` |
+| Test data setup | `test/metabase/test/data.clj` |
+| HTTP test client | `test/metabase/test/http_client.clj` |
+
+---
+
+## REPL Usage Examples
+
+```clojure
+;; Unit benchmarks
+(require '[dev.benchmark.qp.unit :as unit])
+(unit/run-all-unit-benchmarks)
+
+;; Integration benchmarks
+(require '[dev.benchmark.qp.integration :as integration])
+(integration/run-all-integration-benchmarks)
+
+;; Load test with 100 concurrent for 30 seconds
+(require '[dev.benchmark.load.concurrent :as load])
+(load/run-load-test {:concurrent-requests 100
+ :duration-seconds 30
+ :query (queries/simple-select)})
+
+;; CPU flame graph
+(require '[dev.benchmark.profiler :as profiler])
+(profiler/profile-query-cpu (queries/complex-join))
+;; Open target/profiler/*.html
+```
+
+---
+
+## CI Usage
+
+```bash
+# Quick smoke tests
+clojure -X:dev:test :only '["test/metabase/benchmark"]' :include-tags '[:benchmark]'
+
+# Full benchmark with JSON report
+clojure -X:dev:benchmark :output-path target/benchmark-results.json
+```
+
+---
+
+## Risk Mitigations
+
+| Risk | Mitigation |
+|------|------------|
+| Flaky CI tests | Smoke tests verify completion only, no performance thresholds |
+| Thread pool exhaustion | Configure Jetty threads (MB_JETTY_MAXTHREADS) for load tests |
+| Memory leaks during load | Monitor heap, limit duration, force GC between tests |
+| Non-reproducible results | Record JVM version, git SHA, use relative comparisons |
+| H2 unrealistic for production | Postgres available for local testing with larger datasets |
+
+---
+
+## Dependencies to Add
+
+```clojure
+;; In deps.edn :dev alias (if not already present)
+org.clj-commons/claypoole {:mvn/version "1.2.0"} ; Thread pools for load testing
+```
+
+All other dependencies (criterium, clj-async-profiler, clj-memory-meter) already present in `:dev` alias.
diff --git a/PLAN_remote_sync_has_changes.md b/PLAN_remote_sync_has_changes.md
new file mode 100644
index 00000000000..55d959c6fc4
--- /dev/null
+++ b/PLAN_remote_sync_has_changes.md
@@ -0,0 +1,222 @@
+# Plan: Remote Sync "Has Remote Changes" Endpoint
+
+## Goal
+
+Create a new endpoint that checks if there are changes on the remote (Git) repository that can be pulled, with in-memory caching to avoid excessive Git operations.
+
+## Current State
+
+- Remote-sync endpoints are defined in `enterprise/backend/src/metabase_enterprise/remote_sync/api.clj`
+- Version tracking exists via `remote-sync.task/last-version` (returns last successfully synced Git SHA)
+- Remote version can be fetched via `source.p/version` on a snapshot
+- Currently, remote changes are only detected during export (to prevent overwriting)
+- **No endpoint exists to check for remote changes without attempting a sync**
+
+## Design Decisions
+
+- **Never imported before**: Return `has_changes: true` (there is content to pull)
+- **Remote unreachable**: Let error propagate (consistent with other endpoints)
+- **Branch changed**: Reset cache when branch setting changes
+- **Change summary**: Not included - endpoint only returns boolean and versions
+
+## Implementation Steps
+
+### Step 1: Add Cache Atom and TTL Setting
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj`
+
+Add an atom-based cache for storing remote change check results:
+
+```clojure
+(def ^:private remote-changes-cache
+ "Cache for remote changes check to avoid frequent git operations.
+ Structure: {:last-checked-ms <timestamp>
+ :branch <branch-name>
+ :remote-version <git-sha>
+ :has-changes? <boolean>}"
+ (atom nil))
+```
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj`
+
+Add a configurable TTL setting:
+
+```clojure
+(defsetting remote-sync-check-changes-cache-ttl-seconds
+ (deferred-tru "Time-to-live in seconds for the remote changes check cache. Default is 60 seconds.")
+ :type :integer
+ :visibility :admin
+ :export? false
+ :encryption :no
+ :default 60)
+```
+
+### Step 2: Implement Core Logic
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj`
+
+Add function to check for remote changes with caching:
+
+```clojure
+(defn has-remote-changes?
+ "Check if remote has new changes compared to last imported version.
+ Uses cache to avoid frequent git operations. Returns map with:
+ - :has-changes? boolean
+ - :remote-version string (git SHA)
+ - :local-version string (git SHA of last import, or nil)
+ - :cached? boolean (whether result came from cache)
+
+ Cache is invalidated if:
+ - TTL has expired
+ - Branch setting has changed
+ - force-refresh? is true"
+ ([]
+ (has-remote-changes? nil))
+ ([{:keys [force-refresh?]}]
+ (let [cache-state @remote-changes-cache
+ now-ms (System/currentTimeMillis)
+ ttl-ms (* 1000 (settings/remote-sync-check-changes-cache-ttl-seconds))
+ current-branch (settings/remote-sync-branch)
+ cache-valid? (and cache-state
+ (not force-refresh?)
+ (= current-branch (:branch cache-state))
+ (< (- now-ms (:last-checked-ms cache-state)) ttl-ms))]
+ (if cache-valid?
+ (assoc cache-state :cached? true)
+ (let [last-imported (remote-sync.task/last-version)
+ source (source/source-from-settings current-branch)
+ snapshot (source.p/snapshot source)
+ current-remote (source.p/version snapshot)
+ ;; has-changes? is true if:
+ ;; - never imported (last-imported is nil), OR
+ ;; - remote version differs from local version
+ has-changes? (or (nil? last-imported)
+ (not= last-imported current-remote))
+ result {:last-checked-ms now-ms
+ :branch current-branch
+ :remote-version current-remote
+ :local-version last-imported
+ :has-changes? has-changes?}]
+ (reset! remote-changes-cache result)
+ (assoc result :cached? false))))))
+
+(defn invalidate-remote-changes-cache!
+ "Invalidate the remote changes cache. Call this after import/export."
+ []
+ (reset! remote-changes-cache nil))
+```
+
+### Step 3: Invalidate Cache on Import/Export
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj`
+
+Modify `async-import!` and `async-export!` to invalidate cache on success:
+
+```clojure
+;; At the end of successful import in async-import!
+(invalidate-remote-changes-cache!)
+
+;; At the end of successful export in async-export!
+(invalidate-remote-changes-cache!)
+```
+
+### Step 4: Add Response Schema
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/schema.clj`
+
+```clojure
+(def HasRemoteChangesResponse
+ "Response schema for the has-remote-changes endpoint."
+ [:map
+ [:has_changes :boolean]
+ [:remote_version [:maybe :string]]
+ [:local_version [:maybe :string]]
+ [:cached :boolean]])
+```
+
+### Step 5: Add API Endpoint
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/api.clj`
+
+```clojure
+(api.macros/defendpoint :get "/has-remote-changes" :- schema/HasRemoteChangesResponse
+ "Check if there are new changes on the remote branch that can be pulled.
+ Uses in-memory caching (configurable TTL via remote-sync-check-changes-cache-ttl-seconds setting).
+
+ Returns:
+ - has_changes: true if remote version differs from last imported version, or if never imported
+ - remote_version: current Git SHA on remote branch
+ - local_version: Git SHA of last successful import (nil if never imported)
+ - cached: true if result was served from cache"
+ [_route-params
+ _query-params
+ {:keys [force_refresh] :or {force_refresh false}} :- [:map [:force_refresh {:optional true} :boolean]]]
+ (api/check-superuser)
+ (api/check-400 (settings/remote-sync-enabled) (tru "Remote sync is not configured."))
+ (let [result (impl/has-remote-changes? {:force-refresh? force_refresh})]
+ {:has_changes (:has-changes? result)
+ :remote_version (:remote-version result)
+ :local_version (:local-version result)
+ :cached (:cached? result)}))
+```
+
+### Step 6: Add Tests
+
+**File:** `enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj`
+
+Add tests for:
+1. Endpoint returns `has_changes: true` when never imported
+2. Endpoint returns `has_changes: true` when remote version differs
+3. Endpoint returns `has_changes: false` when versions match
+4. Cache is used on subsequent calls within TTL
+5. `force_refresh=true` bypasses cache
+6. Cache is invalidated when branch changes
+7. Cache is invalidated after import
+8. Cache is invalidated after export
+9. Endpoint requires superuser
+10. Endpoint returns 400 if remote-sync not configured
+
+## API Contract
+
+### Request
+
+```
+GET /api/ee/remote-sync/has-remote-changes
+GET /api/ee/remote-sync/has-remote-changes?force_refresh=true
+```
+
+**Headers:** Requires superuser authentication
+
+**Query Parameters:**
+- `force_refresh` (optional, boolean): If true, bypass cache and check remote
+
+### Response
+
+```json
+{
+ "has_changes": true,
+ "remote_version": "abc123def456...",
+ "local_version": "789xyz...",
+ "cached": false
+}
+```
+
+**Fields:**
+- `has_changes`: `true` if there are changes to pull (or never imported), `false` if up-to-date
+- `remote_version`: Current Git SHA on the remote branch
+- `local_version`: Git SHA of last successful import (`null` if never imported)
+- `cached`: `true` if this result came from cache, `false` if freshly fetched
+
+**Status Codes:**
+- `200`: Success
+- `400`: Remote sync not configured
+- `401`: Not authenticated
+- `403`: Not superuser
+
+## Files to Modify
+
+1. `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj` - Add TTL setting
+2. `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj` - Add cache and logic
+3. `enterprise/backend/src/metabase_enterprise/remote_sync/schema.clj` - Add response schema
+4. `enterprise/backend/src/metabase_enterprise/remote_sync/api.clj` - Add endpoint
+5. `enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj` - Add tests
diff --git a/PLAN_remote_sync_snippets.md b/PLAN_remote_sync_snippets.md
new file mode 100644
index 00000000000..eee332b09f3
--- /dev/null
+++ b/PLAN_remote_sync_snippets.md
@@ -0,0 +1,303 @@
+# Plan: Support Remote Syncing NativeQuerySnippets Globally
+
+## Overview
+
+Add support for globally syncing ALL NativeQuerySnippets and their collections when the Library collection is remote-synced. No separate setting is needed - snippet syncing is automatically tied to the Library collection's `is_remote_synced` status.
+
+**Trigger**: When `(collection/library-collection)` has `is_remote_synced = true`, all snippets sync globally.
+
+## Current State
+
+### NativeQuerySnippet Remote Sync (existing)
+- **Location**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj` (lines 110-124)
+- **Eligibility**: Collection-based (`:collection :remote-synced`)
+- **Behavior**: Only syncs snippets in remote-synced collections
+- **Export Path**: `:collection-entity` type
+
+### Transform Remote Sync (reference pattern)
+- Uses `remote-sync-transforms` setting
+- When enabled, syncs ALL transforms globally
+- We'll follow similar pattern but check Library collection sync status instead
+
+## Implementation Plan
+
+### 1. Backend: Add Helper Function to Check Library Sync Status
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj`
+
+Add helper function:
+```clojure
+(defn library-is-remote-synced?
+ "Returns true if the Library collection exists and is remote-synced.
+ When true, all snippets and snippet collections should be synced."
+ []
+ (when-let [library (collection/library-collection)]
+ (collection/remote-synced-collection? library)))
+```
+
+### 2. Backend: Update NativeQuerySnippet Spec
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj`
+
+Change the NativeQuerySnippet spec from collection-based to a new eligibility type that checks Library sync status:
+
+**Current spec** (lines 110-124):
+```clojure
+:model/NativeQuerySnippet
+{:model-type "NativeQuerySnippet"
+ :model-key :model/NativeQuerySnippet
+ :identity :entity-id
+ :events {:prefix :event/snippet
+ :types [:create :update :delete]}
+ :eligibility {:type :collection
+ :collection :remote-synced}
+ :archived-key :archived
+ :tracking {:select-fields [:name :id]
+ :field-mappings {:model_name :name}}
+ :removal {:statuses #{"removed"}
+ :scope-key :collection_id}
+ :export-path {:type :collection-entity}
+ :enabled? true}
+```
+
+**New spec**:
+```clojure
+:model/NativeQuerySnippet
+{:model-type "NativeQuerySnippet"
+ :model-key :model/NativeQuerySnippet
+ :identity :entity-id
+ :events {:prefix :event/snippet
+ :types [:create :update :delete]}
+ :eligibility {:type :library-synced} ; New eligibility type
+ :archived-key :archived
+ :tracking {:select-fields [:name :id]
+ :field-mappings {:model_name :name}}
+ :removal {:statuses #{"removed"}} ; Remove scope-key for global deletion
+ :export-path {:type :snippet-path} ; New path type for snippets directory
+ :export-scope :all ; Export all snippets
+ :enabled? :library-synced} ; Controlled by Library sync status
+```
+
+### 3. Backend: Add `:library-synced` Eligibility Type
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/events.clj`
+
+Add new eligibility check method:
+```clojure
+(defmethod check-eligibility :library-synced
+ [_eligibility _entity]
+ (settings/library-is-remote-synced?))
+```
+
+### 4. Backend: Add `spec-enabled?` Support for `:library-synced`
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj`
+
+Update `spec-enabled?` to handle the `:library-synced` keyword:
+```clojure
+(defn spec-enabled?
+ "Returns true if the spec is enabled."
+ [spec]
+ (let [enabled (:enabled? spec)]
+ (cond
+ (boolean? enabled) enabled
+ (= enabled :library-synced) (settings/library-is-remote-synced?)
+ (keyword? enabled) (boolean ((keyword (name enabled)))))))
+```
+
+### 5. Backend: Add Snippet Path Construction
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj`
+
+Add new export path type `:snippet-path` to `construct-export-path` multimethod:
+
+```clojure
+(defmethod construct-export-path :snippet-path
+ [_path-type entity]
+ (let [collection (when (:collection_id entity)
+ (t2/select-one :model/Collection :id (:collection_id entity)))]
+ (if collection
+ ;; Snippets in collections: snippets/<collection-hierarchy>/<entity-id>
+ (concat ["snippets"]
+ (rest (serdes/path-for-collection collection)) ; Remove "collections" prefix
+ [(serdes/eid->label entity)])
+ ;; Root snippets: snippets/<entity-id>
+ ["snippets" (serdes/eid->label entity)])))
+```
+
+### 6. Backend: Update Collection Eligibility for Snippets Namespace
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj` or `events.clj`
+
+Update Collection eligibility to include snippets-namespace collections when Library is remote-synced:
+
+```clojure
+;; In the Collection eligibility check, a collection is eligible if:
+;; 1. It's remote-synced, OR
+;; 2. It's a transforms-namespace collection and remote-sync-transforms is enabled, OR
+;; 3. It's a snippets-namespace collection and Library is remote-synced
+```
+
+Update `should-sync-collection?` function:
+```clojure
+(defn should-sync-collection?
+ "Returns true if a collection should be synced."
+ [collection]
+ (or (collection/remote-synced-collection? collection)
+ (and (= (:namespace collection) :transforms)
+ (settings/remote-sync-transforms))
+ (and (= (:namespace collection) :snippets)
+ (settings/library-is-remote-synced?))))
+```
+
+### 7. Backend: Add Import Path Filter
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj`
+
+Update the path filters in the import function to include snippets:
+
+**Current** (around line 140):
+```clojure
+(let [path-filters (cond-> [#"collections/.*" #"databases/.*"]
+ (settings/remote-sync-transforms)
+ (conj #"transforms/.*"))]
+```
+
+**New**:
+```clojure
+(let [path-filters (cond-> [#"collections/.*" #"databases/.*"]
+ (settings/remote-sync-transforms)
+ (conj #"transforms/.*")
+ (settings/library-is-remote-synced?)
+ (conj #"snippets/.*"))]
+```
+
+### 8. Backend: Add Snippet Tracking Sync Function
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj`
+
+Add `sync-snippet-tracking!` function to be called when Library collection's sync status changes:
+
+```clojure
+(defn sync-snippet-tracking!
+ "Called when the Library collection's remote sync status changes. When enabled,
+ marks all existing snippets and snippet collections for initial sync. When
+ disabled, removes all snippet tracking entries."
+ [enabled?]
+ (if enabled?
+ ;; Mark all snippets and snippet collections for sync
+ (do
+ ;; Mark snippets-namespace collections
+ (doseq [collection (t2/select :model/Collection :namespace :snippets)]
+ (rso/upsert-status! "Collection" (:entity_id collection) "create"
+ {:model_name (:name collection)
+ :model_collection_id (:id collection)}))
+ ;; Mark all snippets
+ (doseq [snippet (t2/select :model/NativeQuerySnippet)]
+ (rso/upsert-status! "NativeQuerySnippet" (:entity_id snippet) "create"
+ {:model_name (:name snippet)})))
+ ;; Remove all snippet tracking
+ (do
+ (t2/delete! :model/RemoteSyncObject :model_type "NativeQuerySnippet")
+ (t2/delete! :model/RemoteSyncObject
+ :model_type "Collection"
+ :model_collection_id [:in {:select [:id]
+ :from [:collection]
+ :where [:= :namespace "snippets"]}]))))
+```
+
+### 9. Backend: Call Snippet Tracking on Library Sync Status Change
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/events.clj` or appropriate event handler
+
+When the Library collection's `is_remote_synced` changes, call `sync-snippet-tracking!`:
+
+```clojure
+;; In the collection update event handler, detect Library collection sync changes
+(when (and (collection/is-library? collection)
+ (contains? changes :is_remote_synced))
+ (impl/sync-snippet-tracking! (:is_remote_synced changes)))
+```
+
+### 10. Backend: Update Dirty Detection
+
+**File**: `enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj`
+
+Update `excluded-model-types` to exclude NativeQuerySnippet when Library is not remote-synced:
+
+```clojure
+(defn excluded-model-types
+ "Returns set of model types to exclude from dirty detection based on settings."
+ []
+ (cond-> #{}
+ (not (settings/remote-sync-transforms))
+ (conj "Transform" "TransformTag")
+ (not (settings/library-is-remote-synced?))
+ (conj "NativeQuerySnippet")))
+```
+
+### 11. Tests
+
+**File**: `enterprise/backend/test/metabase_enterprise/remote_sync/snippets_test.clj` (new file)
+
+Create comprehensive tests:
+
+1. Test snippet tracking is created when Library becomes remote-synced
+2. Test snippet tracking is removed when Library is no longer remote-synced
+3. Test snippets are exported when Library is remote-synced
+4. Test snippets are not exported when Library is not remote-synced
+5. Test snippet collections (`:snippets` namespace) are synced when Library is remote-synced
+6. Test import includes snippets when Library is remote-synced
+7. Test dirty detection excludes snippets when Library is not remote-synced
+8. Test archived snippets are marked for deletion
+9. Test snippet eligibility check returns correct value based on Library status
+
+## File Changes Summary
+
+| File | Change Type | Description |
+|------|-------------|-------------|
+| `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj` | Modify | Add `library-is-remote-synced?` helper function |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj` | Modify | Update NativeQuerySnippet spec, add `:snippet-path`, add `:library-synced` enabled check |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/events.clj` | Modify | Add `:library-synced` eligibility type, handle Library sync status changes |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj` | Modify | Add snippet path filter, add `sync-snippet-tracking!` |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj` | Modify | Update `excluded-model-types` |
+| `enterprise/backend/test/metabase_enterprise/remote_sync/snippets_test.clj` | New | Comprehensive tests |
+
+## Implementation Order
+
+1. **Phase 1: Core Infrastructure**
+ - Add `library-is-remote-synced?` helper
+ - Add `:library-synced` eligibility type
+ - Update `spec-enabled?` for `:library-synced`
+
+2. **Phase 2: Spec Updates**
+ - Update NativeQuerySnippet spec
+ - Add `:snippet-path` export path type
+ - Update Collection eligibility for snippets namespace
+
+3. **Phase 3: Tracking & Import**
+ - Add `sync-snippet-tracking!` function
+ - Hook into Library collection sync status change events
+ - Add import path filter for snippets
+ - Update dirty detection exclusions
+
+4. **Phase 4: Testing**
+ - Create comprehensive test file
+ - Test all edge cases
+
+## Key Differences from Transform Implementation
+
+| Aspect | Transforms | Snippets |
+|--------|------------|----------|
+| Trigger | `remote-sync-transforms` setting | Library collection `is_remote_synced` |
+| Setting required | Yes | No |
+| Eligibility type | `:setting` | `:library-synced` (new) |
+| Collections namespace | `:transforms` | `:snippets` |
+| Export path | `transforms/...` | `snippets/...` |
+
+## Notes
+
+- No frontend changes needed (no new setting to toggle)
+- Snippet sync is implicitly enabled when Library is remote-synced
+- All snippets sync globally (not scoped to specific collections)
+- Snippet collections (`:snippets` namespace) also sync when enabled
diff --git a/PLAN_standardize_table_permissions.md b/PLAN_standardize_table_permissions.md
new file mode 100644
index 00000000000..c6475c4a9dc
--- /dev/null
+++ b/PLAN_standardize_table_permissions.md
@@ -0,0 +1,314 @@
+# Plan: Standardize Table/Field/Database Permission Checks
+
+## Goal
+
+Consolidate all permission checks for Tables, Fields, and Databases through the standard `metabase.models.interface` methods rather than having ad hoc checks scattered throughout the codebase.
+
+## Key Insight
+
+The current `can-read?` for Table/Field conflates two distinct questions:
+1. "Can I see this table exists and view its metadata?"
+2. "Can I query data from this table?"
+
+Cards handle this correctly: `can-read?` answers "can see metadata" (via collection permissions), while query execution is checked separately.
+
+## Proposed Interface
+
+Add a new multimethod `can-query?` to cleanly separate these concerns:
+
+| Method | Semantics | Table/Field Implementation |
+|--------|-----------|---------------------------|
+| `can-read?` | Can see this exists, view metadata | data perms OR manage-table-metadata OR published-in-accessible-collection |
+| `can-query?` | Can execute queries against data | `:perms/view-data` + `:perms/create-queries` |
+| `can-write?` | Can modify this model | `:perms/manage-table-metadata` (EE) or superuser (OSS) |
+
+## Current State: Ad Hoc Permission Checks
+
+These locations have special-cased permission checks that should use the standard interface:
+
+### Direct Permission Checks (bypass `mi/can-read?`)
+
+| File | Lines | Current Check | Should Use |
+|------|-------|---------------|------------|
+| `src/metabase/search/impl.clj` | 72-80 | Duplicates view-data + create-queries check | `mi/can-read?` |
+| `src/metabase/warehouse_schema_rest/api/table.clj` | 513-518 | Direct `:perms/manage-table-metadata` | `mi/can-write?` |
+| `src/metabase/warehouses_rest/api.clj` | 1241-1251 | Composite schema permission | New `can-read-schema?` helper using `mi/can-read?` |
+| `enterprise/.../sandbox/api/table.clj` | 62-71 | Permission elevation for sandboxed | Should work with updated `can-read?` |
+
+### Filter Functions (should use `mi/can-read?`)
+
+| File | Lines | Function |
+|------|-------|----------|
+| `enterprise/.../advanced_permissions/common.clj` | 91-111 | `filter-tables-by-data-model-perms` |
+| `enterprise/.../advanced_permissions/common.clj` | 113-133 | `filter-schema-by-data-model-perms` |
+| `enterprise/.../advanced_permissions/common.clj` | 135-160 | `filter-databases-by-data-model-perms` |
+
+## Implementation Plan
+
+### Phase 1: Add `can-query?` Multimethod
+
+**File: `src/metabase/models/interface.clj`**
+
+```clojure
+(defmulti can-query?
+ "Return whether [[metabase.api.common/*current-user*]] has permission to
+ execute queries against this model's data. For Tables/Fields, this checks
+ data access permissions. For Cards, this checks if the user can run the
+ underlying query. Models without queryable data return false by default."
+ {:arglists '([instance] [model pk])}
+ dispatch-on-model)
+
+(defmethod can-query? :default
+ ([_instance] false)
+ ([_model _pk] false))
+```
+
+**File: `src/metabase/warehouse_schema/models/table.clj`**
+
+```clojure
+(defmethod mi/can-query? :model/Table
+ ([instance]
+ (and (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/view-data
+ :unrestricted
+ (:db_id instance)
+ (:id instance))
+ (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/create-queries
+ :query-builder
+ (:db_id instance)
+ (:id instance))))
+ ([_ pk]
+ (mi/can-query? (t2/select-one :model/Table pk))))
+```
+
+**File: `src/metabase/warehouse_schema/models/field.clj`**
+
+```clojure
+(defmethod mi/can-query? :model/Field
+ ([instance]
+ (and (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/view-data
+ :unrestricted
+ (field->db-id instance)
+ (:table_id instance))
+ (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/create-queries
+ :query-builder
+ (field->db-id instance)
+ (:table_id instance))))
+ ([model pk]
+ (mi/can-query? (t2/select-one model pk))))
+```
+
+**File: `src/metabase/warehouses/models/database.clj`**
+
+```clojure
+(defmethod mi/can-query? :model/Database
+ ([instance]
+ (mi/can-query? :model/Database (u/the-id instance)))
+ ([_model database-id]
+ (contains? #{:query-builder :query-builder-and-native}
+ (perms/most-permissive-database-permission-for-user
+ api/*current-user-id*
+ :perms/create-queries
+ database-id))))
+```
+
+### Phase 2: Update `can-read?` for Metadata Visibility
+
+Update Table/Field/Database `can-read?` to include all valid access paths:
+
+**File: `src/metabase/warehouse_schema/models/table.clj`**
+
+```clojure
+(defmethod mi/can-read? :model/Table
+ ([instance]
+ (or
+ ;; Path 1: Has data query permissions
+ (mi/can-query? instance)
+ ;; Path 2: Has manage-table-metadata permission
+ (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/manage-table-metadata
+ :yes
+ (:db_id instance)
+ (:id instance))
+ ;; Path 3: Table is published in accessible collection (EE)
+ (published-tables/can-access-via-collection? instance)))
+ ([_ pk]
+ (mi/can-read? (t2/select-one :model/Table pk))))
+```
+
+**File: `src/metabase/warehouse_schema/models/field.clj`**
+
+```clojure
+(defmethod mi/can-read? :model/Field
+ ([instance]
+ (or
+ ;; Path 1: Has data query permissions
+ (mi/can-query? instance)
+ ;; Path 2: Has manage-table-metadata permission on parent table
+ (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/manage-table-metadata
+ :yes
+ (field->db-id instance)
+ (:table_id instance))
+ ;; Path 3: Parent table is published in accessible collection
+ (when-let [table (or (:table instance)
+ (t2/select-one :model/Table :id (:table_id instance)))]
+ (published-tables/can-access-via-collection? table))))
+ ([model pk]
+ (mi/can-read? (t2/select-one model pk))))
+```
+
+**File: `src/metabase/warehouses/models/database.clj`**
+
+Keep existing logic (already includes published table check):
+```clojure
+(defmethod mi/can-read? :model/Database
+ ([_model database-id]
+ (or (mi/can-query? :model/Database database-id)
+ (perms/user-has-published-table-permission-for-database? database-id))))
+```
+
+### Phase 3: Migrate Special-Cased Checks
+
+#### 3.1 Search Permission Check
+
+**File: `src/metabase/search/impl.clj` (lines 72-80)**
+
+Before:
+```clojure
+(defmethod check-permissions-for-model :table
+ [search-ctx instance]
+ (let [instance-id (:id instance)
+ user-id (:current-user-id search-ctx)
+ db-id (database/table-id->database-id instance-id)]
+ (and
+ (perms/user-has-permission-for-table? user-id :perms/view-data :unrestricted db-id instance-id)
+ (perms/user-has-permission-for-table? user-id :perms/create-queries :query-builder db-id instance-id))))
+```
+
+After:
+```clojure
+(defmethod check-permissions-for-model :table
+ [_search-ctx instance]
+ (mi/can-read? :model/Table (:id instance)))
+```
+
+#### 3.2 Sync Schema Endpoint
+
+**File: `src/metabase/warehouse_schema_rest/api/table.clj` (lines 513-518)**
+
+Before:
+```clojure
+(api/check-403
+ (perms/user-has-permission-for-table?
+ api/*current-user-id*
+ :perms/manage-table-metadata
+ :yes
+ (:id database)
+ id))
+```
+
+After:
+```clojure
+(api/write-check :model/Table id)
+```
+
+#### 3.3 Enterprise Filter Functions
+
+**File: `enterprise/.../advanced_permissions/common.clj`**
+
+Update `filter-tables-by-data-model-perms` to use `mi/can-read?`:
+```clojure
+(defn filter-tables-by-data-model-perms
+ [tables]
+ (filter mi/can-read? tables))
+```
+
+Or if specifically filtering for write access:
+```clojure
+(defn filter-tables-by-data-model-perms
+ [tables]
+ (filter mi/can-write? tables))
+```
+
+### Phase 4: Add `api/query-check` Helper
+
+**File: `src/metabase/api/common.clj`**
+
+```clojure
+(defn query-check
+ "Check whether we can query data from `obj`, or `entity` with `id`.
+ If the object doesn't exist, throw a 404; if we don't have permissions, throw 403."
+ ([obj]
+ (check-404 obj)
+ (check-403 (mi/can-query? obj))
+ obj)
+ ([entity id]
+ (query-check (t2/select-one entity :id id)))
+ ([entity id & other-conditions]
+ (query-check (apply t2/select-one entity :id id other-conditions))))
+```
+
+### Phase 5: Update Query Permission Checks
+
+Places that check "can query this table" should use `mi/can-query?` or `api/query-check`:
+
+| File | Current | After |
+|------|---------|-------|
+| Query processor permission middleware | Direct permission checks | `mi/can-query?` |
+| Card query execution | Mix of checks | `mi/can-query?` for underlying tables |
+
+## Files to Modify
+
+### Core Interface
+- `src/metabase/models/interface.clj` - add `can-query?` multimethod
+
+### Model Implementations
+- `src/metabase/warehouse_schema/models/table.clj` - implement `can-query?`, update `can-read?`
+- `src/metabase/warehouse_schema/models/field.clj` - implement `can-query?`, update `can-read?`
+- `src/metabase/warehouses/models/database.clj` - implement `can-query?`, update `can-read?`
+
+### API Layer
+- `src/metabase/api/common.clj` - add `query-check` helper
+
+### Migration of Special Cases
+- `src/metabase/search/impl.clj` - use `mi/can-read?`
+- `src/metabase/warehouse_schema_rest/api/table.clj` - use standard checks
+- `src/metabase/warehouses_rest/api.clj` - use standard checks
+- `enterprise/.../advanced_permissions/common.clj` - use `mi/can-read?`/`mi/can-write?`
+- `enterprise/.../sandbox/api/table.clj` - verify works with updated `can-read?`
+
+## Testing Strategy
+
+1. **Unit tests for `can-query?`** - verify correct permission checks
+2. **Unit tests for updated `can-read?`** - verify all three access paths work
+3. **Integration tests** - verify API endpoints behave correctly
+4. **Regression tests** - ensure no permission escalation or denial
+
+## Permission Matrix After Implementation
+
+| User Has | `can-read?` | `can-query?` | `can-write?` |
+|----------|-------------|--------------|--------------|
+| Data perms (view-data + create-queries) | ✅ | ✅ | ❌ |
+| Manage-table-metadata only | ✅ | ❌ | ✅ |
+| Published table in accessible collection | ✅ | ❌* | ❌ |
+| None of the above | ❌ | ❌ | ❌ |
+
+*Published tables may grant query access depending on EE configuration
+
+## Benefits
+
+1. **Single source of truth** - all permission logic in model interface methods
+2. **Semantic clarity** - `can-read?` means "see metadata", `can-query?` means "execute queries"
+3. **Consistency** - matches how Cards work (collection perms for visibility, data perms for execution)
+4. **Maintainability** - changes to permission logic only need to happen in one place
+5. **Testability** - easier to test permission logic in isolation
diff --git a/PLAN_table_path_in_changes_view.md b/PLAN_table_path_in_changes_view.md
new file mode 100644
index 00000000000..63a88af9519
--- /dev/null
+++ b/PLAN_table_path_in_changes_view.md
@@ -0,0 +1,205 @@
+# Plan: Update Path for Table Children in Git Sync Changes View
+
+## Problem Statement
+
+When displaying changes in the git sync modals, fields and segments are currently grouped only by their collection (inherited from the parent table). The UI doesn't show which table these entities belong to, making it hard for users to understand the context of changes to table children.
+
+**Current behavior:**
+```
+Collection: Root / Sales
+ ├── orders_table (Table) [Create]
+ ├── customer_id (Field) [Create] ← No indication this belongs to orders_table
+ └── active_customers (Segment) [Create] ← No indication this belongs to orders_table
+```
+
+**Desired behavior:**
+```
+Collection: Root / Sales
+ └── orders_table (Table) [Create]
+ ├── customer_id (Field) [Create]
+ └── active_customers (Segment) [Create]
+```
+
+## Implementation Plan
+
+### Step 1: Backend - Add table_id and table_name to dirty state response
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj`
+
+1. Update `build-select-clause` to include `table_id` for models that have it
+2. Update the query to include `table_name` from the joined `metabase_table` for field and segment models
+3. Add new columns to the select clause:
+ - `:table_id` - The ID of the parent table (for fields/segments)
+ - `:table_name` - The name of the parent table (for fields/segments)
+
+Changes needed:
+- Add `:has-table-id` configuration option to model configs
+- Update `build-select-clause` to include table_id when the model has it (fields, segments)
+- Add table_name from the joined metabase_table
+
+**File:** `enterprise/backend/src/metabase_enterprise/remote_sync/schema.clj`
+
+1. Add optional `:table_id` and `:table_name` fields to `DirtyItem` schema
+
+### Step 2: Frontend Types - Update RemoteSyncEntity type
+
+**File:** `frontend/src/metabase-types/api/remote-sync.ts`
+
+1. Add `table` and `field` and `segment` to `RemoteSyncEntityModel` union type
+2. Add optional `table_id?: number` field to `RemoteSyncEntity`
+3. Add optional `table_name?: string` field to `RemoteSyncEntity`
+
+### Step 3: Frontend Utils - Create table grouping utilities
+
+**File:** `enterprise/frontend/src/metabase-enterprise/remote_sync/utils.ts`
+
+1. Add helper function to identify table-child models:
+ ```typescript
+ export const isTableChildModel = (model: string): boolean => {
+ return model === "field" || model === "segment";
+ };
+ ```
+
+2. Add helper to get table info from entity:
+ ```typescript
+ export const getParentTableInfo = (entity: RemoteSyncEntity) => {
+ if (isTableChildModel(entity.model) && entity.table_id) {
+ return { id: entity.table_id, name: entity.table_name };
+ }
+ return null;
+ };
+ ```
+
+### Step 4: Frontend UI - Update AllChangesView to nest table children
+
+**File:** `enterprise/frontend/src/metabase-enterprise/remote_sync/components/ChangesLists/AllChangesView.tsx`
+
+1. Update `groupedData` useMemo to create a two-level hierarchy:
+ - First level: Group by collection (existing)
+ - Second level: Within each collection, separate tables and their children from other items
+
+2. For each collection group:
+ - Identify table entities
+ - Identify field/segment entities that belong to tables (via `table_id`)
+ - Group fields/segments under their parent table
+ - Render table items with nested children
+
+3. Update the rendering to show:
+ ```
+ Collection Path
+ ├── Table Item
+ │ ├── Field Item
+ │ └── Segment Item
+ └── Other Item (Card, Dashboard, etc.)
+ ```
+
+### Step 5: Update EntityLink to handle table children rendering
+
+**File:** `enterprise/frontend/src/metabase-enterprise/remote_sync/components/ChangesLists/EntityLink.tsx`
+
+The EntityLink component may need minor updates to handle the nested indentation visually, or this can be handled in AllChangesView through additional styling.
+
+### Step 6: Test mock utilities
+
+**File:** `frontend/src/metabase-types/api/mocks/remote-sync.ts`
+
+Update `createMockRemoteSyncEntity` to support table_id and table_name.
+
+**File:** `frontend/test/__support__/server-mocks/remote-sync.ts`
+
+Update mock types if needed.
+
+### Step 7: Write tests
+
+Add tests for:
+1. Backend: dirty state query returns correct table_id/table_name for fields and segments
+2. Frontend: grouping logic correctly nests fields/segments under tables
+3. Frontend: UI renders nested structure correctly
+
+## Technical Details
+
+### Backend Query Changes
+
+Current query for field:
+```sql
+SELECT field.id, field.name, ..., metabase_table.collection_id, 'field' as model
+FROM metabase_field field
+INNER JOIN remote_sync_object rs_obj ON ...
+INNER JOIN metabase_table ON field.table_id = metabase_table.id
+```
+
+Updated query:
+```sql
+SELECT field.id, field.name, ...,
+ metabase_table.collection_id,
+ field.table_id,
+ metabase_table.name as table_name,
+ 'field' as model
+FROM metabase_field field
+INNER JOIN remote_sync_object rs_obj ON ...
+INNER JOIN metabase_table ON field.table_id = metabase_table.id
+```
+
+### Frontend Grouping Logic
+
+```typescript
+// Pseudocode for the grouping logic
+const groupedData = useMemo(() => {
+ const byCollection = _.groupBy(entities, (e) => e.collection_id || 0);
+
+ return Object.entries(byCollection).map(([collectionId, items]) => {
+ // Separate tables and their children from other items
+ const tables = items.filter(i => i.model === "table");
+ const tableChildren = items.filter(i => isTableChildModel(i.model));
+ const otherItems = items.filter(i => i.model !== "table" && !isTableChildModel(i.model));
+
+ // Group table children by their parent table
+ const tableChildrenByTable = _.groupBy(tableChildren, (e) => e.table_id);
+
+ // Create table items with their nested children
+ const tableItems = tables.map(table => ({
+ ...table,
+ children: tableChildrenByTable[table.id] || []
+ }));
+
+ // Handle orphan table children (table not in dirty set)
+ const orphanTableIds = Object.keys(tableChildrenByTable)
+ .filter(id => !tables.some(t => t.id === Number(id)));
+
+ const orphanTableGroups = orphanTableIds.map(tableId => ({
+ tableId: Number(tableId),
+ tableName: tableChildrenByTable[tableId][0]?.table_name,
+ children: tableChildrenByTable[tableId]
+ }));
+
+ return {
+ pathSegments: getCollectionPathSegments(...),
+ collectionId,
+ collectionEntity,
+ tableItems, // Tables with their children
+ orphanTableGroups, // Children of tables not in dirty set
+ items: otherItems // Non-table items (cards, dashboards, etc.)
+ };
+ });
+}, [entities, collectionMap]);
+```
+
+## Files to Modify
+
+1. `enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj`
+2. `enterprise/backend/src/metabase_enterprise/remote_sync/schema.clj`
+3. `frontend/src/metabase-types/api/remote-sync.ts`
+4. `enterprise/frontend/src/metabase-enterprise/remote_sync/utils.ts`
+5. `enterprise/frontend/src/metabase-enterprise/remote_sync/components/ChangesLists/AllChangesView.tsx`
+6. `frontend/src/metabase-types/api/mocks/remote-sync.ts` (optional)
+7. `frontend/test/__support__/server-mocks/remote-sync.ts` (optional)
+
+## Considerations
+
+1. **Orphan children**: Fields/segments may be dirty without their parent table being dirty. Need to handle this case by showing a "virtual" table parent grouping using the table_name from the field/segment.
+
+2. **Performance**: The grouping logic adds complexity but operates on already-fetched data, so impact should be minimal.
+
+3. **Backwards compatibility**: Adding optional fields to the API response is backward compatible.
+
+4. **UI clarity**: The nested structure should make it clearer which table fields/segments belong to, improving user understanding of changes.
diff --git a/PLAN_transforms_collection_dirty_state.md b/PLAN_transforms_collection_dirty_state.md
new file mode 100644
index 00000000000..a09fce86a2e
--- /dev/null
+++ b/PLAN_transforms_collection_dirty_state.md
@@ -0,0 +1,153 @@
+# Plan: Fix Transforms Collections Dirty State Issues
+
+## Problem Summary
+
+Two issues exist with transforms collections in DataStudio:
+
+1. **Creating new transforms collections doesn't trigger dirty state rechecking**
+2. **Transforms collections don't show in the changes view hierarchy**
+
+---
+
+## Issue 1: Dirty State Not Invalidated for Transforms Collections
+
+### Root Cause
+
+When a transforms collection is created, the frontend middleware doesn't invalidate the dirty state cache.
+
+**Location**: `enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/remote-sync-listener-middleware.ts:239-248`
+
+```typescript
+remoteSyncListenerMiddleware.startListening({
+ matcher: collectionApi.endpoints.createCollection.matchFulfilled,
+ effect: async (action: PayloadAction<Collection>, { dispatch }) => {
+ const collection = action.payload;
+
+ if (collection.is_remote_synced) { // ← BUG: Only checks is_remote_synced
+ invalidateRemoteSyncTags(dispatch);
+ }
+ },
+});
+```
+
+The backend tracks transforms-namespace collections differently:
+- Regular collections are tracked when `is_remote_synced = true`
+- Transforms collections are tracked when `namespace = "transforms"` AND `remote-sync-transforms` setting is enabled
+
+See `spec.clj:328-345`:
+```clojure
+(defmethod check-eligibility :collection
+ [{:keys [eligibility]} object]
+ (case (:collection eligibility)
+ :any
+ (or (collections/remote-synced-collection? (or collection-id object))
+ (and (rs-settings/remote-sync-transforms)
+ (transforms-namespace-collection? object)))))
+```
+
+### Fix
+
+Modify the middleware to also invalidate tags when creating a transforms-namespace collection:
+
+```typescript
+// remote-sync-listener-middleware.ts
+
+remoteSyncListenerMiddleware.startListening({
+ matcher: collectionApi.endpoints.createCollection.matchFulfilled,
+ effect: async (action: PayloadAction<Collection>, { dispatch, getState }) => {
+ const collection = action.payload;
+
+ // Check if this is a remote-synced collection
+ const isRemoteSynced = collection.is_remote_synced;
+
+ // Check if this is a transforms-namespace collection with transforms sync enabled
+ const state = getState();
+ const isTransformsSyncEnabled = getSetting(state, "remote-sync-transforms");
+ const isTransformsCollection = collection.namespace === "transforms";
+
+ if (isRemoteSynced || (isTransformsCollection && isTransformsSyncEnabled)) {
+ invalidateRemoteSyncTags(dispatch);
+ }
+ },
+});
+```
+
+Similar changes needed for:
+- `collectionApi.endpoints.updateCollection.matchFulfilled` (lines 250-266)
+- `collectionApi.endpoints.deleteCollection.matchFulfilled` (lines 268-286)
+
+---
+
+## Issue 2: Transforms Collections Not in Changes View Hierarchy
+
+### Root Cause
+
+The `AllChangesView` component fetches the collection tree without including the "transforms" namespace.
+
+**Location**: `enterprise/frontend/src/metabase-enterprise/remote_sync/components/ChangesLists/AllChangesView.tsx:51-58`
+
+```typescript
+const { data: collectionTree = [] } = useListCollectionsTreeQuery({
+ namespaces: [
+ "",
+ "analytics",
+ ...(isUsingTenants ? ["shared-tenant-collection"] : []),
+ ],
+ "include-library": true,
+});
+```
+
+When transforms entities are dirty, they have a `collection_id` that points to a transforms-namespace collection. Since this collection isn't in the tree, `buildCollectionMap` can't resolve it, and the path segments are wrong/empty.
+
+### Fix
+
+Add "transforms" to the namespaces when `remote-sync-transforms` setting is enabled:
+
+```typescript
+// AllChangesView.tsx
+
+const isTransformsSyncEnabled = useSetting("remote-sync-transforms");
+
+const { data: collectionTree = [] } = useListCollectionsTreeQuery({
+ namespaces: [
+ "",
+ "analytics",
+ ...(isUsingTenants ? ["shared-tenant-collection"] : []),
+ ...(isTransformsSyncEnabled ? ["transforms"] : []),
+ ],
+ "include-library": true,
+});
+```
+
+---
+
+## Files to Modify
+
+1. **`enterprise/frontend/src/metabase-enterprise/remote_sync/middleware/remote-sync-listener-middleware.ts`**
+ - Add state access to collection create/update/delete listeners
+ - Check for transforms-namespace collections in addition to `is_remote_synced`
+
+2. **`enterprise/frontend/src/metabase-enterprise/remote_sync/components/ChangesLists/AllChangesView.tsx`**
+ - Add `useSetting("remote-sync-transforms")` hook
+ - Include "transforms" namespace conditionally in the collection tree query
+
+---
+
+## Testing
+
+1. **Dirty state invalidation test**:
+ - Enable remote-sync-transforms setting
+ - Create a new transforms collection
+ - Verify the dirty badge appears immediately without focus/refetch
+
+2. **Changes view hierarchy test**:
+ - Create transforms and transforms collections
+ - Open the changes view
+ - Verify transforms collections appear with correct hierarchy/path
+
+---
+
+## Implementation Order
+
+1. Fix Issue 2 first (AllChangesView) - simpler change, one file
+2. Fix Issue 1 (middleware) - more involved, needs state access pattern
diff --git a/PLAN_transforms_remote_sync_toggle.md b/PLAN_transforms_remote_sync_toggle.md
new file mode 100644
index 00000000000..b71013a6f21
--- /dev/null
+++ b/PLAN_transforms_remote_sync_toggle.md
@@ -0,0 +1,259 @@
+# Plan: Add Transforms Remote-Sync Toggle Setting
+
+## Overview
+
+Add a toggle setting to enable/disable syncing transforms via remote-sync. The toggle should appear in:
+1. The admin remote-sync settings page (`/admin/settings/remote-sync`)
+2. The "Set up git sync" modal in the data studio
+
+The backend setting `remote-sync-transforms` already exists in `settings.clj`, but is not exposed through the API or frontend.
+
+## Current State
+
+### Backend
+- **Setting exists**: `remote-sync-transforms` in `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj:89-95`
+ - Type: boolean, default: false, visibility: admin
+- **Not in API**: The PUT `/api/ee/remote-sync/settings` endpoint does NOT accept `remote-sync-transforms` in its schema
+- **Not in settings update function**: `check-and-update-remote-settings!` doesn't process this setting
+- **Backend tests exist**: Extensive tests in `transforms_test.clj` and `spec_test.clj`
+
+### Frontend
+- **No toggle exists**: `RemoteSyncSettingsForm.tsx` has no transforms toggle
+- **No constant**: `constants.ts` doesn't define a key for transforms
+- **No type definition**: `EnterpriseSettings` interface doesn't include `remote-sync-transforms`
+- **No schema validation**: `REMOTE_SYNC_SCHEMA` doesn't include transforms
+
+## Implementation Plan
+
+### Step 1: Backend API Updates
+
+**File: `enterprise/backend/src/metabase_enterprise/remote_sync/api.clj`**
+
+Update the PUT `/api/ee/remote-sync/settings` endpoint schema (lines 111-121) to accept `remote-sync-transforms`:
+
+```clojure
+:- [:map
+ [:remote-sync-url {:optional true} [:maybe :string]]
+ [:remote-sync-token {:optional true} [:maybe :string]]
+ [:remote-sync-type {:optional true} [:maybe [:enum :read-only :read-write]]]
+ [:remote-sync-branch {:optional true} [:maybe :string]]
+ [:remote-sync-auto-import {:optional true} [:maybe :boolean]]
+ [:remote-sync-transforms {:optional true} [:maybe :boolean]] ;; ADD THIS
+ [:collections {:optional true} [:maybe [:map-of pos-int? :boolean]]]]
+```
+
+**File: `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj`**
+
+Update `check-and-update-remote-settings!` function (line 144) to include `:remote-sync-transforms` in the settings keys list:
+
+```clojure
+(doseq [k [:remote-sync-url :remote-sync-token :remote-sync-type
+ :remote-sync-branch :remote-sync-auto-import :remote-sync-transforms]]
+ ...)
+```
+
+### Step 2: Frontend Type Definitions
+
+**File: `frontend/src/metabase-types/api/settings.ts`**
+
+Add to `EnterpriseSettings` interface (around line 678):
+
+```typescript
+"remote-sync-transforms"?: boolean | null;
+```
+
+**File: `frontend/src/metabase-types/api/remote-sync.ts`**
+
+Update `RemoteSyncConfigurationSettings` type (lines 74-83) to include transforms:
+
+```typescript
+export type RemoteSyncConfigurationSettings = Pick<
+ EnterpriseSettings,
+ | "remote-sync-enabled"
+ | "remote-sync-url"
+ | "remote-sync-token"
+ | "remote-sync-type"
+ | "remote-sync-branch"
+ | "remote-sync-transforms" // ADD THIS
+> & {
+ collections?: CollectionSyncPreferences;
+};
+```
+
+### Step 3: Frontend Constants and Schema
+
+**File: `enterprise/frontend/src/metabase-enterprise/remote_sync/constants.ts`**
+
+Add new constant and update schema:
+
+```typescript
+export const TRANSFORMS_KEY = "remote-sync-transforms";
+
+export const REMOTE_SYNC_SCHEMA = Yup.object({
+ [REMOTE_SYNC_KEY]: Yup.boolean().nullable().default(true),
+ [URL_KEY]: Yup.string().nullable().default(null),
+ [TOKEN_KEY]: Yup.string().nullable().default(null),
+ [AUTO_IMPORT_KEY]: Yup.boolean().nullable().default(false),
+ [TRANSFORMS_KEY]: Yup.boolean().nullable().default(false), // ADD THIS
+ [TYPE_KEY]: Yup.string()
+ .oneOf(["read-only", "read-write"] as const)
+ .nullable()
+ .default("read-only"),
+ [BRANCH_KEY]: Yup.string().nullable().default("main"),
+ [COLLECTIONS_KEY]: Yup.object().nullable().default({}),
+});
+```
+
+### Step 4: Frontend Settings Form UI
+
+**File: `enterprise/frontend/src/metabase-enterprise/remote_sync/components/RemoteSyncAdminSettings/RemoteSyncSettingsForm.tsx`**
+
+1. Import the new constant:
+```typescript
+import {
+ AUTO_IMPORT_KEY,
+ BRANCH_KEY,
+ COLLECTIONS_KEY,
+ REMOTE_SYNC_KEY,
+ REMOTE_SYNC_SCHEMA,
+ TOKEN_KEY,
+ TRANSFORMS_KEY, // ADD THIS
+ TYPE_KEY,
+ URL_KEY,
+} from "../../constants";
+```
+
+2. Add a new section for transforms toggle. The toggle should appear after the "Collections to sync" section (around line 432). Add a new section:
+
+```tsx
+{/* Section 5: Transforms sync (when enabled and read-write mode) */}
+{isRemoteSyncEnabled && (
+ <RemoteSyncSettingsSection
+ title={t`Transforms`}
+ variant={variant}
+ >
+ <FormSwitch
+ label={t`Sync transforms with git`}
+ description={t`When enabled, all transforms, transform tags, and transform jobs will be synced.`}
+ name={TRANSFORMS_KEY}
+ size="sm"
+ />
+ </RemoteSyncSettingsSection>
+)}
+```
+
+**Design Considerations:**
+- The toggle should only appear when remote-sync is enabled (similar to the Collections section)
+- The toggle should work in both read-only and read-write modes
+- Include a description explaining what transforms syncing includes
+
+### Step 5: Initial Values in Form
+
+Update the `initialValues` useMemo in `RemoteSyncSettingsForm.tsx` to include the transforms setting from server data (the schema cast will handle this automatically since it's already in settingValues).
+
+No code change needed - the `REMOTE_SYNC_SCHEMA.cast(settingValues, { stripUnknown: true })` will automatically include the transforms setting once it's added to the schema.
+
+### Step 6: Tests
+
+#### Backend Tests
+
+**File: `enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj`**
+
+Add test cases for the settings endpoint accepting `remote-sync-transforms`:
+
+```clojure
+(deftest update-settings-transforms-test
+ (testing "PUT /api/ee/remote-sync/settings can update remote-sync-transforms"
+ (mt/with-temporary-setting-values [remote-sync-url "https://github.com/test/repo.git"
+ remote-sync-transforms false]
+ (mt/user-http-request :crowberto :put 200 "ee/remote-sync/settings"
+ {:remote-sync-transforms true})
+ (is (true? (remote-sync.settings/remote-sync-transforms))))))
+```
+
+#### Frontend Unit Tests
+
+**File: `enterprise/frontend/src/metabase-enterprise/remote_sync/components/RemoteSyncAdminSettings/RemoteSyncSettingsForm.unit.spec.tsx`**
+
+Add test cases:
+
+```typescript
+describe("Transforms sync toggle", () => {
+ it("should display transforms toggle when remote sync is enabled", async () => {
+ await setup({ isRemoteSyncEnabled: true });
+ expect(screen.getByText("Sync transforms with git")).toBeInTheDocument();
+ });
+
+ it("should not display transforms toggle when remote sync is disabled", async () => {
+ await setup({ isRemoteSyncEnabled: false });
+ expect(screen.queryByText("Sync transforms with git")).not.toBeInTheDocument();
+ });
+
+ it("should include transforms setting in form submission", async () => {
+ const { onSubmit } = await setup({ isRemoteSyncEnabled: true });
+
+ const transformsSwitch = screen.getByRole("switch", { name: /sync transforms/i });
+ await userEvent.click(transformsSwitch);
+
+ await userEvent.click(screen.getByTestId("remote-sync-submit-button"));
+
+ expect(onSubmit).toHaveBeenCalledWith(
+ expect.objectContaining({
+ "remote-sync-transforms": true,
+ })
+ );
+ });
+});
+```
+
+#### E2E Tests (Optional)
+
+**File: `e2e/test/scenarios/admin/remote-sync.cy.spec.ts`**
+
+Add E2E test for the transforms toggle:
+
+```typescript
+describe("Transforms sync toggle", () => {
+ it("should allow enabling/disabling transforms sync", () => {
+ cy.visit("/admin/settings/remote-sync");
+
+ // Enable remote sync first
+ cy.findByLabelText("Repository URL").type("https://github.com/test/repo.git");
+ cy.findByLabelText("Access Token").type("test-token");
+ cy.findByText("Set up Remote Sync").click();
+
+ // Toggle transforms
+ cy.findByText("Sync transforms with git").should("be.visible");
+ cy.findByRole("switch", { name: /sync transforms/i }).click();
+ cy.findByText("Save changes").click();
+
+ // Verify saved
+ cy.reload();
+ cy.findByRole("switch", { name: /sync transforms/i }).should("be.checked");
+ });
+});
+```
+
+## File Change Summary
+
+| File | Change Type | Description |
+|------|-------------|-------------|
+| `enterprise/backend/src/metabase_enterprise/remote_sync/api.clj` | Modify | Add `remote-sync-transforms` to API schema |
+| `enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj` | Modify | Add transforms to `check-and-update-remote-settings!` |
+| `frontend/src/metabase-types/api/settings.ts` | Modify | Add `remote-sync-transforms` to EnterpriseSettings |
+| `frontend/src/metabase-types/api/remote-sync.ts` | Modify | Add transforms to RemoteSyncConfigurationSettings |
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/constants.ts` | Modify | Add TRANSFORMS_KEY and update schema |
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/components/RemoteSyncAdminSettings/RemoteSyncSettingsForm.tsx` | Modify | Add transforms toggle UI |
+| `enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj` | Modify | Add tests for transforms setting |
+| `enterprise/frontend/src/metabase-enterprise/remote_sync/components/RemoteSyncAdminSettings/RemoteSyncSettingsForm.unit.spec.tsx` | Modify | Add tests for transforms toggle |
+| `e2e/test/scenarios/admin/remote-sync.cy.spec.ts` | Modify (Optional) | Add E2E tests |
+
+## Notes
+
+1. **Git Settings Modal**: Since `GitSettingsModal` uses `RemoteSyncSettingsForm` with `variant="settings-modal"`, the transforms toggle will automatically appear in the modal once added to the form component. No separate changes needed for the modal.
+
+2. **Read-only vs Read-write**: The transforms toggle should be available in both modes since transforms can be synced in either direction.
+
+3. **Existing Backend Logic**: The backend already has comprehensive logic for handling transforms sync based on the `remote-sync-transforms` setting. This plan only needs to expose the setting through the API and UI.
+
+4. **Backwards Compatibility**: Setting defaults to `false`, so existing installations won't have transforms sync enabled automatically.
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/api.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/api.clj
index 53c983f0e1f..7a297d18b15 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/api.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/api.clj
@@ -51,7 +51,7 @@
to the remote sync source."
[]
(api/check-superuser)
- {:is_dirty (remote-sync.object/dirty-global?)})
+ {:is_dirty (remote-sync.object/dirty?)})
(api.macros/defendpoint :get "/has-remote-changes" :- remote-sync.schema/HasRemoteChangesResponse
"Check if there are new changes on the remote branch that can be pulled.
@@ -80,7 +80,7 @@
(api/check-superuser)
{:dirty (into []
(m/distinct-by (juxt :id :model))
- (remote-sync.object/dirty-for-global))})
+ (remote-sync.object/dirty-objects))})
(api.macros/defendpoint :post "/export" :- remote-sync.schema/ExportResponse
"Export the current state of the Remote Sync collection to a Source.
@@ -140,7 +140,7 @@
[:remote-sync-branch {:optional true} [:maybe :string]]
[:collections {:optional true} [:maybe [:map-of pos-int? :boolean]]]]]
(api/check-superuser)
- (api/check-400 (not (and (remote-sync.object/dirty-global?) (= :read-only remote-sync-type)))
+ (api/check-400 (not (and (remote-sync.object/dirty?) (= :read-only remote-sync-type)))
"There are unsaved changes in the Remote Sync collection which will be overwritten switching to read-only mode.")
;; Check if trying to change collections while in read-only mode
(let [effective-type (or remote-sync-type (settings/remote-sync-type))]
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/core.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/core.clj
index 8fae4bc6a11..b2207b335e6 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/core.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/core.clj
@@ -53,6 +53,18 @@
(not (collections/remote-synced-collection? (or (:collection table)
(:collection_id table))))))
+(defenterprise transforms-editable?
+ "Determines if transforms should be editable.
+
+ Returns true if transforms are editable, false otherwise. Transforms are globally
+ read-only when remote-sync is enabled and remote-sync-type is :read-only.
+
+ Always returns true on OSS."
+ :feature :none
+ []
+ (or (not (settings/remote-sync-enabled))
+ (= (settings/remote-sync-type) :read-write)))
+
(mu/defn bulk-set-remote-sync :- :nil
"Sets remote sync to true/false on one or collections in a single transaction. Checks that the remote sync state
afterwards is consistent in terms of dependency rules. Collections are provided as a map of collection-id -> sync state."
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/events.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/events.clj
index 2af78510ca6..3e0dc052769 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/events.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/events.clj
@@ -6,17 +6,18 @@
This enables the remote sync system to know which objects have changed
since the last sync operation.
- Handlers are defined using the `defmodel-change-handler` macro which:
- - Derives event hierarchies for create/update/delete events
- - Checks if the model is in a remote-synced collection
- - Creates or updates RemoteSyncObject entries with denormalized model details
+ Event handlers are registered using specs from `metabase-enterprise.remote-sync.spec`.
+ Each spec defines eligibility checking, field hydration, and other configuration.
Tracked model types:
- Card, Dashboard, Document, NativeQuerySnippet, Timeline, Collection
- Table (when published in a remote-synced collection)
- - Field, Segment (when belonging to a published table in a remote-synced collection)"
+ - Field, Segment (when belonging to a published table in a remote-synced collection)
+ - Transform, TransformTag, transforms-namespace Collections (when remote-sync-transforms setting is enabled)"
(:require
[java-time.api :as t]
+ [metabase-enterprise.remote-sync.settings :as settings]
+ [metabase-enterprise.remote-sync.spec :as spec]
[metabase.collections.core :as collections]
[metabase.events.core :as events]
[metabase.util.log :as log]
@@ -25,13 +26,6 @@
;; Helper functions for model change tracking
-(defn- model-in-remote-synced-collection?
- "Checks if a model (card, dashboard, document) is in a remote-synced collection. Takes a model instance with a
- collection_id field. Returns true if the model is in a remote-synced collection, false otherwise."
- [{:keys [collection_id]}]
- (boolean
- (collections/remote-synced-collection? collection_id)))
-
(defn- create-or-update-remote-sync-object-entry!
"Creates or updates a remote sync object entry for a model change.
@@ -79,92 +73,100 @@
:model_table_id (:table_id model-details)
:model_table_name (:table_name model-details)})))))
-;; Model change tracking event handlers
-
-(defmacro ^:private defmodel-change-handler
- "Defines event derivations and handler for a standard model change event.
-
- Usage:
- (defmodel-change-handler card
- {:model-type \"Card\"
- :event-prefix :event/card
- :log-name \"card\"
- :hydrate-details (fn [id] (t2/select-one [:model/Card :name :collection_id :display] :id id))})
-
- Configuration options:
- - :model-type - String for RemoteSyncObject model_type (e.g. \"Card\")
- - :event-prefix - Keyword prefix for events (e.g. :event/card)
- - :log-name - String for log messages (e.g. \"card\")
- - :hydrate-details - Function (fn [model-id]) that returns map with :name, :collection_id,
- and optionally :display, :table_id, :table_name
- - :archived-key - Key to check for archived status (default :archived)
- - :in-sync-pred - Predicate fn (default model-in-remote-synced-collection?)"
- [event-group {:keys [model-type event-prefix log-name hydrate-details archived-key in-sync-pred]
- :or {archived-key :archived
- in-sync-pred `model-in-remote-synced-collection?}}]
- (let [parent-kw (keyword (str *ns*) (str (name event-group) "-change-event"))
- create-kw (keyword (namespace event-prefix) (str (name event-prefix) "-create"))
- update-kw (keyword (namespace event-prefix) (str (name event-prefix) "-update"))
- delete-kw (keyword (namespace event-prefix) (str (name event-prefix) "-delete"))]
- `(do
- (derive ~parent-kw :metabase/event)
- (derive ~create-kw ~parent-kw)
- (derive ~update-kw ~parent-kw)
- (derive ~delete-kw ~parent-kw)
-
- (methodical/defmethod events/publish-event! ~parent-kw
- [topic# event#]
- (let [{:keys [~'object]} event#
- in-remote-synced?# (~in-sync-pred ~'object)
- existing-entry# (t2/select-one :model/RemoteSyncObject
- :model_type ~model-type
- :model_id (:id ~'object))
- status# (if (get ~'object ~archived-key)
- "delete"
- (case topic#
- ~create-kw "create"
- ~update-kw "update"
- ~delete-kw "delete"))]
- (cond
- in-remote-synced?#
- (do
- (log/infof "Creating remote sync object entry for %s %s (status: %s)"
- ~log-name (:id ~'object) status#)
- (create-or-update-remote-sync-object-entry!
- ~model-type (:id ~'object) status# ~hydrate-details))
+;;; ----------------------------------------- Spec-based Event Handling ------------------------------------------------
- (and existing-entry# (not in-remote-synced?#))
- (do
- (log/infof "%s %s moved out of remote-synced collection, marking as removed"
- ~log-name (:id ~'object))
- (create-or-update-remote-sync-object-entry!
- ~model-type (:id ~'object) "removed" ~hydrate-details))))))))
-
-;; Standard model change handlers
+(defn- create-or-update-sync-object-from-spec!
+ "Creates or updates a RemoteSyncObject entry using a spec for field hydration.
+ This is the spec-based version of create-or-update-remote-sync-object-entry!."
+ [model-spec model-id status]
+ (let [model-type (:model-type model-spec)
+ existing (t2/select-one :model/RemoteSyncObject :model_type model-type :model_id model-id)]
+ (cond
+ ;; No existing entry - create new one
+ (not existing)
+ (let [model-details (spec/hydrate-model-details model-spec model-id)
+ fields (spec/build-sync-object-fields model-spec model-details)]
+ (t2/insert! :model/RemoteSyncObject
+ (merge {:model_type model-type
+ :model_id model-id
+ :status status
+ :status_changed_at (t/offset-date-time)}
+ fields)))
-(defmodel-change-handler :card
- {:model-type "Card"
- :event-prefix :event/card
- :log-name "card"
- :hydrate-details (fn [id] (t2/select-one [:model/Card :name :collection_id :display] :id id))})
+ ;; Created then removed/deleted before sync - delete the entry
+ (and (= "create" (:status existing)) (contains? #{"removed" "delete"} status))
+ (t2/delete! :model/RemoteSyncObject (:id existing))
-(defmodel-change-handler :dashboard
- {:model-type "Dashboard"
- :event-prefix :event/dashboard
- :log-name "dashboard"
- :hydrate-details (fn [id] (t2/select-one [:model/Dashboard :name :collection_id] :id id))})
+ ;; Already deleted - just update status
+ (= "delete" (:status existing))
+ (t2/update! :model/RemoteSyncObject (:id existing)
+ {:status status
+ :status_changed_at (t/offset-date-time)})
-(defmodel-change-handler :document
- {:model-type "Document"
- :event-prefix :event/document
- :log-name "document"
- :hydrate-details (fn [id] (t2/select-one [:model/Document :name :collection_id] :id id))})
+ ;; Status is not "create" - update with new details
+ (not= "create" (:status existing))
+ (let [model-details (spec/hydrate-model-details model-spec model-id)
+ fields (spec/build-sync-object-fields model-spec model-details)]
+ (t2/update! :model/RemoteSyncObject (:id existing)
+ (merge {:status status
+ :status_changed_at (t/offset-date-time)}
+ fields))))))
+
+(defn- handle-model-event-from-spec
+ "Generic event handler that uses a spec for all configuration.
+ Checks eligibility, determines status, and creates/updates the sync object."
+ [model-spec topic {:keys [object]}]
+ (let [model-type (:model-type model-spec)
+ model-id (:id object)
+ eligible? (spec/check-eligibility model-spec object)
+ existing-entry (t2/select-one :model/RemoteSyncObject :model_type model-type :model_id model-id)
+ status (spec/determine-status model-spec topic object)]
+ (cond
+ ;; Model is eligible for sync
+ eligible?
+ (do
+ (log/infof "Creating remote sync object entry for %s %s (status: %s)"
+ model-type model-id status)
+ (create-or-update-sync-object-from-spec! model-spec model-id status))
-(defmodel-change-handler :snippet
- {:model-type "NativeQuerySnippet"
- :event-prefix :event/snippet
- :log-name "snippet"
- :hydrate-details (fn [id] (t2/select-one [:model/NativeQuerySnippet :name :id] :id id))})
+ ;; Model was synced but no longer eligible - mark as removed
+ (and existing-entry (not eligible?))
+ (do
+ (log/infof "%s %s moved out of sync scope, marking as removed" model-type model-id)
+ (create-or-update-sync-object-from-spec! model-spec model-id "removed")))))
+
+(defn- register-events-for-spec!
+ "Registers event handlers for a single spec. Creates event hierarchy and
+ registers a methodical handler for the parent event."
+ [model-spec]
+ (let [event-kws (spec/event-keywords model-spec)
+ parent-kw (:parent event-kws)]
+ ;; Derive event hierarchy
+ (derive parent-kw :metabase/event)
+ (doseq [[_event-type event-kw] (dissoc event-kws :parent)]
+ (derive event-kw parent-kw))
+
+ ;; Register the handler using methodical's runtime API
+ ;; (defmethod requires compile-time dispatch values, so we use add-primary-method!)
+ ;; Note: We use #'events/publish-event! (the var) because add-primary-method! expects a var/atom
+ ;; The method signature matches defmethod: [topic event], not [next-method topic event]
+ (methodical/add-primary-method!
+ #'events/publish-event!
+ parent-kw
+ (fn [topic event]
+ (handle-model-event-from-spec model-spec topic event)))))
+
+;;; --------------------------------- Spec-based Event Registration (Non-Collection) -----------------------------------
+;; Register event handlers for models that use standard spec-based handling.
+;; Collection has special handling and is registered separately below.
+
+(doseq [[_model-key model-spec] (dissoc spec/remote-sync-specs :model/Collection)]
+ (register-events-for-spec! model-spec))
+
+;;; ----------------------------------------- Collection Event Handler -------------------------------------------------
+;; Collection has special handling due to side effects (tracking published tables when
+;; a collection becomes remote-synced). This handler is kept separate from the standard
+;; spec-based registration.
;; Collection create/update events - derive from common parent for shared handling
(derive ::collection-change-event :metabase/event)
@@ -176,6 +178,18 @@
[id]
(t2/select-one [:model/Collection :name [:id :collection_id]] :id id))
+(defn- transforms-namespace-collection?
+ "Check if this is a transforms-namespace collection."
+ [collection]
+ (= (keyword (:namespace collection)) :transforms))
+
+(defn- should-sync-collection?
+ "Check if a collection should be synced - either remote-synced or transforms-namespace with setting enabled."
+ [collection]
+ (or (collections/remote-synced-collection? collection)
+ (and (settings/remote-sync-transforms)
+ (transforms-namespace-collection? collection))))
+
(defn- hydrate-table-details
"Hydrates details for a Table. For tables, table_id and table_name refer to themselves."
[id]
@@ -203,96 +217,36 @@
(methodical/defmethod events/publish-event! ::collection-change-event
[topic event]
(let [{:keys [object]} event
+ should-sync? (should-sync-collection? object)
is-remote-synced? (collections/remote-synced-collection? object)
existing-entry (t2/select-one :model/RemoteSyncObject :model_type "Collection" :model_id (:id object))
- was-remote-synced? (and existing-entry
- (not (contains? #{"removed"} (:status existing-entry))))
+ was-synced? (and existing-entry
+ (not (contains? #{"removed"} (:status existing-entry))))
status (if (:archived object)
"delete"
(case topic
:event/collection-create "create"
:event/collection-update "update"))]
(cond
- ;; Collection is remote-synced - create or update entry
- is-remote-synced?
+ ;; Collection should be synced (remote-synced or transforms-namespace with setting enabled)
+ should-sync?
(do
(log/infof "Creating remote sync object entry for collection %s (status: %s)" (:id object) status)
(create-or-update-remote-sync-object-entry! "Collection" (:id object) status hydrate-collection-details)
;; If collection just became remote-synced, track all published tables in it
- (when (not was-remote-synced?)
- (track-published-tables-in-collection! (:id object))))
-
- ;; Collection was remote-synced but type changed - mark as removed
- (and existing-entry (not is-remote-synced?))
+ ;; (only for actual remote-synced collections, not transforms-namespace)
+ (when (and is-remote-synced? (not was-synced?))
+ (track-published-tables-in-collection! (:id object)))
+ ;; When a transforms-namespace collection is archived, also mark child transforms for deletion
+ (when (and (:archived object) (transforms-namespace-collection? object))
+ (let [transform-ids (t2/select-pks-set :model/Transform :collection_id (:id object))]
+ (doseq [transform-id transform-ids]
+ (log/infof "Marking transform %s for deletion (parent collection archived)" transform-id)
+ (create-or-update-remote-sync-object-entry! "Transform" transform-id "delete"
+ (fn [id] (t2/select-one [:model/Transform :name [:collection_id :model_collection_id]] :id id)))))))
+
+ ;; Collection was synced but no longer should be - mark as removed
+ (and existing-entry (not should-sync?))
(do
- (log/infof "Collection %s type changed from remote-synced, marking as removed" (:id object))
+ (log/infof "Collection %s no longer needs syncing, marking as removed" (:id object))
(create-or-update-remote-sync-object-entry! "Collection" (:id object) "removed" hydrate-collection-details)))))
-
-(defmodel-change-handler :timeline
- {:model-type "Timeline"
- :event-prefix :event/timeline
- :log-name "timeline"
- :hydrate-details (fn [id] (t2/select-one [:model/Timeline :name :collection_id] :id id))})
-
-;; Table events - only track published tables in remote-synced collections
-
-(defn- published-table-in-remote-synced-collection?
- "Check if a table is published AND in a remote-synced collection.
- Tables are only considered part of remote sync when they are both published
- and their collection_id points to a remote-synced collection."
- [{:keys [is_published collection_id]}]
- (boolean
- (and is_published
- (collections/remote-synced-collection? collection_id))))
-
-(defn- model-in-published-table-in-remote-synced-collection?
- "Check if a model (field, segment) belongs to a published table in a remote-synced collection.
- The model must have a :table_id that points to a table with :is_published true
- and :collection_id in a remote-synced collection."
- [{:keys [table_id]}]
- (boolean
- (when table_id
- (when-let [table (t2/select-one :model/Table :id table_id)]
- (published-table-in-remote-synced-collection? table)))))
-
-(defmodel-change-handler :table
- {:model-type "Table"
- :event-prefix :event/table
- :log-name "table"
- :archived-key :archived_at
- :in-sync-pred published-table-in-remote-synced-collection?
- :hydrate-details hydrate-table-details})
-
-;; Segment events - track segments in published tables in remote-synced collections
-
-(defn- hydrate-segment-details
- "Hydrates details for a Segment, including parent table info."
- [id]
- (first (t2/query {:select [:s.name :s.table_id [:t.collection_id :collection_id] [:t.name :table_name]]
- :from [[:segment :s]]
- :join [[:metabase_table :t] [:= :s.table_id :t.id]]
- :where [:= :s.id id]})))
-
-(defmodel-change-handler :segment
- {:model-type "Segment"
- :event-prefix :event/segment
- :log-name "segment"
- :in-sync-pred model-in-published-table-in-remote-synced-collection?
- :hydrate-details hydrate-segment-details})
-
-;; Field events - track fields in published tables in remote-synced collections
-
-(defn- hydrate-field-details
- "Hydrates details for a Field, including parent table info."
- [id]
- (first (t2/query {:select [:f.name :f.table_id [:t.collection_id :collection_id] [:t.name :table_name]]
- :from [[:metabase_field :f]]
- :join [[:metabase_table :t] [:= :f.table_id :t.id]]
- :where [:= :f.id id]})))
-
-(defmodel-change-handler :field
- {:model-type "Field"
- :event-prefix :event/field
- :log-name "field"
- :in-sync-pred model-in-published-table-in-remote-synced-collection?
- :hydrate-details hydrate-field-details})
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj
index c51b848c5a3..b27c79cbc70 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/impl.clj
@@ -9,6 +9,7 @@
[metabase-enterprise.remote-sync.source :as source]
[metabase-enterprise.remote-sync.source.ingestable :as source.ingestable]
[metabase-enterprise.remote-sync.source.protocol :as source.p]
+ [metabase-enterprise.remote-sync.spec :as spec]
[metabase-enterprise.serialization.core :as serialization]
[metabase.analytics.core :as analytics]
[metabase.api.common :as api]
@@ -27,105 +28,17 @@
[]
(t2/select-pks-vec :model/Collection :is_remote_synced true))
-(defn- model-fields-for-sync
- "Returns the fields to select for a given model type when syncing."
- [model-name]
- (case model-name
- "Card" [:id :name :collection_id :display]
- "NativeQuerySnippet" [:id :name]
- "Collection" [:id :name [:id :collection_id]]
- "Segment" [:id :name :table_id]
- [:id :name :collection_id]))
-
(defn- sync-objects!
"Populates the remote-sync-object table with imported entities. Deletes all existing RemoteSyncObject records and
inserts new ones for each imported entity, marking them as 'synced' with the given timestamp.
Takes a timestamp instant, a map of imported entities grouped by model name (for standard models with entity_id),
- and separate path collections for Table and Field models which use name-based identity."
+ and separate path collections for Table and Field models which use name-based identity.
+
+ Uses the spec system to determine how to query and build sync objects for each model type."
[timestamp imported-entities-by-model table-paths field-paths]
(t2/delete! :model/RemoteSyncObject)
- (let [;; Standard models use entity_id UUIDs (excluding Segment which needs special handling)
- ;; TODO(edpaget 2026-01-19): Actions are not dirty tracked
- standard-inserts (->> (dissoc imported-entities-by-model "Segment" "Action")
- (mapcat (fn [[model entity-ids]]
- (when (seq entity-ids)
- (let [fields (model-fields-for-sync model)
- model-kw (keyword "model" model)]
- (t2/select (into [model-kw] fields) :entity_id [:in entity-ids])))))
- (map (fn [{:keys [id name collection_id display] :as model}]
- {:model_type (clojure.core/name (t2/model model))
- :model_id id
- :model_name name
- :model_collection_id collection_id
- :model_display (some-> display clojure.core/name)
- :model_table_id nil
- :model_table_name nil
- :status "synced"
- :status_changed_at timestamp})))
- ;; Segments: need to join to get table info
- segment-entity-ids (get imported-entities-by-model "Segment")
- segment-inserts (when (seq segment-entity-ids)
- (->> (t2/query {:select [:s.id :s.name :s.table_id [:t.name :table_name] [:t.collection_id :collection_id]]
- :from [[:segment :s]]
- :join [[:metabase_table :t] [:= :s.table_id :t.id]]
- :where [:in :s.entity_id segment-entity-ids]})
- (map (fn [{:keys [id name table_id table_name collection_id]}]
- {:model_type "Segment"
- :model_id id
- :model_name name
- :model_collection_id collection_id
- :model_display nil
- :model_table_id table_id
- :model_table_name table_name
- :status "synced"
- :status_changed_at timestamp}))))
- ;; Tables: batch lookup using join - include name and collection_id
- ;; For tables, table_id = self id, table_name = self name
- table-inserts (when (seq table-paths)
- (->> (t2/query {:select [:t.id :t.name :t.collection_id]
- :from [[:metabase_table :t]]
- :join [[:metabase_database :db] [:= :db.id :t.db_id]]
- :where (into [:or]
- (for [{:keys [db_name schema table_name]} table-paths]
- [:and
- [:= :db.name db_name]
- (if schema [:= :t.schema schema] [:is :t.schema nil])
- [:= :t.name table_name]]))})
- (map (fn [{:keys [id name collection_id]}]
- {:model_type "Table"
- :model_id id
- :model_name name
- :model_collection_id collection_id
- :model_display nil
- :model_table_id id
- :model_table_name name
- :status "synced"
- :status_changed_at timestamp}))))
- ;; Fields: batch lookup using join through table - include name and collection_id from table
- field-inserts (when (seq field-paths)
- (->> (t2/query {:select [:f.id :f.name :f.table_id [:t.collection_id :collection_id] [:t.name :table_name]]
- :from [[:metabase_field :f]]
- :join [[:metabase_table :t] [:= :t.id :f.table_id]
- [:metabase_database :db] [:= :db.id :t.db_id]]
- :where (into [:or]
- (for [{:keys [db_name schema table_name field_name]} field-paths]
- [:and
- [:= :db.name db_name]
- (if schema [:= :t.schema schema] [:is :t.schema nil])
- [:= :t.name table_name]
- [:= :f.name field_name]]))})
- (map (fn [{:keys [id name table_id table_name collection_id]}]
- {:model_type "Field"
- :model_id id
- :model_name name
- :model_collection_id collection_id
- :model_display nil
- :model_table_id table_id
- :model_table_name table_name
- :status "synced"
- :status_changed_at timestamp}))))
- all-inserts (concat standard-inserts segment-inserts table-inserts field-inserts)]
+ (let [all-inserts (spec/sync-all-entities! timestamp imported-entities-by-model table-paths field-paths)]
(when (seq all-inserts)
(t2/insert! :model/RemoteSyncObject all-inserts))))
@@ -133,30 +46,28 @@
"Deletes any remote sync content that was NOT part of the import.
Takes a sequence of remote-synced collection IDs and a map of imported entities grouped by model name. For each
- model type (Collection, Card, Dashboard, etc.), deletes any entities in the remote-synced collections whose
- entity_id is not in the imported set."
+ entity-id based model, deletes entities whose entity_id is not in the imported set.
+
+ Models with :scope-key in their spec are scoped to synced collections (using :id for Collection, :collection_id
+ for others). Models without :scope-key (like TransformTag) are deleted globally by entity_id."
[synced-collection-ids imported-entities-by-model]
- (when (seq synced-collection-ids)
- (doseq [model [:model/Collection
- :model/Card
- :model/Dashboard
- :model/NativeQuerySnippet
- :model/Timeline
- :model/Document]
- :let [serdes-model (name model)
- entity-ids (get imported-entities-by-model serdes-model [])]]
- (if (= model :model/Collection)
- (t2/delete! :model/Collection
- :id [:in synced-collection-ids]
- ;; if we didn't sync any, then delete all collections in the remote sync
- :entity_id (if (seq entity-ids)
- [:not-in entity-ids]
- :entity_id))
- (t2/delete! model
- :collection_id [:in synced-collection-ids]
- :entity_id (if (seq entity-ids)
- [:not-in entity-ids]
- :entity_id))))))
+ (doseq [[model-key model-spec] (spec/specs-by-identity-type :entity-id)
+ :let [serdes-model (:model-type model-spec)
+ entity-ids (get imported-entities-by-model serdes-model [])
+ entity-id-clause (if (seq entity-ids)
+ [:not-in entity-ids]
+ :entity_id)
+ ;; :scope-key determines collection scoping - nil means global
+ scope-key (get-in model-spec [:removal :scope-key])]]
+ (if scope-key
+ ;; Collection-scoped: delete only within synced collections
+ (when (seq synced-collection-ids)
+ (t2/delete! model-key
+ scope-key [:in synced-collection-ids]
+ :entity_id entity-id-clause))
+ ;; Global: delete by entity_id only
+ (t2/delete! model-key
+ :entity_id entity-id-clause))))
(defn source-error-message
"Constructs user-friendly error messages from remote sync source exceptions.
@@ -226,7 +137,10 @@
:version (source.p/version snapshot)
:message (format "Skipping import: snapshot version %s matches last imported version" snapshot-version)}
(log/infof (:message <>)))
- (let [ingestable-snapshot (->> (source.p/->ingestable snapshot {:path-filters [#"collections/.*" #"databases/.*" #"actions/.*"]})
+ (let [path-filters (cond-> [#"collections/.*" #"databases/.*" #"actions/.*"]
+ (settings/remote-sync-transforms)
+ (conj #"transforms/.*"))
+ ingestable-snapshot (->> (source.p/->ingestable snapshot {:path-filters path-filters})
(source.ingestable/wrap-progress-ingestable task-id 0.7))
load-result (serdes/with-cache
(serialization/load-metabase! ingestable-snapshot))
@@ -290,90 +204,33 @@
exceptions may be thrown during export and are caught and converted to error status maps."
[^SourceSnapshot snapshot task-id message]
(if snapshot
- (let [sync-timestamp (t/instant)
- collections (t2/select-fn-set :entity_id :model/Collection :is_remote_synced true :location "/")]
- (if (empty? collections)
- {:status :error
- :message "No remote-synced collections available to sync."}
- (try
- (analytics/inc! :metabase-remote-sync/exports)
- (serdes/with-cache
- (let [models (serialization/extract {:targets (mapv #(vector "Collection" %) collections)
- :no-collections false
- :no-data-model true
- :no-settings true
- :no-transforms true
- :include-field-values false
- :include-database-secrets false
- :continue-on-error false
- :skip-archived true})]
+ (let [sync-timestamp (t/instant)]
+ (try
+ (analytics/inc! :metabase-remote-sync/exports)
+ (serdes/with-cache
+ (if-let [models (spec/extract-entities-for-export)]
+ (do
(remote-sync.task/update-progress! task-id 0.3)
- (let [top-level-removed-prefixes (->> (t2/query {:select [:c.entity_id]
- :from [[:collection :c]]
- :join [[:remote_sync_object :rso]
- [:and [:= :rso.model_type [:inline "Collection"]]
- [:= :rso.status [:inline "removed"]]
- [:= :rso.model_id :c.id]]]
- :where [:= :location "/"]})
- (map #(str "collections/" (:entity_id %))))
- ;; Add removed table paths - join to get db_name + schema in one query
- removed-table-paths (->> (t2/query {:select [[:t.name :table_name] :t.schema [:db.name :db_name]]
- :from [[:metabase_table :t]]
- :join [[:remote_sync_object :rso]
- [:and [:= :rso.model_type [:inline "Table"]]
- [:= :rso.status [:inline "removed"]]
- [:= :rso.model_id :t.id]]
- [:metabase_database :db]
- [:= :db.id :t.db_id]]})
- (map (fn [{:keys [table_name schema db_name]}]
- (str/join "/" (cond-> ["databases" db_name]
- schema (conj "schemas" schema)
- true (conj "tables" table_name))))))
- ;; Add removed/deleted segment paths - join to get all path components in one query
- ;; Segments use entity_id for path construction
- ;; Include both "removed" (moved out of remote-synced collection) and "delete" (archived)
- removed-segment-paths (->> (t2/query {:select [:s.entity_id
- [:t.name :table_name]
- :t.schema
- [:db.name :db_name]]
- :from [[:segment :s]]
- :join [[:remote_sync_object :rso]
- [:and [:= :rso.model_type [:inline "Segment"]]
- [:in :rso.status [[:inline "removed"] [:inline "delete"]]]
- [:= :rso.model_id :s.id]]
- [:metabase_table :t] [:= :t.id :s.table_id]
- [:metabase_database :db] [:= :db.id :t.db_id]]})
- (map (fn [{:keys [entity_id table_name schema db_name]}]
- (str/join "/" (cond-> ["databases" db_name]
- schema (conj "schemas" schema)
- true (conj "tables" table_name "segments" entity_id))))))
- all-delete-prefixes (concat top-level-removed-prefixes
- removed-table-paths
- removed-segment-paths)
+ (let [all-delete-prefixes (spec/build-all-removal-paths)
written-version (source/store! models all-delete-prefixes snapshot task-id message)]
(remote-sync.task/set-version! task-id written-version))
- ;; Delete removed Table/Field/Segment entries (they've been deleted from remote)
- (t2/delete! :model/RemoteSyncObject
- :model_type [:in ["Table" "Field" "Segment"]]
- :status "removed")
- ;; Also delete archived Segment entries (status "delete")
- (t2/delete! :model/RemoteSyncObject
- :model_type "Segment"
- :status "delete")
- (t2/update! :model/RemoteSyncObject {:status "synced" :status_changed_at sync-timestamp})))
- {:status :success
- :version (source.p/version snapshot)}
- (catch Exception e
- (if (:cancelled? (ex-data e))
- (log/info "Export to git repository was cancelled")
- (do
- (analytics/inc! :metabase-remote-sync/exports-failed)
- (remote-sync.task/fail-sync-task! task-id (ex-message e))
- {:status :error
- :version (source.p/version snapshot)
- :message (format "Failed to export to git repository: %s" (ex-message e))})))
- (finally
- (analytics/observe! :metabase-remote-sync/export-duration-ms (t/as (t/duration sync-timestamp (t/instant)) :millis))))))
+ (t2/update! :model/RemoteSyncObject {:status "synced" :status_changed_at sync-timestamp})
+ {:status :success
+ :version (source.p/version snapshot)})
+ {:status :error
+ :message "No remote-syncable content available."}))
+ (catch Exception e
+ (if (:cancelled? (ex-data e))
+ (log/info "Export to git repository was cancelled")
+ (do
+ (log/errorf e "Failed to export to git repository: %s" (ex-message e))
+ (analytics/inc! :metabase-remote-sync/exports-failed)
+ (remote-sync.task/fail-sync-task! task-id (ex-message e))
+ {:status :error
+ :version (source.p/version snapshot)
+ :message (format "Failed to export to git repository: %s" (ex-message e))})))
+ (finally
+ (analytics/observe! :metabase-remote-sync/export-duration-ms (t/as (t/duration sync-timestamp (t/instant)) :millis)))))
{:status :error
:message "Remote sync source is not enabled. Please configure MB_GIT_SOURCE_REPO_URL environment variable."}))
@@ -500,7 +357,7 @@
are unsaved changes and force? is false."
[branch force? import-args]
(let [source (source/source-from-settings branch)
- has-dirty? (remote-sync.object/dirty-global?)]
+ has-dirty? (remote-sync.object/dirty?)]
(when (and has-dirty? (not force?))
(throw (ex-info "There are unsaved changes in the Remote Sync collection which will be overwritten by the import. Force the import to discard these changes."
{:status-code 400
@@ -542,3 +399,39 @@
(:id (async-import! (settings/remote-sync-branch) true {}))))
(u/prog1 nil
(collection/clear-remote-synced-collection!))))
+
+(defn sync-transform-tracking!
+ "Called when remote-sync-transforms setting changes.
+ When enabled: mark all existing transforms, transform tags, and transforms-namespace collections
+ as 'create' for initial sync.
+ When disabled: remove all transform-related tracking entries."
+ [enabled?]
+ (let [timestamp (t/offset-date-time)]
+ (if enabled?
+ (do
+ ;; Mark all transforms-namespace collections for initial sync
+ (doseq [coll (t2/select [:model/Collection :id :name] :namespace (name collection/transforms-ns))]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "Collection"
+ :model_id (:id coll)
+ :model_name (:name coll)
+ :status "create"
+ :status_changed_at timestamp}))
+ ;; Mark all existing transforms and transform tags for initial sync
+ (doseq [[model name-key] [[:model/Transform :name]
+ [:model/TransformTag :name]]]
+ (doseq [entity (t2/select [model :id name-key])]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type (name (last (str/split (name model) #"/")))
+ :model_id (:id entity)
+ :model_name (get entity name-key)
+ :status "create"
+ :status_changed_at timestamp}))))
+ ;; Remove all transform-related tracking (including transforms-namespace collections)
+ (let [transform-coll-ids (t2/select-pks-set :model/Collection :namespace (name collection/transforms-ns))]
+ (t2/delete! :model/RemoteSyncObject
+ :model_type [:in ["Transform" "TransformTag"]])
+ (when (seq transform-coll-ids)
+ (t2/delete! :model/RemoteSyncObject
+ :model_type "Collection"
+ :model_id [:in transform-coll-ids]))))))
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj
index 930515adea9..b06098c60ce 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/models/remote_sync_object.clj
@@ -6,6 +6,7 @@
and for Field/Segment/Table models, parent table information."
(:require
[clojure.set :as set]
+ [metabase-enterprise.remote-sync.spec :as spec]
[metabase.util :as u]
[methodical.core :as methodical]
[toucan2.core :as t2]))
@@ -18,27 +19,40 @@
;;; ------------------------------------------------- Public API -------------------------------------------------------
-(defn dirty-global?
+(defn dirty?
"Checks if any collection has changes since the last sync.
- Returns true if any remote-synced object has a status other than 'synced', false otherwise."
+ Returns true if any remote-synced object has a status other than 'synced', false otherwise.
+ Excludes transform model types when transform sync is disabled."
[]
- (t2/exists? :model/RemoteSyncObject :status [:not= "synced"]))
+ (let [excluded (spec/excluded-model-types)]
+ (if (empty? excluded)
+ (t2/exists? :model/RemoteSyncObject :status [:not= "synced"])
+ (t2/exists? :model/RemoteSyncObject
+ :status [:not= "synced"]
+ :model_type [:not-in excluded]))))
-(defn dirty-for-global
+(defn dirty-objects
"Gets all models in any collection that are dirty with their sync status.
- Returns a sequence of model maps that have changed since the last remote sync,
- including details about their current state and sync status."
+ Returns a sequence of model maps that have changed since the last remote sync,
+ including details about their current state and sync status.
+ Excludes transform model types when transform sync is disabled."
[]
- (->> (t2/select :model/RemoteSyncObject :status [:not= "synced"])
- (map #(-> %
- (dissoc :id :status_changed_at)
- (set/rename-keys {:model_id :id
- :model_name :name
- :model_type :model
- :model_collection_id :collection_id
- :model_display :display
- :model_table_id :table_id
- :model_table_name :table_name
- :status :sync_status})
- (update :model u/lower-case-en)))
- (into [])))
+ (let [excluded (spec/excluded-model-types)
+ query (if (empty? excluded)
+ (t2/select :model/RemoteSyncObject :status [:not= "synced"])
+ (t2/select :model/RemoteSyncObject
+ :status [:not= "synced"]
+ :model_type [:not-in excluded]))]
+ (->> query
+ (map #(-> %
+ (dissoc :id :status_changed_at)
+ (set/rename-keys {:model_id :id
+ :model_name :name
+ :model_type :model
+ :model_collection_id :collection_id
+ :model_display :display
+ :model_table_id :table_id
+ :model_table_name :table_name
+ :status :sync_status})
+ (update :model u/lower-case-en)))
+ (into []))))
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
index abdd6117dc9..843fcccf9ff 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj
@@ -86,6 +86,14 @@
:encryption :no
:default (* 1000 60 5))
+(defsetting remote-sync-transforms
+ (deferred-tru "Whether to sync transforms via remote-sync. When enabled, all transforms, transform tags, and transform jobs are synced as a single unit (all-or-nothing).")
+ :type :boolean
+ :visibility :admin
+ :export? false
+ :encryption :no
+ :default false)
+
(defsetting remote-sync-check-changes-cache-ttl-seconds
(deferred-tru "Time-to-live in seconds for the remote changes check cache. Default is 60 seconds.")
:type :integer
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/source/git.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/source/git.clj
index d99bd08033c..b01046a13f7 100644
--- a/enterprise/backend/src/metabase_enterprise/remote_sync/source/git.clj
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/source/git.clj
@@ -250,9 +250,13 @@
- :path and :content keys for writing/updating a file
- :path and :remove? true for recursively removing all files at that path
- Replaces all files in the branch organized by collection prefix - files not in the provided list but in the same
- collection prefix will be deleted. Removal entries with empty paths are no-ops. Removing non-existent paths
- is also a no-op (idempotent).
+ For writes within collection directories, ALL files in the same collection are replaced
+ (using the collection entity_id prefix to identify the collection scope). This ensures
+ that stale files don't remain when a collection's contents change.
+
+ For removals, all files matching the path as a prefix are deleted (allowing recursive
+ directory deletion). Removal entries with empty paths are no-ops. Removing non-existent
+ paths is also a no-op (idempotent).
Returns the version written. Throws ExceptionInfo if the write or push
operation fails."
@@ -264,21 +268,35 @@
(with-open [inserter (.newObjectInserter repo)]
(let [index (DirCache/newInCore)
builder (.builder index)
- updated-prefixes (into #{}
+ ;; Extract collection prefixes from written paths - all files in these
+ ;; collections will be deleted and replaced with the new files
+ write-prefixes (into #{}
+ (comp
+ (remove :remove?)
+ (map :path)
+ (remove str/blank?)
+ (map path-prefix))
+ files)
+ ;; Collect removal paths/prefixes for explicit deletions
+ removal-prefixes (into #{}
(comp
- (map (fn [{:keys [path content remove?]}]
- (when-not remove?
- (let [blob-id (.insert inserter Constants/OBJ_BLOB (.getBytes ^String content "UTF-8"))
- entry (doto (DirCacheEntry. ^String path)
- (.setFileMode FileMode/REGULAR_FILE)
- (.setObjectId blob-id))]
- (.add builder entry)))
- (when (not-empty path)
- (path-prefix path))))
- (remove nil?))
+ (filter :remove?)
+ (map :path)
+ (remove str/blank?))
files)]
- ;; Copy existing tree entries, excluding files under updated-prefixes
+ ;; Add new/updated files to the index
+ (doseq [{:keys [path content remove?]} files
+ :when (and (not remove?) (not (str/blank? path)))]
+ (let [blob-id (.insert inserter Constants/OBJ_BLOB (.getBytes ^String content "UTF-8"))
+ entry (doto (DirCacheEntry. ^String path)
+ (.setFileMode FileMode/REGULAR_FILE)
+ (.setObjectId blob-id))]
+ (.add builder entry)))
+
+ ;; Copy existing tree entries, excluding:
+ ;; 1. Files in collections being written to (using write-prefixes)
+ ;; 2. Files matching explicit removal prefixes
(when parent-id
(with-open [rev-walk (RevWalk. repo)
tree-walk (TreeWalk. repo)]
@@ -286,8 +304,10 @@
(.addTree tree-walk (.getTree commit))
(.setRecursive tree-walk true)
(while (.next tree-walk)
- (let [path (.getPathString tree-walk)]
- (when-not (matches-prefix path updated-prefixes)
+ (let [path (.getPathString tree-walk)
+ existing-prefix (path-prefix path)]
+ (when-not (or (contains? write-prefixes existing-prefix)
+ (matches-prefix path removal-prefixes))
(let [entry (doto (DirCacheEntry. path)
(.setFileMode (.getFileMode tree-walk 0))
(.setObjectId (.getObjectId tree-walk 0)))]
diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj
new file mode 100644
index 00000000000..b0579d9c20c
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/spec.clj
@@ -0,0 +1,783 @@
+(ns metabase-enterprise.remote-sync.spec
+ "Centralized specifications for remote-syncable models.
+
+ Each spec defines everything needed for a model to participate in remote sync:
+ - Event handling (what events to listen to, how to determine status)
+ - Eligibility checking (when should this model be tracked)
+ - Dirty tracking (how to create/update RemoteSyncObject entries)
+ - Identity resolution (for import/export matching)
+ - Path construction (for serialization paths)
+
+ To add a new syncable model, add a spec entry to `remote-sync-specs` and
+ optionally implement custom multimethods if the default behavior doesn't fit."
+ (:require
+ [clojure.string :as str]
+ [clojure.walk :as walk]
+ [metabase-enterprise.remote-sync.settings :as rs-settings]
+ [metabase.collections.core :as collections]
+ [metabase.models.serialization :as serdes]
+ [metabase.settings.core :as setting]
+ [metabase.util :as u]
+ [metabase.util.log :as log]
+ [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+;;; ------------------------------------------------- Spec Definitions -------------------------------------------------
+
+(def remote-sync-specs
+ "Map of model keyword to its remote-sync specification.
+
+ Each spec contains:
+ - :model-type - String name for RemoteSyncObject model_type column
+ - :model-key - Toucan2 model keyword (e.g., :model/Card)
+ - :identity - Identity strategy: :entity-id, :path, or :hybrid
+ - :path-keys - For path-based: vector of path components [:database :schema :table :field]
+ - :events - Event configuration map:
+ :prefix - Event keyword prefix (e.g., :event/card)
+ :types - Vector of event types to handle [:create :update :delete]
+ - :eligibility - Eligibility configuration:
+ :type - :collection, :published-table, :parent-table, or :setting
+ :collection - For :collection type: :remote-synced, :transforms-namespace, or :any
+ :setting - For :setting type: setting keyword to check
+ - :archived-key - Key to check for archived status (nil if model has no archived concept)
+ - :tracking - RemoteSyncObject field configuration:
+ :select-fields - Fields to select for hydration
+ :hydrate-query - Optional custom query for joins (overrides select-fields)
+ :field-mappings - Map of RemoteSyncObject column -> source field or [field transform-fn]
+ - :removal - Removal/cleanup configuration:
+ :statuses - Set of statuses to check for removal (e.g., #{\"removed\" \"delete\"})
+ :scope-key - Optional key for scoping deletions (e.g., :collection_id, :id).
+ If nil, deletions are global (by entity_id only).
+ - :export-path - Export path construction:
+ :type - Path type: :collection-entity, :table-path, :field-path, :segment-path,
+ :transform-path, :transform-tag-path
+ - :side-effects - Optional side effects:
+ :on-first-sync - Function to call when model first becomes synced
+ - :enabled? - true or setting keyword (e.g., :remote-sync-transforms)"
+ {:model/Card
+ {:model-type "Card"
+ :model-key :model/Card
+ :identity :entity-id
+ :events {:prefix :event/card
+ :types [:create :update :delete]}
+ :eligibility {:type :collection
+ :collection :remote-synced}
+ :archived-key :archived
+ :tracking {:select-fields [:name :collection_id :display]
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id
+ :model_display [:display #(some-> % name)]}}
+ :removal {:statuses #{"removed"}
+ :scope-key :collection_id}
+ :export-path {:type :collection-entity}
+ :enabled? true}
+
+ :model/Dashboard
+ {:model-type "Dashboard"
+ :model-key :model/Dashboard
+ :identity :entity-id
+ :events {:prefix :event/dashboard
+ :types [:create :update :delete]}
+ :eligibility {:type :collection
+ :collection :remote-synced}
+ :archived-key :archived
+ :tracking {:select-fields [:name :collection_id]
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id}}
+ :removal {:statuses #{"removed"}
+ :scope-key :collection_id}
+ :export-path {:type :collection-entity}
+ :enabled? true}
+
+ :model/Document
+ {:model-type "Document"
+ :model-key :model/Document
+ :identity :entity-id
+ :events {:prefix :event/document
+ :types [:create :update :delete]}
+ :eligibility {:type :collection
+ :collection :remote-synced}
+ :archived-key :archived
+ :tracking {:select-fields [:name :collection_id]
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id}}
+ :removal {:statuses #{"removed"}
+ :scope-key :collection_id}
+ :export-path {:type :collection-entity}
+ :enabled? true}
+
+ :model/NativeQuerySnippet
+ {:model-type "NativeQuerySnippet"
+ :model-key :model/NativeQuerySnippet
+ :identity :entity-id
+ :events {:prefix :event/snippet
+ :types [:create :update :delete]}
+ :eligibility {:type :collection
+ :collection :remote-synced}
+ :archived-key :archived
+ :tracking {:select-fields [:name :id] ; snippets don't have collection_id directly
+ :field-mappings {:model_name :name}}
+ :removal {:statuses #{"removed"}
+ :scope-key :collection_id}
+ :export-path {:type :collection-entity}
+ :enabled? true}
+
+ :model/Timeline
+ {:model-type "Timeline"
+ :model-key :model/Timeline
+ :identity :entity-id
+ :events {:prefix :event/timeline
+ :types [:create :update :delete]}
+ :eligibility {:type :collection
+ :collection :remote-synced}
+ :archived-key :archived
+ :tracking {:select-fields [:name :collection_id]
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id}}
+ :removal {:statuses #{"removed"}
+ :scope-key :collection_id}
+ :export-path {:type :collection-entity}
+ :enabled? true}
+
+ :model/Collection
+ {:model-type "Collection"
+ :model-key :model/Collection
+ :identity :entity-id
+ :events {:prefix :event/collection
+ :types [:create :update]} ; no delete event for collections
+ :eligibility {:type :collection
+ :collection :any} ; remote-synced OR transforms-namespace
+ :archived-key :archived
+ :tracking {:select-fields [:name :id]
+ :field-mappings {:model_name :name
+ :model_collection_id :id}} ; collection_id is self
+ :removal {:statuses #{"removed" "delete"} ; delete is set when collection is archived
+ :scope-key :id} ; collections are scoped by their own id
+ :export-path {:type :collection-entity}
+ :export-scope :root-collections ; query for root-level remote-synced + transforms-namespace collections
+ :side-effects {:on-first-sync :track-published-tables}
+ :enabled? true}
+
+ :model/Table
+ {:model-type "Table"
+ :model-key :model/Table
+ :identity :path
+ :path-keys [:database :schema :table]
+ :events {:prefix :event/table
+ :types [:create :update :delete]}
+ :eligibility {:type :published-table}
+ :archived-key :archived_at ; tables use archived_at, not archived
+ :tracking {:select-fields [:name :collection_id]
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id
+ :model_table_id :id
+ :model_table_name :name}} ; self-reference
+ :removal {:statuses #{"removed"}}
+ :export-path {:type :table-path}
+ :enabled? true}
+
+ :model/Field
+ {:model-type "Field"
+ :model-key :model/Field
+ :identity :path
+ :path-keys [:database :schema :table :field]
+ :parent-model :model/Table
+ :events {:prefix :event/field
+ :types [:create :update :delete]}
+ :eligibility {:type :parent-table}
+ :archived-key nil ; fields don't have archived
+ :tracking {:hydrate-query {:select [:f.name :f.table_id
+ [:t.collection_id :collection_id]
+ [:t.name :table_name]]
+ :from [[:metabase_field :f]]
+ :join [[:metabase_table :t] [:= :f.table_id :t.id]]
+ :where [:= :f.id :?id]}
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id
+ :model_table_id :table_id
+ :model_table_name :table_name}}
+ :removal {:statuses #{"removed"}}
+ :export-path {:type :field-path}
+ :enabled? true}
+
+ :model/Segment
+ {:model-type "Segment"
+ :model-key :model/Segment
+ :identity :hybrid ; entity_id but needs table context for path
+ :path-keys [:database :schema :table]
+ :parent-model :model/Table
+ :events {:prefix :event/segment
+ :types [:create :update :delete]}
+ :eligibility {:type :parent-table}
+ :archived-key :archived
+ :tracking {:hydrate-query {:select [:s.name :s.table_id
+ [:t.collection_id :collection_id]
+ [:t.name :table_name]]
+ :from [[:segment :s]]
+ :join [[:metabase_table :t] [:= :s.table_id :t.id]]
+ :where [:= :s.id :?id]}
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id
+ :model_table_id :table_id
+ :model_table_name :table_name}}
+ :removal {:statuses #{"removed" "delete"}} ; Segment has both
+ :export-path {:type :segment-path}
+ :enabled? true}
+
+ :model/Transform
+ {:model-type "Transform"
+ :model-key :model/Transform
+ :identity :entity-id
+ :events {:prefix :event/transform
+ :types [:create :update :delete]}
+ :eligibility {:type :setting
+ :setting :remote-sync-transforms}
+ :archived-key :archived
+ :tracking {:select-fields [:name :collection_id]
+ :field-mappings {:model_name :name
+ :model_collection_id :collection_id}}
+ :removal {:statuses #{"removed" "delete"}} ; no scope-key = global deletion
+ :export-path {:type :transform-path}
+ :export-scope :root-only ; query for root transforms (collection_id = nil)
+ :enabled? :remote-sync-transforms}
+
+ :model/TransformTag
+ {:model-type "TransformTag"
+ :model-key :model/TransformTag
+ :identity :entity-id
+ :events {:prefix :event/transform-tag
+ :types [:create :update :delete]}
+ :eligibility {:type :setting
+ :setting :remote-sync-transforms}
+ :archived-key nil ; no archived field
+ :tracking {:select-fields [:name]
+ :field-mappings {:model_name :name}}
+ :removal {:statuses #{"removed" "delete"}} ; no scope-key = global deletion
+ :export-path {:type :transform-tag-path}
+ :export-scope :all ; query for all instances
+ :enabled? :remote-sync-transforms}})
+
+;;; ------------------------------------------------- Helper Functions -------------------------------------------------
+
+(defn spec-enabled?
+ "Returns true if the spec is currently enabled based on its :enabled? value."
+ [{:keys [enabled?]}]
+ (cond
+ (true? enabled?) true
+ (keyword? enabled?) (boolean (setting/get-value-of-type :boolean enabled?))
+ :else false))
+
+(defn enabled-specs
+ "Returns a map of model-key -> spec for all currently enabled specs."
+ []
+ (into {}
+ (filter (fn [[_ spec]] (spec-enabled? spec)))
+ remote-sync-specs))
+
+(defn spec-for-model-type
+ "Returns the spec for a given model type string (e.g., \"Card\")."
+ [model-type-str]
+ (some (fn [[_ spec]]
+ (when (= (:model-type spec) model-type-str)
+ spec))
+ remote-sync-specs))
+
+(defn spec-for-model-key
+ "Returns the spec for a given model key (e.g., :model/Card)."
+ [model-key]
+ (get remote-sync-specs model-key))
+
+(defn specs-by-identity-type
+ "Returns a map of model-key -> spec filtered by identity type."
+ [identity-type]
+ (into {}
+ (filter (fn [[_ spec]] (= (:identity spec) identity-type)))
+ (enabled-specs)))
+
+(defn excluded-model-types
+ "Returns a set of model type strings that should be excluded from dirty detection
+ based on current settings. Models with a setting-based :enabled? that is currently
+ false will be excluded."
+ []
+ (->> remote-sync-specs
+ (filter (fn [[_ spec]]
+ (and (keyword? (:enabled? spec))
+ (not (setting/get-value-of-type :boolean (:enabled? spec))))))
+ (map (fn [[_ spec]] (:model-type spec)))
+ set))
+
+(defn all-model-types
+ "Returns a set of all model type strings."
+ []
+ (into #{} (map :model-type (vals remote-sync-specs))))
+
+;;; ------------------------------------------------ Eligibility Checking ----------------------------------------------
+
+(defn- transforms-namespace-collection?
+ "Check if this is a transforms-namespace collection."
+ [object]
+ (= (keyword (:namespace object)) :transforms))
+
+(defmulti check-eligibility
+ "Determines if a model instance should be tracked for remote sync.
+ Dispatches on the eligibility type defined in the spec."
+ {:arglists '([spec object])}
+ (fn [spec _object] (get-in spec [:eligibility :type])))
+
+(defmethod check-eligibility :collection
+ [{:keys [eligibility]} object]
+ (let [collection-type (:collection eligibility)
+ collection-id (:collection_id object)]
+ (case collection-type
+ :remote-synced
+ (boolean (collections/remote-synced-collection? collection-id))
+
+ :transforms-namespace
+ (and (rs-settings/remote-sync-transforms)
+ (transforms-namespace-collection? object))
+
+ :any
+ (or (collections/remote-synced-collection? (or collection-id object))
+ (and (rs-settings/remote-sync-transforms)
+ (transforms-namespace-collection? object)))
+
+ false)))
+
+(defmethod check-eligibility :published-table
+ [_ {:keys [is_published collection_id]}]
+ (boolean
+ (and is_published
+ (collections/remote-synced-collection? collection_id))))
+
+(defmethod check-eligibility :parent-table
+ [{:keys [parent-model]} {:keys [table_id]}]
+ (boolean
+ (when table_id
+ (when-let [table (t2/select-one parent-model :id table_id)]
+ (check-eligibility (spec-for-model-key parent-model) table)))))
+
+(defmethod check-eligibility :setting
+ [{:keys [eligibility]} _object]
+ (boolean (setting/get-value-of-type :boolean (:setting eligibility))))
+
+(defmethod check-eligibility :default
+ [_ _]
+ false)
+
+;;; ---------------------------------------------------- Hydration -----------------------------------------------------
+
+(defn hydrate-model-details
+ "Hydrates model details for RemoteSyncObject based on spec.
+ Returns a map with the fields needed to populate the sync object."
+ [{:keys [model-key tracking]} model-id]
+ (if-let [query (:hydrate-query tracking)]
+ ;; Use custom query for joins (Field, Segment)
+ (first (t2/query (update query :where
+ (fn [where-clause]
+ (walk/postwalk
+ #(if (= % :?id) model-id %)
+ where-clause)))))
+ ;; Simple select using select-fields
+ (when-let [fields (:select-fields tracking)]
+ (t2/select-one (into [model-key] fields) :id model-id))))
+
+(defn build-sync-object-fields
+ "Builds the fields map for RemoteSyncObject from hydrated model details.
+ Applies any transform functions defined in field-mappings."
+ [{:keys [tracking]} model-details]
+ (when model-details
+ (into {}
+ (for [[sync-field source-spec] (:field-mappings tracking)]
+ (let [value (cond
+ ;; Simple keyword - just get the field
+ (keyword? source-spec)
+ (get model-details source-spec)
+
+ ;; Vector with transform function [field transform-fn]
+ (vector? source-spec)
+ (let [[field transform-fn] source-spec]
+ (transform-fn (get model-details field)))
+
+ :else
+ source-spec)]
+ [sync-field value])))))
+
+;;; ------------------------------------------------ Identity Resolution -----------------------------------------------
+
+(defmulti extract-identity
+ "Extracts the identity for a model instance based on the spec's identity strategy.
+ Used for matching during import/export."
+ {:arglists '([spec instance])}
+ (fn [spec _instance] (:identity spec)))
+
+(defmethod extract-identity :entity-id
+ [_ instance]
+ {:entity-id (:entity_id instance)})
+
+(defmethod extract-identity :path
+ [{:keys [path-keys]} instance]
+ ;; Path-based identity uses the path-keys to build a unique identifier
+ (select-keys instance path-keys))
+
+(defmethod extract-identity :hybrid
+ [{:keys [path-keys]} instance]
+ ;; Hybrid uses both entity_id and path context
+ {:entity-id (:entity_id instance)
+ :path (select-keys instance path-keys)})
+
+;;; --------------------------------------------- Export Path Construction ---------------------------------------------
+
+(defn- transform-entity-for-serdes
+ "Transforms entity fields to serdes format for path generation.
+ - Converts integer table_id to [db-name schema table-name] format"
+ [entity]
+ (cond-> entity
+ ;; Transform table_id for Segment (and other table-based entities)
+ (and (:table_id entity) (integer? (:table_id entity)))
+ (assoc :table_id (serdes/*export-table-fk* (:table_id entity)))))
+
+(defn- entity->serdes-path
+ "Builds the file path for an entity using serdes/storage-path.
+ For Collections, returns the directory path (for recursive deletion).
+ For other entities, returns the full file path.
+ Returns the path as a string (without extension), or nil if path cannot be built."
+ [model-type entity ctx]
+ (try
+ (when-let [serdes-meta (serdes/generate-path model-type entity)]
+ (let [;; Transform entity fields to serdes format (e.g., table_id -> [db schema table])
+ transformed-entity (transform-entity-for-serdes entity)
+ entity-with-meta (assoc transformed-entity :serdes/meta serdes-meta)
+ storage-path (serdes/storage-path entity-with-meta ctx)]
+ ;; For Collections, drop the last segment (filename) to get directory path
+ (if (= model-type "Collection")
+ (str/join "/" (butlast storage-path))
+ (str/join "/" storage-path))))
+ (catch Exception e
+ (log/warnf "Failed to build storage path for %s %s: %s"
+ model-type (:id entity) (ex-message e))
+ nil)))
+
+;;; -------------------------------------------- Event Helper Functions ------------------------------------------------
+
+(defn determine-status
+ "Determines the sync status based on event type and archived state."
+ [{:keys [archived-key]} topic object]
+ (let [archived? (and archived-key (get object archived-key))]
+ (if archived?
+ "delete"
+ (let [topic-name (name topic)]
+ (cond
+ (str/ends-with? topic-name "-create") "create"
+ (str/ends-with? topic-name "-update") "update"
+ (str/ends-with? topic-name "-delete") "delete"
+ :else "update")))))
+
+(defn event-keywords
+ "Returns a map of event keywords for a spec: {:parent :create :update :delete}."
+ [{:keys [events]}]
+ (let [{:keys [prefix types]} events
+ ns-str (namespace prefix)
+ base (name prefix)]
+ (merge
+ ;; Parent keyword uses the event prefix name (e.g., :card -> card-change-event)
+ {:parent (keyword "metabase-enterprise.remote-sync.events"
+ (str base "-change-event"))}
+ (into {}
+ (for [event-type types]
+ [event-type (keyword ns-str (str base "-" (name event-type)))])))))
+
+;;; ---------------------------------------------- Spec Field Accessors ------------------------------------------------
+
+(defn select-fields-for-sync
+ "Returns the fields to select for a model during sync operations.
+ Falls back to default if not specified in spec."
+ [model-type-str]
+ (if-let [spec (spec-for-model-type model-type-str)]
+ (or (get-in spec [:tracking :select-fields])
+ [:id :name :collection_id])
+ [:id :name :collection_id]))
+
+;;; -------------------------------------------- Removal Path Building --------------------------------------------------
+
+(defn- query-removed-ids
+ "Queries RemoteSyncObject for model IDs marked for removal."
+ [model-type statuses]
+ (t2/select-fn-set :model_id :model/RemoteSyncObject
+ :model_type model-type
+ :status [:in (vec statuses)]))
+
+(defn- query-removed-entities
+ "Queries entities marked for removal with all fields needed for serdes path generation.
+ serdes/generate-path needs: entity_id, name (for label)
+ serdes/storage-path needs: collection_id (for collection context), table_id (for segment paths)"
+ [{:keys [model-type model-key removal]}]
+ (let [{:keys [statuses]} removal
+ removed-ids (query-removed-ids model-type statuses)]
+ (when (seq removed-ids)
+ ;; Select all columns since different models need different fields for serdes paths
+ (t2/select model-key :id [:in removed-ids]))))
+
+(defn build-all-removal-paths
+ "Builds full file paths for all entities marked for removal in RemoteSyncObject.
+ Uses serdes/storage-path to generate paths that exactly match the file structure.
+
+ Does bulk queries per model type for efficiency:
+ 1. Query RemoteSyncObject for model_ids with removal statuses
+ 2. Query actual entities by those IDs
+ 3. Use serdes/storage-path to build the exact file path
+
+ Returns paths without file extensions - the git source adds .yaml as needed."
+ []
+ (let [ctx (serdes/storage-base-context)]
+ (into []
+ (for [[_model-key spec] (enabled-specs)
+ :when (spec-enabled? spec)
+ :let [{:keys [statuses]} (:removal spec)
+ model-type (:model-type spec)]
+ :when (seq statuses)
+ entity (query-removed-entities spec)
+ :let [path (entity->serdes-path model-type entity ctx)]
+ :when path]
+ path))))
+
+;;; ----------------------------------------- Sync Object Query Functions --------------------------------------------
+
+(defmulti query-entities-for-sync
+ "Queries entities for sync based on identity type.
+ Returns a sequence of maps ready for RemoteSyncObject insertion.
+
+ Parameters:
+ - spec: The model spec
+ - data: For :entity-id/:hybrid - a collection of entity_ids
+ For :path - a collection of path maps {:db_name :schema :table_name [:field_name]}
+ - timestamp: The sync timestamp"
+ {:arglists '([spec data timestamp])}
+ (fn [spec _data _timestamp] (:identity spec)))
+
+(defmethod query-entities-for-sync :entity-id
+ [{:keys [model-type model-key tracking]} entity-ids timestamp]
+ (when (seq entity-ids)
+ (let [;; Get select fields from spec, with :id always included
+ select-fields (into [:id] (or (:select-fields tracking) [:name :collection_id]))
+ entities (t2/select (into [model-key] select-fields) :entity_id [:in entity-ids])]
+ (map (fn [entity]
+ (let [;; Apply field mappings
+ field-mappings (:field-mappings tracking)
+ mapped-fields (into {}
+ (for [[sync-field source-spec] field-mappings]
+ (let [value (cond
+ (keyword? source-spec)
+ (get entity source-spec)
+ (vector? source-spec)
+ (let [[field transform-fn] source-spec]
+ (transform-fn (get entity field)))
+ :else source-spec)]
+ [sync-field value])))]
+ (merge {:model_type model-type
+ :model_id (:id entity)
+ :status "synced"
+ :status_changed_at timestamp}
+ mapped-fields)))
+ entities))))
+
+(defmethod query-entities-for-sync :hybrid
+ ;; Hybrid models like Segment need a join query for table info
+ [{:keys [model-type tracking]} entity-ids timestamp]
+ (when (seq entity-ids)
+ (if-let [query-template (:hydrate-query tracking)]
+ ;; Use custom hydrate query adapted for batch lookup
+ ;; Keep the existing from/join structure, just modify select and where
+ ;; The query template uses alias :s for the main model (segment)
+ (let [base-query (-> query-template
+ ;; Add :id to select for model_id field
+ (update :select (fn [cols] (vec (concat [:s.id] cols))))
+ ;; Replace where clause to use batch entity_id lookup
+ (assoc :where [:in :s.entity_id entity-ids]))]
+ (->> (t2/query base-query)
+ (map (fn [entity]
+ (let [field-mappings (:field-mappings tracking)
+ mapped-fields (into {}
+ (for [[sync-field source-spec] field-mappings]
+ [sync-field (get entity source-spec)]))]
+ (merge {:model_type model-type
+ :model_id (:id entity)
+ :status "synced"
+ :status_changed_at timestamp}
+ mapped-fields))))))
+ ;; Fallback - shouldn't happen for hybrid models with proper spec
+ nil)))
+
+(defn- build-path-where-clause
+ "Builds an :or where clause for path-based lookups."
+ [paths has-field?]
+ (into [:or]
+ (for [path paths]
+ (let [{:keys [db_name schema table_name field_name]} path]
+ (cond-> [:and
+ [:= :db.name db_name]
+ (if schema [:= :t.schema schema] [:is :t.schema nil])
+ [:= :t.name table_name]]
+ (and has-field? field_name)
+ (conj [:= :f.name field_name]))))))
+
+(defmethod query-entities-for-sync :path
+ [{:keys [model-type model-key]} paths timestamp]
+ (when (seq paths)
+ (case model-key
+ :model/Table
+ (->> (t2/query {:select [:t.id :t.name :t.collection_id]
+ :from [[:metabase_table :t]]
+ :join [[:metabase_database :db] [:= :db.id :t.db_id]]
+ :where (build-path-where-clause paths false)})
+ (map (fn [{:keys [id name collection_id]}]
+ {:model_type model-type
+ :model_id id
+ :model_name name
+ :model_collection_id collection_id
+ :model_display nil
+ :model_table_id id
+ :model_table_name name
+ :status "synced"
+ :status_changed_at timestamp})))
+
+ :model/Field
+ (->> (t2/query {:select [:f.id :f.name :f.table_id [:t.collection_id :collection_id] [:t.name :table_name]]
+ :from [[:metabase_field :f]]
+ :join [[:metabase_table :t] [:= :t.id :f.table_id]
+ [:metabase_database :db] [:= :db.id :t.db_id]]
+ :where (build-path-where-clause paths true)})
+ (map (fn [{:keys [id name table_id table_name collection_id]}]
+ {:model_type model-type
+ :model_id id
+ :model_name name
+ :model_collection_id collection_id
+ :model_display nil
+ :model_table_id table_id
+ :model_table_name table_name
+ :status "synced"
+ :status_changed_at timestamp})))
+
+ ;; Default for unknown path models
+ nil)))
+
+(defmethod query-entities-for-sync :default
+ [_ _ _]
+ nil)
+
+(defn sync-all-entities!
+ "Syncs all entities based on specs. Takes:
+ - timestamp: The sync timestamp
+ - imported-entities-by-model: Map of model-type string -> set of entity_ids
+ - table-paths: Collection of table path maps
+ - field-paths: Collection of field path maps
+
+ Returns a sequence of all sync objects to insert."
+ [timestamp imported-entities-by-model table-paths field-paths]
+ (into []
+ (for [[model-key spec] (enabled-specs)
+ :let [identity-type (:identity spec)
+ model-type (:model-type spec)
+ data (case identity-type
+ :entity-id (get imported-entities-by-model model-type)
+ :path (case model-key
+ :model/Table table-paths
+ :model/Field field-paths
+ nil)
+ :hybrid (get imported-entities-by-model model-type))]
+ :when (seq data)
+ entity (query-entities-for-sync spec data timestamp)]
+ entity)))
+
+;;; -------------------------------------------- Export Extraction ------------------------------------------------
+
+(defmulti query-export-roots
+ "Queries for root-level export targets based on eligibility configuration.
+ Returns a sequence of [model-type id] tuples for initial targets, or nil if
+ this model type's targets are derived from other models (e.g., via serdes/descendants)."
+ {:arglists '([spec])}
+ (fn [spec] (get-in spec [:eligibility :type])))
+
+(defmethod query-export-roots :collection
+ [{:keys [export-scope]}]
+ (case (or export-scope :derived)
+ :root-collections
+ ;; Collection model: query for root-level remote-synced + transforms-namespace collections
+ ;; Excludes archived collections - their files are handled by the removal logic
+ (concat
+ (t2/select-fn-set (juxt (constantly "Collection") :id)
+ :model/Collection
+ {:where [:and
+ [:= :is_remote_synced true]
+ [:= :location "/"]
+ [:not :archived]]})
+ (when (rs-settings/remote-sync-transforms)
+ (t2/select-fn-set (juxt (constantly "Collection") :id)
+ :model/Collection
+ {:where [:and
+ [:= :namespace (name collections/transforms-ns)]
+ [:= :location "/"]
+ [:not :archived]]})))
+ :derived
+ ;; Other collection-based models: no root query, derived from collection expansion
+ nil))
+
+(defmethod query-export-roots :setting
+ [{:keys [export-scope model-key model-type] :as spec}]
+ (when (spec-enabled? spec)
+ (case export-scope
+ :root-only
+ ;; Transform: root transforms (collection_id = nil)
+ (t2/select-fn-set (juxt (constantly model-type) :id) model-key :collection_id nil)
+ :all
+ ;; TransformTag: all instances
+ (t2/select-fn-set (juxt (constantly model-type) :id) model-key)
+ ;; Default: no root query
+ nil)))
+
+(defmethod query-export-roots :published-table [_] nil) ; Derived via serdes/descendants
+(defmethod query-export-roots :parent-table [_] nil) ; Derived via serdes/descendants
+
+(defmethod query-export-roots :default [_] nil)
+
+(defn- resolve-targets
+ "Expands collection IDs to include all descendant collections.
+ Takes a set of collection IDs and returns a set including the original IDs
+ plus all IDs of nested collections."
+ [targets opts]
+ (when (seq targets)
+ (merge-with into
+ (u/traverse targets #(serdes/descendants (first %) (second %) opts))
+ (u/traverse targets #(serdes/required (first %) (second %))))))
+
+(defn extract-entities-for-export
+ "Extracts all entities for remote-sync export based on enabled specs.
+
+ Returns a lazy sequence of serialized entities ready for storage.
+
+ Iterates over enabled specs and uses `query-export-roots` to find root targets
+ for each model type. Models with `:export-scope :derived` or no export-scope
+ are expanded from other targets via serdes/descendants.
+
+ Only extracts models that:
+ 1. Have a spec in remote-sync-specs
+ 2. Are currently enabled (based on :enabled? field)
+ 3. Are in one of the provided collections (or descendants)"
+ []
+ (let [specs (enabled-specs)
+ ;; Collect all root targets by iterating over enabled specs
+ root-targets (into []
+ (comp (map val)
+ (mapcat query-export-roots)
+ (filter identity))
+ specs)]
+ (when-let [targets (resolve-targets
+ root-targets
+ {:include-field-values false
+ :include-database-secrets false
+ :continue-on-error false
+ :skip-archived true})]
+ (eduction (map (fn [[model ids]]
+ (serdes/extract-all model {:where [:in :id ids]})))
+ cat
+ (u/group-by first second (keys targets))))))
diff --git a/enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj b/enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj
index c2f2093bb22..8c388e88567 100644
--- a/enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj
+++ b/enterprise/backend/src/metabase_enterprise/serialization/v2/models.clj
@@ -27,7 +27,8 @@
"Document"
"Glossary"
"NativeQuerySnippet"
- "Timeline"])
+ "Timeline"
+ "Transform"])
(def exported-models
"The list of all models exported by serialization by default. Used for production code and by tests."
@@ -36,7 +37,6 @@
["FieldValues"
"Metabot"
"Setting"
- "Transform"
"TransformJob"
"TransformTag"]))
diff --git a/enterprise/backend/src/metabase_enterprise/transforms/api.clj b/enterprise/backend/src/metabase_enterprise/transforms/api.clj
index e100e92e66d..3e6c465a616 100644
--- a/enterprise/backend/src/metabase_enterprise/transforms/api.clj
+++ b/enterprise/backend/src/metabase_enterprise/transforms/api.clj
@@ -140,7 +140,7 @@
[:collection_id {:optional true} [:maybe ms/PositiveInt]]
[:owner_user_id {:optional true} [:maybe ms/PositiveInt]]
[:owner_email {:optional true} [:maybe :string]]]]
- (api/check-superuser)
+ (api/create-check :model/Transform body)
(check-database-feature body)
(check-feature-enabled! body)
@@ -168,8 +168,7 @@
(defn get-transform
"Get a specific transform."
[id]
- (api/check-superuser)
- (let [{:keys [target] :as transform} (api/check-404 (t2/select-one :model/Transform id))
+ (let [{:keys [target] :as transform} (api/read-check :model/Transform id)
target-table (transforms.util/target-table (transforms.i/target-db-id transform) target :active true)]
(-> transform
(t2/hydrate :last_run :transform_tag_ids :creator :owner)
@@ -195,9 +194,8 @@
"Get the dependencies of a specific transform."
[{:keys [id]} :- [:map
[:id ms/PositiveInt]]]
- (api/check-superuser)
+ (api/read-check :model/Transform id)
(let [id->transform (t2/select-pk->fn identity :model/Transform)
- _ (api/check-404 (get id->transform id))
global-ordering (transforms.ordering/transform-ordering (vals id->transform))
dep-ids (get global-ordering id)
dependencies (map id->transform dep-ids)]
@@ -249,7 +247,7 @@
[:collection_id {:optional true} [:maybe ms/PositiveInt]]
[:owner_user_id {:optional true} [:maybe ms/PositiveInt]]
[:owner_email {:optional true} [:maybe :string]]]]
- (api/check-superuser)
+ (api/write-check :model/Transform id)
(let [transform (t2/with-transaction [_]
;; Cycle detection should occur within the transaction to avoid race
(let [old (t2/select-one :model/Transform id)
@@ -282,8 +280,7 @@
"Delete a transform."
[{:keys [id]} :- [:map
[:id ms/PositiveInt]]]
- (api/check-superuser)
- (let [transform (api/check-404 (t2/select-one :model/Transform id))]
+ (let [transform (api/write-check :model/Transform id)]
(t2/delete! :model/Transform id)
(events/publish-event! :event/transform-delete
{:object transform
@@ -298,7 +295,7 @@
"Delete a transform's output table."
[{:keys [id]} :- [:map
[:id ms/PositiveInt]]]
- (api/check-superuser)
+ (api/write-check :model/Transform id)
(transforms.util/delete-target-table-by-id! id)
nil)
@@ -310,8 +307,7 @@
"Cancel the current run for a given transform."
[{:keys [id]} :- [:map
[:id ms/PositiveInt]]]
- (api/check-superuser)
- (let [transform (api/check-404 (t2/select-one :model/Transform id))
+ (let [transform (api/write-check :model/Transform id)
run (api/check-404 (transform-run/running-run-for-transform-id id))]
(transform-run-cancelation/mark-cancel-started-run! (:id run))
(when (transforms.util/python-transform? transform)
@@ -326,8 +322,7 @@
"Run a transform."
[{:keys [id]} :- [:map
[:id ms/PositiveInt]]]
- (api/check-superuser)
- (let [transform (api/check-404 (t2/select-one :model/Transform id))
+ (let [transform (api/write-check :model/Transform id)
_ (check-feature-enabled! transform)
start-promise (promise)]
(u.jvm/in-virtual-thread*
diff --git a/enterprise/backend/src/metabase_enterprise/transforms/models/transform.clj b/enterprise/backend/src/metabase_enterprise/transforms/models/transform.clj
index cc8cecbdce8..3193c0d42d4 100644
--- a/enterprise/backend/src/metabase_enterprise/transforms/models/transform.clj
+++ b/enterprise/backend/src/metabase_enterprise/transforms/models/transform.clj
@@ -11,6 +11,7 @@
[metabase.lib.core :as lib]
[metabase.models.interface :as mi]
[metabase.models.serialization :as serdes]
+ [metabase.remote-sync.core :as remote-sync]
[metabase.search.core :as search.core]
[metabase.search.ingestion :as search]
[metabase.search.spec :as search.spec]
@@ -25,10 +26,24 @@
(doseq [trait [:metabase/model :hook/entity-id :hook/timestamped?]]
(derive :model/Transform trait))
-;; Only superusers can access transforms
-(doto :model/Transform
- (derive ::mi/read-policy.superuser)
- (derive ::mi/write-policy.superuser))
+;; Only superusers can access transforms, and writes/creates are blocked globally in remote-sync read-only mode
+(defmethod mi/can-read? :model/Transform
+ ([_instance]
+ (mi/superuser?))
+ ([model pk]
+ (mi/can-read? (t2/select-one model pk))))
+
+(defmethod mi/can-write? :model/Transform
+ ([_instance]
+ (and (mi/superuser?)
+ (remote-sync/transforms-editable?)))
+ ([model pk]
+ (mi/can-write? (t2/select-one model pk))))
+
+(defmethod mi/can-create? :model/Transform
+ [_model _instance]
+ (and (mi/superuser?)
+ (remote-sync/transforms-editable?)))
(defn- keywordize-source-table-refs
"Keywordize keys in source-tables map values (refs are maps, ints pass through)."
@@ -270,16 +285,27 @@
:tags (serdes/nested :model/TransformTransformTag :transform_id opts)}})
(defmethod serdes/dependencies "Transform"
- [{:keys [source tags]}]
+ [{:keys [collection_id source tags]}]
(set
(concat
+ (when collection_id
+ [[{:model "Collection" :id collection_id}]])
(for [{tag-id :tag_id} tags]
[{:model "TransformTag" :id tag-id}])
(serdes/mbql-deps source))))
-(defmethod serdes/storage-path "Transform" [transform _ctx]
- (let [{:keys [id label]} (-> transform serdes/path last)]
- ["transforms" (serdes/storage-leaf-file-name id label)]))
+(defmethod serdes/storage-path "Transform" [transform ctx]
+ ;; Path: ["collections" "<nested ... collections>" "transforms" "<entity_id_name>"]
+ ;; Use default collection path, then restructure similar to NativeQuerySnippet
+ (let [basis (serdes/storage-default-collection-path transform ctx)
+ file (last basis)
+ colls (->> basis rest (drop-last 2))] ; Drop "collections" at start, and last two elements
+ (concat ["collections"] colls ["transforms" file])))
+
+(defmethod serdes/required "Transform"
+ [_model id]
+ (when-let [collection-id (t2/select-one-fn :collection_id :model/Transform :id id)]
+ {["Collection" collection-id] {"Transform" id}}))
(defn- maybe-extract-transform-query-text
"Return the query text (truncated to `max-searchable-value-length`) from transform source; else nil.
diff --git a/enterprise/backend/src/metabase_enterprise/transforms/models/transform_tag.clj b/enterprise/backend/src/metabase_enterprise/transforms/models/transform_tag.clj
index c4ca09002a6..d789e030815 100644
--- a/enterprise/backend/src/metabase_enterprise/transforms/models/transform_tag.clj
+++ b/enterprise/backend/src/metabase_enterprise/transforms/models/transform_tag.clj
@@ -1,5 +1,6 @@
(ns metabase-enterprise.transforms.models.transform-tag
(:require
+ [metabase.events.core :as events]
[metabase.models.interface :as mi]
[metabase.models.serialization :as serdes]
[metabase.util.i18n :as i18n]
@@ -58,3 +59,16 @@
(defmethod serdes/storage-path "TransformTag" [tt _ctx]
(let [{:keys [id label]} (-> tt serdes/path last)]
["transforms" "transform_tags" (serdes/storage-leaf-file-name id label)]))
+
+;; Event hooks for remote-sync tracking
+(t2/define-after-insert :model/TransformTag [tag]
+ (events/publish-event! :event/transform-tag-create {:object tag})
+ tag)
+
+(t2/define-after-update :model/TransformTag [tag]
+ (events/publish-event! :event/transform-tag-update {:object tag})
+ tag)
+
+(t2/define-before-delete :model/TransformTag [tag]
+ (events/publish-event! :event/transform-tag-delete {:object tag})
+ tag)
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj
index 123e15bfb54..f5e63f01004 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/api_test.clj
@@ -544,7 +544,7 @@
(deftest settings-cannot-change-with-dirty-data
(testing "PUT /api/ee/remote-sync/settings doesn't allow losing dirty data"
- (with-redefs [remote-sync.object/dirty-global? (constantly true)
+ (with-redefs [remote-sync.object/dirty? (constantly true)
settings/check-and-update-remote-settings! #(throw (Exception. "Should not be called"))]
(mt/with-temporary-setting-values [remote-sync-url "https://github.com/test/repo.git"
remote-sync-token "test-token"
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/events_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/events_test.clj
index 2c96f811fe3..a2d235a328b 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/events_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/events_test.clj
@@ -15,28 +15,6 @@
(use-fixtures :once (fixtures/initialize :db))
-;;; Helper Functions Tests
-
-(deftest model-in-remote-synced-collection-returns-true-test
- (testing "model-in-remote-synced-collection? returns true for models in remote-synced collections"
- (mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}]
- (is (true? (#'lib.events/model-in-remote-synced-collection?
- {:collection_id (:id remote-sync-collection)}))))))
-
-(deftest model-in-remote-synced-collection-returns-false-for-normal-test
- (testing "model-in-remote-synced-collection? returns false for models in normal collections"
- (mt/with-temp [:model/Collection normal-collection {:name "Normal"}]
- (is (false? (#'lib.events/model-in-remote-synced-collection?
- {:collection_id (:id normal-collection)}))))))
-
-(deftest model-in-remote-synced-collection-returns-false-for-nil-test
- (testing "model-in-remote-synced-collection? returns false when collection_id is nil"
- (is (false? (#'lib.events/model-in-remote-synced-collection? {:collection_id nil})))))
-
-(deftest model-in-remote-synced-collection-returns-false-for-missing-test
- (testing "model-in-remote-synced-collection? returns false when model has no collection_id"
- (is (false? (#'lib.events/model-in-remote-synced-collection? {})))))
-
;;; Model Change Event Tests
(deftest card-create-event-creates-entry-test
@@ -513,58 +491,59 @@
(deftest create-remote-sync-object-entry-creates-new-entry-test
(testing "create-remote-sync-object-entry! creates new entry when none exists"
- (mt/with-current-user (mt/user->id :rasta)
- (t2/delete! :model/RemoteSyncObject)
- (let [existing-card-id (t2/select-one-fn :id [:model/Card :id])
- hydrate-fn (fn [id] (t2/select-one [:model/Card :name :collection_id :display] :id id))]
- (#'lib.events/create-or-update-remote-sync-object-entry! "Card" existing-card-id "create" hydrate-fn)
- (let [entries (t2/select :model/RemoteSyncObject)]
- (is (= 1 (count entries)))
- (is (=? {:model_type "Card"
- :model_id existing-card-id
- :status "create"}
- (first entries))))))))
+ (mt/with-temp [:model/Card card {:name "Test Card"
+ :dataset_query (mt/mbql-query venues)}]
+ (mt/with-current-user (mt/user->id :rasta)
+ (t2/delete! :model/RemoteSyncObject)
+ (let [hydrate-fn (fn [id] (t2/select-one [:model/Card :name :collection_id :display] :id id))]
+ (#'lib.events/create-or-update-remote-sync-object-entry! "Card" (:id card) "create" hydrate-fn)
+ (let [entries (t2/select :model/RemoteSyncObject)]
+ (is (= 1 (count entries)))
+ (is (=? {:model_type "Card"
+ :model_id (:id card)
+ :status "create"}
+ (first entries)))))))))
(deftest create-remote-sync-object-entry-updates-existing-entry-test
(testing "create-remote-sync-object-entry! updates existing entry when one exists"
- (mt/with-current-user (mt/user->id :rasta)
- (t2/delete! :model/RemoteSyncObject)
- (let [clock-t1 (t/mock-clock (t/instant "2024-01-01T10:00:00Z") (t/zone-id "UTC"))
- existing-dashboard-id (t2/select-one-fn :id [:model/Dashboard :id])
- hydrate-fn (fn [id] (t2/select-one [:model/Dashboard :name :collection_id] :id id))]
- (t/with-clock clock-t1
- (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" existing-dashboard-id "update" hydrate-fn))
- (let [initial-entry (t2/select-one :model/RemoteSyncObject :model_type "Dashboard" :model_id existing-dashboard-id)
- initial-time (:status_changed_at initial-entry)
- clock-t2 (t/mock-clock (t/instant "2024-01-01T11:00:00Z") (t/zone-id "UTC"))]
- (t/with-clock clock-t2
- (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" existing-dashboard-id "synced" hydrate-fn))
- (let [entries (t2/select :model/RemoteSyncObject :model_type "Dashboard" :model_id existing-dashboard-id)]
- (is (= 1 (count entries)))
- (let [update-entry (first entries)]
- (is (= (:id initial-entry) (:id update-entry)))
- (is (= "synced" (:status update-entry)))
- (is (t/after? (:status_changed_at update-entry) initial-time)))))))))
+ (mt/with-temp [:model/Dashboard dashboard {:name "Test Dashboard"}]
+ (mt/with-current-user (mt/user->id :rasta)
+ (t2/delete! :model/RemoteSyncObject)
+ (let [clock-t1 (t/mock-clock (t/instant "2024-01-01T10:00:00Z") (t/zone-id "UTC"))
+ hydrate-fn (fn [id] (t2/select-one [:model/Dashboard :name :collection_id] :id id))]
+ (t/with-clock clock-t1
+ (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" (:id dashboard) "update" hydrate-fn))
+ (let [initial-entry (t2/select-one :model/RemoteSyncObject :model_type "Dashboard" :model_id (:id dashboard))
+ initial-time (:status_changed_at initial-entry)
+ clock-t2 (t/mock-clock (t/instant "2024-01-01T11:00:00Z") (t/zone-id "UTC"))]
+ (t/with-clock clock-t2
+ (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" (:id dashboard) "synced" hydrate-fn))
+ (let [entries (t2/select :model/RemoteSyncObject :model_type "Dashboard" :model_id (:id dashboard))]
+ (is (= 1 (count entries)))
+ (let [update-entry (first entries)]
+ (is (= (:id initial-entry) (:id update-entry)))
+ (is (= "synced" (:status update-entry)))
+ (is (t/after? (:status_changed_at update-entry) initial-time))))))))))
(deftest create-remote-sync-object-entry-does-not-update-create-status-test
(testing "create-remote-sync-object-entry! does not update entry when status is 'create'"
- (mt/with-current-user (mt/user->id :rasta)
- (t2/delete! :model/RemoteSyncObject)
- (let [clock-t1 (t/mock-clock (t/instant "2024-01-01T10:00:00Z") (t/zone-id "UTC"))
- existing-dashboard-id (t2/select-one-fn :id [:model/Dashboard :id])
- hydrate-fn (fn [id] (t2/select-one [:model/Dashboard :name :collection_id] :id id))]
- (t/with-clock clock-t1
- (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" existing-dashboard-id "create" hydrate-fn))
- (let [initial-entry (t2/select-one :model/RemoteSyncObject :model_type "Dashboard" :model_id existing-dashboard-id)
- clock-t2 (t/mock-clock (t/instant "2024-01-01T11:00:00Z") (t/zone-id "UTC"))]
- (t/with-clock clock-t2
- (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" existing-dashboard-id "synced" hydrate-fn))
- (let [entries (t2/select :model/RemoteSyncObject :model_type "Dashboard" :model_id existing-dashboard-id)]
- (is (= 1 (count entries)))
- (let [update-entry (first entries)]
- (is (= (:id initial-entry) (:id update-entry)))
- (is (= "create" (:status update-entry)))
- (is (= (:status_changed_at initial-entry) (:status_changed_at update-entry))))))))))
+ (mt/with-temp [:model/Dashboard dashboard {:name "Test Dashboard"}]
+ (mt/with-current-user (mt/user->id :rasta)
+ (t2/delete! :model/RemoteSyncObject)
+ (let [clock-t1 (t/mock-clock (t/instant "2024-01-01T10:00:00Z") (t/zone-id "UTC"))
+ hydrate-fn (fn [id] (t2/select-one [:model/Dashboard :name :collection_id] :id id))]
+ (t/with-clock clock-t1
+ (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" (:id dashboard) "create" hydrate-fn))
+ (let [initial-entry (t2/select-one :model/RemoteSyncObject :model_type "Dashboard" :model_id (:id dashboard))
+ clock-t2 (t/mock-clock (t/instant "2024-01-01T11:00:00Z") (t/zone-id "UTC"))]
+ (t/with-clock clock-t2
+ (#'lib.events/create-or-update-remote-sync-object-entry! "Dashboard" (:id dashboard) "synced" hydrate-fn))
+ (let [entries (t2/select :model/RemoteSyncObject :model_type "Dashboard" :model_id (:id dashboard))]
+ (is (= 1 (count entries)))
+ (let [update-entry (first entries)]
+ (is (= (:id initial-entry) (:id update-entry)))
+ (is (= "create" (:status update-entry)))
+ (is (= (:status_changed_at initial-entry) (:status_changed_at update-entry)))))))))))
(deftest create-remote-sync-object-entry-with-hydrate-fn-test
(testing "create-remote-sync-object-entry! creates entry with hydrated details from hydrate-fn"
@@ -896,23 +875,6 @@
;;; Table Event Tests
-(deftest published-table-in-remote-synced-collection-helper-test
- (testing "published-table-in-remote-synced-collection? returns true for published tables in remote-synced collections"
- (mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}]
- (is (true? (#'lib.events/published-table-in-remote-synced-collection?
- {:is_published true :collection_id (:id remote-sync-collection)})))))
- (testing "published-table-in-remote-synced-collection? returns false when not published"
- (mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}]
- (is (false? (#'lib.events/published-table-in-remote-synced-collection?
- {:is_published false :collection_id (:id remote-sync-collection)})))))
- (testing "published-table-in-remote-synced-collection? returns false when collection is not remote-synced"
- (mt/with-temp [:model/Collection normal-collection {:name "Normal"}]
- (is (false? (#'lib.events/published-table-in-remote-synced-collection?
- {:is_published true :collection_id (:id normal-collection)})))))
- (testing "published-table-in-remote-synced-collection? returns false when collection_id is nil"
- (is (false? (#'lib.events/published-table-in-remote-synced-collection?
- {:is_published true :collection_id nil})))))
-
(deftest table-update-event-creates-entry-for-published-table-test
(testing "table-update event creates remote sync object entry for published table in remote-synced collection"
(mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}
@@ -1065,32 +1027,6 @@
;;; Segment Event Tests
-(deftest model-in-published-table-in-remote-synced-collection-helper-test
- (testing "model-in-published-table-in-remote-synced-collection? returns true for models in published tables in remote-synced collections"
- (mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}
- :model/Table table {:name "Test Table"
- :is_published true
- :collection_id (:id remote-sync-collection)}]
- (is (true? (#'lib.events/model-in-published-table-in-remote-synced-collection?
- {:table_id (:id table)})))))
- (testing "model-in-published-table-in-remote-synced-collection? returns false when table is not published"
- (mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}
- :model/Table table {:name "Test Table"
- :is_published false
- :collection_id (:id remote-sync-collection)}]
- (is (false? (#'lib.events/model-in-published-table-in-remote-synced-collection?
- {:table_id (:id table)})))))
- (testing "model-in-published-table-in-remote-synced-collection? returns false when collection is not remote-synced"
- (mt/with-temp [:model/Collection normal-collection {:name "Normal"}
- :model/Table table {:name "Test Table"
- :is_published true
- :collection_id (:id normal-collection)}]
- (is (false? (#'lib.events/model-in-published-table-in-remote-synced-collection?
- {:table_id (:id table)})))))
- (testing "model-in-published-table-in-remote-synced-collection? returns false when table_id is nil"
- (is (false? (#'lib.events/model-in-published-table-in-remote-synced-collection?
- {:table_id nil})))))
-
(deftest segment-create-event-creates-entry-test
(testing "segment-create event creates remote sync object entry for segment in published table in remote-synced collection"
(mt/with-temp [:model/Collection remote-sync-collection {:is_remote_synced true :name "Remote-Sync"}
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj
index d8e35c01107..29e1f235a4b 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/impl_test.clj
@@ -146,10 +146,11 @@
(let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "export" :initiated_by (mt/user->id :rasta)})]
;; Create a regular collection (not remote-synced) to verify it's not included
(mt/with-temp [:model/Collection {_coll-id :id} {:name "Regular Collection" :type nil :location "/"}]
- (let [mock-source (test-helpers/create-mock-source)
- result (impl/export! (source.p/snapshot mock-source) task-id "Test commit")]
- (is (= :error (:status result)))
- (is (= "No remote-synced collections available to sync." (:message result)))))))))
+ (mt/with-temporary-setting-values [remote-sync-transforms false]
+ (let [mock-source (test-helpers/create-mock-source)
+ result (impl/export! (source.p/snapshot mock-source) task-id "Test commit")]
+ (is (= :error (:status result)))
+ (is (= "No remote-syncable content available." (:message result))))))))))
(deftest export!-successful-with-default-collections-test
(testing "export! successful with default collections"
@@ -643,17 +644,18 @@
(t2/insert! :model/RemoteSyncObject
[{:model_type "Collection" :model_id coll-id :model_name "Active Collection" :status "synced" :status_changed_at (t/offset-date-time)}
{:model_type "Segment" :model_id segment-id :model_name "Test Segment" :model_table_id table-id :model_table_name "test-table" :status "removed" :status_changed_at (t/offset-date-time)}])
- (let [initial-files {"main" {"databases/test-db/tables/test-table/segments/test-segment-xxxxxxxx_Test Segment.yaml"
+ (let [initial-files {"main" {;; File path uses slugified name to match what serdes/storage-path generates
+ "databases/test-db/tables/test-table/segments/test-segment-xxxxxxxx_test_segment.yaml"
(test-helpers/generate-segment-yaml "Test Segment" "test-table" "test-db")}}
mock-source (test-helpers/create-mock-source :initial-files initial-files)
result (impl/export! (source.p/snapshot mock-source) task-id "Test commit")]
(is (= :success (:status result)))
(let [files-after-export (get @(:files-atom mock-source) "main")]
- (is (not (some #(str/includes? % "Test Segment") (keys files-after-export)))
+ (is (not (some #(str/includes? % "test-segment-xxxxxxxx") (keys files-after-export)))
"Removed segment files should be deleted after export")))))))))
-(deftest export!-cleans-up-removed-table-entries-test
- (testing "export! deletes RemoteSyncObject entries for 'removed' Tables/Fields/Segments after successful export"
+(deftest export!-updates-removed-table-entries-to-synced-test
+ (testing "export! updates RemoteSyncObject entries for 'removed' Tables/Fields/Segments to 'synced' status after successful export"
(mt/with-model-cleanup [:model/RemoteSyncObject :model/Segment]
(mt/with-temporary-setting-values [remote-sync-type :read-write]
(let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "export" :initiated_by (mt/user->id :rasta)})]
@@ -682,17 +684,21 @@
(let [mock-source (test-helpers/create-mock-source)
result (impl/export! (source.p/snapshot mock-source) task-id "Test commit")]
(is (= :success (:status result)))
- ;; After export, removed Table/Field/Segment entries should be deleted
- (is (nil? (t2/select-one :model/RemoteSyncObject :model_type "Table" :model_id table-id))
- "Table entry should be deleted after export")
- (is (nil? (t2/select-one :model/RemoteSyncObject :model_type "Field" :model_id field-id))
- "Field entry should be deleted after export")
- (is (nil? (t2/select-one :model/RemoteSyncObject :model_type "Segment" :model_id segment-id))
- "Segment entry should be deleted after export")
- ;; Collection should remain and be marked as synced
- (let [coll-entry (t2/select-one :model/RemoteSyncObject :model_type "Collection" :model_id coll-id)]
- (is (some? coll-entry) "Collection entry should remain")
- (is (= "synced" (:status coll-entry)) "Collection should be marked as synced")))))))))
+ ;; After export, all entries should have status 'synced'
+ (is (= 4 (t2/count :model/RemoteSyncObject))
+ "All RemoteSyncObject entries should remain after export")
+ (let [table-entry (t2/select-one :model/RemoteSyncObject :model_type "Table" :model_id table-id)
+ field-entry (t2/select-one :model/RemoteSyncObject :model_type "Field" :model_id field-id)
+ segment-entry (t2/select-one :model/RemoteSyncObject :model_type "Segment" :model_id segment-id)
+ coll-entry (t2/select-one :model/RemoteSyncObject :model_type "Collection" :model_id coll-id)]
+ (is (= "synced" (:status table-entry))
+ "Table entry should be updated to synced after export")
+ (is (= "synced" (:status field-entry))
+ "Field entry should be updated to synced after export")
+ (is (= "synced" (:status segment-entry))
+ "Segment entry should be updated to synced after export")
+ (is (= "synced" (:status coll-entry))
+ "Collection should remain marked as synced")))))))))
(deftest export!-generates-correct-delete-paths-for-tables-with-schema-test
(testing "export! generates correct delete paths for tables with schema"
@@ -787,7 +793,8 @@
(t2/insert! :model/RemoteSyncObject
[{:model_type "Collection" :model_id coll-id :model_name "Active Collection" :status "synced" :status_changed_at (t/offset-date-time)}
{:model_type "Segment" :model_id segment-id :model_name "Archived Segment" :model_table_id table-id :model_table_name "test-table" :status "delete" :status_changed_at (t/offset-date-time)}])
- (let [initial-files {"main" {"databases/test-db/tables/test-table/segments/archived-seg-xxxxxxxx_Archived Segment.yaml"
+ ;; File path uses slugified name to match what serdes/storage-path generates
+ (let [initial-files {"main" {"databases/test-db/tables/test-table/segments/archived-seg-xxxxxxxx_archived_segment.yaml"
(test-helpers/generate-segment-yaml "Archived Segment" "test-table" "test-db")}}
mock-source (test-helpers/create-mock-source :initial-files initial-files)
result (impl/export! (source.p/snapshot mock-source) task-id "Test commit")]
@@ -796,8 +803,8 @@
(is (not (some #(str/includes? % "archived-seg-xxxxxxxx") (keys files-after-export)))
"Archived segment files should be deleted after export"))
(testing "RemoteSyncObject entry is cleaned up after export"
- (is (nil? (t2/select-one :model/RemoteSyncObject :model_type "Segment" :model_id segment-id))
- "RemoteSyncObject entry for archived segment should be deleted")))))))))
+ (is (= "synced" (:status (t2/select-one :model/RemoteSyncObject :model_type "Segment" :model_id segment-id)))
+ "RemoteSyncObject entry for archived segment should have synced status")))))))))
;; Import Table/Field/Segment tracking tests
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/models/remote_sync_object_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/models/remote_sync_object_test.clj
index 71ceb441e18..9d6154aa582 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/models/remote_sync_object_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/models/remote_sync_object_test.clj
@@ -1,6 +1,6 @@
(ns metabase-enterprise.remote-sync.models.remote-sync-object-test
"Unit tests for the remote-sync-object namespace.
- Tests the public methods: dirty-global? and dirty-for-global."
+ Tests the public methods: dirty? and dirty-objects."
(:require
[clojure.test :refer :all]
[metabase-enterprise.remote-sync.models.remote-sync-object :as rs-object]
@@ -14,15 +14,15 @@
(use-fixtures :each th/clean-remote-sync-state)
;;; ------------------------------------------------------------------------------------------------
-;;; Tests for dirty-global?
+;;; Tests for dirty?
;;; ------------------------------------------------------------------------------------------------
-(deftest dirty-global?-no-objects-test
- (testing "dirty-global? returns false when no remote sync objects exist"
- (is (false? (rs-object/dirty-global?)))))
+(deftest dirty?-no-objects-test
+ (testing "dirty? returns false when no remote sync objects exist"
+ (is (false? (rs-object/dirty?)))))
-(deftest dirty-global?-synced-objects-test
- (testing "dirty-global? returns false when all objects are synced"
+(deftest dirty?-synced-objects-test
+ (testing "dirty? returns false when all objects are synced"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -36,10 +36,10 @@
:model_name (:name card)
:status "synced"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (is (false? (rs-object/dirty-global?))))))
+ (is (false? (rs-object/dirty?))))))
-(deftest dirty-global?-pending-objects-test
- (testing "dirty-global? returns true when objects have pending status"
+(deftest dirty?-pending-objects-test
+ (testing "dirty? returns true when objects have pending status"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -53,10 +53,10 @@
:model_name (:name card)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (is (true? (rs-object/dirty-global?))))))
+ (is (true? (rs-object/dirty?))))))
-(deftest dirty-global?-error-objects-test
- (testing "dirty-global? returns true when objects have error status"
+(deftest dirty?-error-objects-test
+ (testing "dirty? returns true when objects have error status"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -70,10 +70,10 @@
:model_name (:name card)
:status "error"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (is (true? (rs-object/dirty-global?))))))
+ (is (true? (rs-object/dirty?))))))
-(deftest dirty-global?-mixed-status-test
- (testing "dirty-global? returns true when some objects are not synced"
+(deftest dirty?-mixed-status-test
+ (testing "dirty? returns true when some objects are not synced"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -97,18 +97,18 @@
:model_name (:name card2)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (is (true? (rs-object/dirty-global?))))))
+ (is (true? (rs-object/dirty?))))))
;;; ------------------------------------------------------------------------------------------------
-;;; Tests for dirty-for-global
+;;; Tests for dirty-objects
;;; ------------------------------------------------------------------------------------------------
-(deftest dirty-for-global-no-objects-test
- (testing "dirty-for-global returns empty when no remote sync objects exist"
- (is (empty? (rs-object/dirty-for-global)))))
+(deftest dirty-objects-no-objects-test
+ (testing "dirty-objects returns empty when no remote sync objects exist"
+ (is (empty? (rs-object/dirty-objects)))))
-(deftest dirty-for-global-synced-objects-test
- (testing "dirty-for-global returns empty when all objects are synced"
+(deftest dirty-objects-synced-objects-test
+ (testing "dirty-objects returns empty when all objects are synced"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -122,10 +122,10 @@
:model_name (:name card)
:status "synced"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (is (empty? (rs-object/dirty-for-global))))))
+ (is (empty? (rs-object/dirty-objects))))))
-(deftest dirty-for-global-returns-dirty-items-test
- (testing "dirty-for-global returns items with non-synced status"
+(deftest dirty-objects-returns-dirty-items-test
+ (testing "dirty-objects returns items with non-synced status"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -139,7 +139,7 @@
:model_name (:name card)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id card) (:id item)))
@@ -147,8 +147,8 @@
(is (= "card" (:model item)))
(is (= "pending" (:sync_status item))))))))
-(deftest dirty-for-global-multiple-model-types-test
- (testing "dirty-for-global with multiple model types"
+(deftest dirty-objects-multiple-model-types-test
+ (testing "dirty-objects with multiple model types"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -170,7 +170,7 @@
:model_name (:name dashboard)
:status "error"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)
+ (let [dirty-items (rs-object/dirty-objects)
models (set (map :model dirty-items))]
(is (= 2 (count dirty-items)))
(is (= #{"card" "dashboard"} models))
@@ -178,8 +178,8 @@
(is (= "pending" (:sync_status (first (get items-by-model "card")))))
(is (= "error" (:sync_status (first (get items-by-model "dashboard"))))))))))
-(deftest dirty-for-global-includes-collections-test
- (testing "dirty-for-global includes collection items"
+(deftest dirty-objects-includes-collections-test
+ (testing "dirty-objects includes collection items"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -188,15 +188,15 @@
:model_name (:name collection)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id collection) (:id item)))
(is (= "Test Collection" (:name item)))
(is (= "collection" (:model item))))))))
-(deftest dirty-for-global-filters-synced-items-test
- (testing "dirty-for-global filters out synced items"
+(deftest dirty-objects-filters-synced-items-test
+ (testing "dirty-objects filters out synced items"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -220,13 +220,13 @@
:model_name (:name card2)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)
+ (let [dirty-items (rs-object/dirty-objects)
names (set (map :name dirty-items))]
(is (= 1 (count dirty-items)))
(is (= #{"Pending Card"} names))))))
-(deftest dirty-for-global-includes-snippets-test
- (testing "dirty-for-global includes snippet items"
+(deftest dirty-objects-includes-snippets-test
+ (testing "dirty-objects includes snippet items"
(mt/with-temp
[:model/Collection snip-collection {:name "Snippet Collection"
:namespace "snippets"
@@ -240,15 +240,15 @@
:model_name (:name snippet)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id snippet) (:id item)))
(is (= "Test Snippet" (:name item)))
(is (= "snippet" (:model item))))))))
-(deftest dirty-for-global-includes-documents-test
- (testing "dirty-for-global includes document items"
+(deftest dirty-objects-includes-documents-test
+ (testing "dirty-objects includes document items"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -260,7 +260,7 @@
:model_name (:name document)
:status "error"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id document) (:id item)))
@@ -268,8 +268,8 @@
(is (= "document" (:model item)))
(is (= "error" (:sync_status item))))))))
-(deftest dirty-for-global-all-model-types-test
- (testing "dirty-for-global handles all supported model types"
+(deftest dirty-objects-all-model-types-test
+ (testing "dirty-objects handles all supported model types"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -316,7 +316,7 @@
:model_name (:name snippet)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)
+ (let [dirty-items (rs-object/dirty-objects)
models (set (map :model dirty-items))]
(is (= 5 (count dirty-items)))
(is (= #{"collection" "card" "dashboard" "document" "snippet"} models))))))
@@ -326,10 +326,10 @@
;;; ------------------------------------------------------------------------------------------------
(deftest dirty-functions-consistency-test
- (testing "dirty-global? and dirty-for-global are consistent"
+ (testing "dirty? and dirty-objects are consistent"
(testing "when no dirty items exist"
- (is (false? (rs-object/dirty-global?)))
- (is (empty? (rs-object/dirty-for-global))))
+ (is (false? (rs-object/dirty?)))
+ (is (empty? (rs-object/dirty-objects))))
(testing "when dirty items exist"
(mt/with-temp
@@ -345,15 +345,15 @@
:model_name (:name card)
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (is (true? (rs-object/dirty-global?)))
- (is (= 1 (count (rs-object/dirty-for-global))))))))
+ (is (true? (rs-object/dirty?)))
+ (is (= 1 (count (rs-object/dirty-objects))))))))
;;; ------------------------------------------------------------------------------------------------
;;; Tests for Table, Field, and Segment in dirty-state queries
;;; ------------------------------------------------------------------------------------------------
-(deftest dirty-for-global-includes-tables-test
- (testing "dirty-for-global includes table items with table_id and table_name"
+(deftest dirty-objects-includes-tables-test
+ (testing "dirty-objects includes table items with table_id and table_name"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -368,7 +368,7 @@
:model_table_name "Test Table"
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id table) (:id item)))
@@ -379,8 +379,8 @@
(is (= (:id table) (:table_id item)))
(is (= "Test Table" (:table_name item))))))))
-(deftest dirty-for-global-includes-fields-test
- (testing "dirty-for-global includes field items with table_id and table_name from parent table"
+(deftest dirty-objects-includes-fields-test
+ (testing "dirty-objects includes field items with table_id and table_name from parent table"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -399,7 +399,7 @@
:model_table_name "Test Table"
:status "update"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id field) (:id item)))
@@ -412,8 +412,8 @@
(is (= (:id table) (:table_id item)))
(is (= "Test Table" (:table_name item))))))))
-(deftest dirty-for-global-includes-segments-test
- (testing "dirty-for-global includes segment items with table_id and table_name from parent table"
+(deftest dirty-objects-includes-segments-test
+ (testing "dirty-objects includes segment items with table_id and table_name from parent table"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -437,7 +437,7 @@
:model_table_name "Test Table"
:status "create"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)]
+ (let [dirty-items (rs-object/dirty-objects)]
(is (= 1 (count dirty-items)))
(let [item (first dirty-items)]
(is (= (:id segment) (:id item)))
@@ -450,8 +450,8 @@
(is (= (:id table) (:table_id item)))
(is (= "Test Table" (:table_name item))))))))
-(deftest dirty-for-global-all-model-types-including-new-test
- (testing "dirty-for-global handles all supported model types including Table, Field, and Segment"
+(deftest dirty-objects-all-model-types-including-new-test
+ (testing "dirty-objects handles all supported model types including Table, Field, and Segment"
(mt/with-temp
[:model/Collection collection {:name "Test Collection"
:location "/"}
@@ -531,7 +531,7 @@
:model_table_name "Table Item"
:status "pending"
:status_changed_at (java.time.OffsetDateTime/now)}]
- (let [dirty-items (rs-object/dirty-for-global)
+ (let [dirty-items (rs-object/dirty-objects)
models (set (map :model dirty-items))]
(is (= 8 (count dirty-items)))
(is (= #{"collection" "card" "dashboard" "document" "nativequerysnippet" "table" "field" "segment"} models))))))
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/permissions_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/permissions_test.clj
index be724afee22..72f52ca4666 100644
--- a/enterprise/backend/test/metabase_enterprise/remote_sync/permissions_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/permissions_test.clj
@@ -555,3 +555,188 @@
(t2/update! :model/Table table-id
{:is_published false
:collection_id nil}))))))))))
+
+;;; ------------------------------------------------ Transform Remote Sync Tests ------------------------------------------------
+;; Transforms are globally read-only when remote-sync is enabled and remote-sync-type is :read-only.
+;; This is different from other models which use collection-based editability checks.
+
+(deftest transform-superuser-can-read-test
+ (testing "can_read should be true for superusers"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :crowberto)
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}
+ :model/Transform {transform-id :id} {:name "Test Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}]
+ (is (true? (mi/can-read? (t2/select-one :model/Transform :id transform-id)))
+ "Superuser should be able to read transforms"))))))
+
+(deftest transform-non-superuser-cannot-read-test
+ (testing "can_read should be false for non-superusers"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :rasta)
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}
+ :model/Transform {transform-id :id} {:name "Test Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}]
+ (is (false? (mi/can-read? (t2/select-one :model/Transform :id transform-id)))
+ "Non-superuser should not be able to read transforms"))))))
+
+(deftest transform-globally-read-only-when-remote-sync-enabled-test
+ (testing "can_write should be false for ALL transforms when remote-sync is enabled and type is read-only"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :crowberto)
+ (mt/with-temporary-setting-values [settings/remote-sync-url "https://github.com/test/repo.git"
+ settings/remote-sync-type :read-only]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}
+ :model/Transform {transform-id :id} {:name "Test Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}]
+ (is (false? (mi/can-write? (t2/select-one :model/Transform :id transform-id)))
+ "All transforms should be read-only when remote-sync is enabled and type is read-only")))))))
+
+(deftest transform-writable-when-remote-sync-read-write-test
+ (testing "can_write should be true for transforms when remote-sync-type is read-write"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :crowberto)
+ (mt/with-temporary-setting-values [settings/remote-sync-url "https://github.com/test/repo.git"
+ settings/remote-sync-type :read-write]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}
+ :model/Transform {transform-id :id} {:name "Test Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}]
+ (is (true? (mi/can-write? (t2/select-one :model/Transform :id transform-id)))
+ "Transforms should be writable when remote-sync-type is read-write")))))))
+
+(deftest transform-writable-when-remote-sync-disabled-test
+ (testing "can_write should be true for transforms when remote-sync is not enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :crowberto)
+ ;; remote-sync-url is not set, so remote-sync-enabled returns false
+ (mt/with-temporary-setting-values [settings/remote-sync-url nil
+ settings/remote-sync-type :read-only]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}
+ :model/Transform {transform-id :id} {:name "Test Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}]
+ (is (true? (mi/can-write? (t2/select-one :model/Transform :id transform-id)))
+ "Transforms should be writable when remote-sync is not enabled")))))))
+
+(deftest transform-creation-blocked-in-read-only-mode-test
+ (testing "can_create should be false for transforms when remote-sync is enabled and type is read-only"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :crowberto)
+ (mt/with-temporary-setting-values [settings/remote-sync-url "https://github.com/test/repo.git"
+ settings/remote-sync-type :read-only]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}]
+ (is (false? (mi/can-create? :model/Transform {:name "New Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}))
+ "Transform creation should be blocked when remote-sync is enabled and type is read-only")))))))
+
+(deftest transform-creation-allowed-in-read-write-mode-test
+ (testing "can_create should be true for transforms when remote-sync-type is read-write"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :crowberto)
+ (mt/with-temporary-setting-values [settings/remote-sync-url "https://github.com/test/repo.git"
+ settings/remote-sync-type :read-write]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}]
+ (is (true? (mi/can-create? :model/Transform {:name "New Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}))
+ "Transform creation should be allowed when remote-sync-type is read-write")))))))
+
+(deftest transform-non-superuser-cannot-write-test
+ (testing "can_write should be false for non-superusers even when remote-sync-type is read-write"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :rasta)
+ (mt/with-temporary-setting-values [settings/remote-sync-type :read-write]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}
+ :model/Transform {transform-id :id} {:name "Test Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}]
+ (is (false? (mi/can-write? (t2/select-one :model/Transform :id transform-id)))
+ "Non-superuser should not be able to write transforms even when remote-sync-type is read-write")))))))
+
+(deftest transform-non-superuser-cannot-create-test
+ (testing "can_create should be false for non-superusers even when remote-sync-type is read-write"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-current-user (mt/user->id :rasta)
+ (mt/with-temporary-setting-values [settings/remote-sync-type :read-write]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection"
+ :namespace :transforms}]
+ (is (false? (mi/can-create? :model/Transform {:name "New Transform"
+ :collection_id coll-id
+ :source {:type "query"
+ :query {:database (mt/id)
+ :type :query
+ :query {:source-table (mt/id :venues)}}}
+ :target {:database (mt/id)
+ :type "table"
+ :schema "public"
+ :name "target_table"}}))
+ "Non-superuser should not be able to create transforms even when remote-sync-type is read-write")))))))
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/spec_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/spec_test.clj
new file mode 100644
index 00000000000..64f63548e25
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/spec_test.clj
@@ -0,0 +1,305 @@
+(ns metabase-enterprise.remote-sync.spec-test
+ (:require
+ [clojure.set :as set]
+ [clojure.test :refer :all]
+ [metabase-enterprise.remote-sync.spec :as spec]
+ [metabase.test :as mt]))
+
+(set! *warn-on-reflection* true)
+
+;;; ------------------------------------------------ Spec Validation Tests ---------------------------------------------
+
+(deftest all-specs-have-required-keys-test
+ (testing "Every spec has all required keys"
+ (let [required-keys #{:model-type :model-key :identity :events :eligibility
+ :archived-key :tracking :removal :export-path :enabled?}]
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (let [missing-keys (set/difference required-keys (set (keys spec)))]
+ (is (empty? missing-keys)
+ (str "Missing keys: " missing-keys))))))))
+
+(deftest all-specs-have-valid-identity-test
+ (testing "Every spec has a valid identity type"
+ (let [valid-identity-types #{:entity-id :path :hybrid}]
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (is (contains? valid-identity-types (:identity spec))
+ (str "Invalid identity type: " (:identity spec))))))))
+
+(deftest path-based-specs-have-path-keys-test
+ (testing "Specs with :path or :hybrid identity have :path-keys"
+ (doseq [[model-key spec] spec/remote-sync-specs
+ :when (#{:path :hybrid} (:identity spec))]
+ (testing (str "Spec for " model-key)
+ (is (vector? (:path-keys spec))
+ "path-keys should be a vector")
+ (is (seq (:path-keys spec))
+ "path-keys should not be empty")))))
+
+(deftest all-specs-have-valid-eligibility-test
+ (testing "Every spec has a valid eligibility type"
+ (let [valid-eligibility-types #{:collection :published-table :parent-table :setting}]
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (is (contains? valid-eligibility-types (get-in spec [:eligibility :type]))
+ (str "Invalid eligibility type: " (get-in spec [:eligibility :type]))))))))
+
+(deftest all-specs-have-valid-events-test
+ (testing "Every spec has valid events configuration"
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (is (keyword? (get-in spec [:events :prefix]))
+ "events :prefix should be a keyword")
+ (is (vector? (get-in spec [:events :types]))
+ "events :types should be a vector")
+ (is (every? #{:create :update :delete} (get-in spec [:events :types]))
+ "events :types should only contain :create, :update, :delete")))))
+
+(deftest all-specs-have-valid-tracking-test
+ (testing "Every spec has valid tracking configuration"
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (let [tracking (:tracking spec)]
+ (is (or (contains? tracking :select-fields)
+ (contains? tracking :hydrate-query))
+ "tracking should have either :select-fields or :hydrate-query")
+ (is (map? (:field-mappings tracking))
+ "tracking :field-mappings should be a map"))))))
+
+(deftest all-specs-have-valid-removal-test
+ (testing "Every spec has valid removal configuration"
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (is (set? (get-in spec [:removal :statuses]))
+ "removal :statuses should be a set")
+ ;; :scope-key is optional - if present it should be a keyword
+ (when-let [scope-key (get-in spec [:removal :scope-key])]
+ (is (keyword? scope-key)
+ "removal :scope-key should be a keyword when present"))))))
+
+(deftest all-specs-have-valid-export-path-test
+ (testing "Every spec has valid export-path configuration"
+ (let [valid-path-types #{:collection-entity :table-path :field-path
+ :segment-path :transform-path :transform-tag-path}]
+ (doseq [[model-key spec] spec/remote-sync-specs]
+ (testing (str "Spec for " model-key)
+ (is (contains? valid-path-types (get-in spec [:export-path :type]))
+ (str "Invalid export-path type: " (get-in spec [:export-path :type]))))))))
+
+;;; ------------------------------------------------ Helper Function Tests ---------------------------------------------
+
+(deftest spec-for-model-type-test
+ (testing "spec-for-model-type returns correct spec"
+ (is (= :model/Card (:model-key (spec/spec-for-model-type "Card"))))
+ (is (= :model/Dashboard (:model-key (spec/spec-for-model-type "Dashboard"))))
+ (is (= :model/Table (:model-key (spec/spec-for-model-type "Table"))))
+ (is (nil? (spec/spec-for-model-type "NonExistent")))))
+
+(deftest spec-for-model-key-test
+ (testing "spec-for-model-key returns correct spec"
+ (is (= "Card" (:model-type (spec/spec-for-model-key :model/Card))))
+ (is (= "Dashboard" (:model-type (spec/spec-for-model-key :model/Dashboard))))
+ (is (nil? (spec/spec-for-model-key :model/NonExistent)))))
+
+(deftest all-model-types-test
+ (testing "all-model-types returns all model type strings"
+ (let [types (spec/all-model-types)]
+ (is (set? types))
+ (is (contains? types "Card"))
+ (is (contains? types "Dashboard"))
+ (is (contains? types "Table"))
+ (is (contains? types "Field"))
+ (is (contains? types "Segment"))
+ (is (contains? types "Transform"))
+ (is (contains? types "TransformTag"))
+ (is (= 11 (count types))))))
+
+(deftest specs-by-identity-type-test
+ (testing "specs-by-identity-type filters correctly"
+ (mt/with-temporary-setting-values [remote-sync-transforms false]
+ (let [entity-id-specs (spec/specs-by-identity-type :entity-id)
+ path-specs (spec/specs-by-identity-type :path)
+ hybrid-specs (spec/specs-by-identity-type :hybrid)]
+ (testing "entity-id specs"
+ (is (contains? entity-id-specs :model/Card))
+ (is (contains? entity-id-specs :model/Dashboard))
+ (is (contains? entity-id-specs :model/Collection))
+ (is (not (contains? entity-id-specs :model/Table)))
+ (is (not (contains? entity-id-specs :model/Field))))
+ (testing "path specs"
+ (is (contains? path-specs :model/Table))
+ (is (contains? path-specs :model/Field))
+ (is (not (contains? path-specs :model/Card))))
+ (testing "hybrid specs"
+ (is (contains? hybrid-specs :model/Segment))
+ (is (not (contains? hybrid-specs :model/Card))))))))
+
+(deftest excluded-model-types-test
+ (testing "excluded-model-types when transforms disabled"
+ (mt/with-temporary-setting-values [remote-sync-transforms false]
+ (let [excluded (spec/excluded-model-types)]
+ (is (contains? excluded "Transform"))
+ (is (contains? excluded "TransformTag"))
+ (is (not (contains? excluded "Card")))
+ (is (not (contains? excluded "Dashboard"))))))
+
+ (testing "excluded-model-types when transforms enabled"
+ (mt/with-temporary-setting-values [remote-sync-transforms true]
+ (let [excluded (spec/excluded-model-types)]
+ (is (empty? excluded))))))
+
+(deftest spec-enabled?-test
+ (testing "spec-enabled? with always-enabled spec"
+ (is (true? (spec/spec-enabled? {:enabled? true}))))
+
+ (testing "spec-enabled? with setting-based spec"
+ (mt/with-temporary-setting-values [remote-sync-transforms false]
+ (is (false? (spec/spec-enabled? {:enabled? :remote-sync-transforms}))))
+
+ (mt/with-temporary-setting-values [remote-sync-transforms true]
+ (is (true? (spec/spec-enabled? {:enabled? :remote-sync-transforms}))))))
+
+(deftest enabled-specs-test
+ (testing "enabled-specs excludes setting-disabled specs"
+ (mt/with-temporary-setting-values [remote-sync-transforms false]
+ (let [enabled (spec/enabled-specs)]
+ (is (contains? enabled :model/Card))
+ (is (contains? enabled :model/Dashboard))
+ (is (not (contains? enabled :model/Transform)))
+ (is (not (contains? enabled :model/TransformTag)))))
+
+ (mt/with-temporary-setting-values [remote-sync-transforms true]
+ (let [enabled (spec/enabled-specs)]
+ (is (contains? enabled :model/Card))
+ (is (contains? enabled :model/Transform))
+ (is (contains? enabled :model/TransformTag))))))
+
+;;; ------------------------------------------------ Event Helper Tests ------------------------------------------------
+
+(deftest determine-status-test
+ (testing "determine-status for create event"
+ (let [spec (spec/spec-for-model-key :model/Card)]
+ (is (= "create" (spec/determine-status spec :event/card-create {:archived false})))))
+
+ (testing "determine-status for update event"
+ (let [spec (spec/spec-for-model-key :model/Card)]
+ (is (= "update" (spec/determine-status spec :event/card-update {:archived false})))))
+
+ (testing "determine-status for delete event"
+ (let [spec (spec/spec-for-model-key :model/Card)]
+ (is (= "delete" (spec/determine-status spec :event/card-delete {:archived false})))))
+
+ (testing "determine-status for archived object returns delete"
+ (let [spec (spec/spec-for-model-key :model/Card)]
+ (is (= "delete" (spec/determine-status spec :event/card-update {:archived true}))))))
+
+(deftest event-keywords-test
+ (testing "event-keywords generates correct keywords"
+ (let [spec (spec/spec-for-model-key :model/Card)
+ keywords (spec/event-keywords spec)]
+ (is (= :metabase-enterprise.remote-sync.events/card-change-event (:parent keywords)))
+ (is (= :event/card-create (:create keywords)))
+ (is (= :event/card-update (:update keywords)))
+ (is (= :event/card-delete (:delete keywords))))))
+
+;;; -------------------------------------------- Build Sync Object Fields Tests ----------------------------------------
+
+(deftest build-sync-object-fields-test
+ (testing "build-sync-object-fields with simple mappings"
+ (let [spec (spec/spec-for-model-key :model/Dashboard)
+ details {:name "My Dashboard" :collection_id 123}
+ fields (spec/build-sync-object-fields spec details)]
+ (is (= "My Dashboard" (:model_name fields)))
+ (is (= 123 (:model_collection_id fields)))))
+
+ (testing "build-sync-object-fields with transform function"
+ (let [spec (spec/spec-for-model-key :model/Card)
+ details {:name "My Card" :collection_id 456 :display :table}
+ fields (spec/build-sync-object-fields spec details)]
+ (is (= "My Card" (:model_name fields)))
+ (is (= 456 (:model_collection_id fields)))
+ (is (= "table" (:model_display fields)))))
+
+ (testing "build-sync-object-fields with nil details returns nil"
+ (let [spec (spec/spec-for-model-key :model/Card)]
+ (is (nil? (spec/build-sync-object-fields spec nil))))))
+
+;;; ------------------------------------------- Removal Path Building Tests ------------------------------------------
+
+;; Note: build-all-removal-paths uses serdes/storage-path to generate paths.
+;; The actual paths are tested in impl_test.clj integration tests which verify
+;; the full export/import cycle with real database entities.
+
+;;; --------------------------------------------- Select Fields for Sync Tests -----------------------------------------
+
+(deftest select-fields-for-sync-test
+ (testing "select-fields-for-sync returns spec fields"
+ (is (= [:name :collection_id :display]
+ (spec/select-fields-for-sync "Card")))
+ (is (= [:name :collection_id]
+ (spec/select-fields-for-sync "Dashboard")))
+ (is (= [:name :id]
+ (spec/select-fields-for-sync "NativeQuerySnippet"))))
+
+ (testing "select-fields-for-sync returns default for unknown type"
+ (is (= [:id :name :collection_id]
+ (spec/select-fields-for-sync "UnknownModel")))))
+
+;;; --------------------------------------------- Export Scope Tests -------------------------------------------------
+
+(deftest all-specs-have-valid-export-scope-test
+ (testing "Every spec with :export-scope has a valid value"
+ (let [valid-export-scopes #{:root-collections :root-only :all :derived}]
+ (doseq [[model-key spec] spec/remote-sync-specs
+ :when (contains? spec :export-scope)]
+ (testing (str "Spec for " model-key)
+ (is (contains? valid-export-scopes (:export-scope spec))
+ (str "Invalid export-scope: " (:export-scope spec))))))))
+
+(deftest export-scope-required-for-certain-models-test
+ (testing "Collection spec has :root-collections export-scope"
+ (is (= :root-collections (:export-scope (spec/spec-for-model-key :model/Collection)))))
+
+ (testing "Transform spec has :root-only export-scope"
+ (is (= :root-only (:export-scope (spec/spec-for-model-key :model/Transform)))))
+
+ (testing "TransformTag spec has :all export-scope"
+ (is (= :all (:export-scope (spec/spec-for-model-key :model/TransformTag)))))
+
+ (testing "Other collection-based models have no export-scope (defaults to :derived)"
+ (is (nil? (:export-scope (spec/spec-for-model-key :model/Card))))
+ (is (nil? (:export-scope (spec/spec-for-model-key :model/Dashboard))))))
+
+;;; --------------------------------------------- Query Export Roots Tests -------------------------------------------
+
+(deftest query-export-roots-collection-eligibility-test
+ (testing "query-export-roots with :collection eligibility and :derived scope returns nil"
+ (let [card-spec (spec/spec-for-model-key :model/Card)]
+ (is (nil? (spec/query-export-roots card-spec)))))
+
+ (testing "query-export-roots with :collection eligibility and :root-collections scope queries collections"
+ ;; This test verifies the multimethod dispatches correctly - actual database queries
+ ;; are tested in impl_test.clj integration tests
+ (let [collection-spec (spec/spec-for-model-key :model/Collection)]
+ ;; The function should run without error (returns empty when no remote-synced collections)
+ (is (not (nil? (spec/query-export-roots collection-spec)))))))
+
+(deftest query-export-roots-setting-eligibility-test
+ (testing "query-export-roots with :setting eligibility returns nil when setting disabled"
+ (mt/with-temporary-setting-values [remote-sync-transforms false]
+ (let [transform-spec (spec/spec-for-model-key :model/Transform)
+ tag-spec (spec/spec-for-model-key :model/TransformTag)]
+ (is (nil? (spec/query-export-roots transform-spec)))
+ (is (nil? (spec/query-export-roots tag-spec)))))))
+
+(deftest query-export-roots-derived-eligibility-test
+ (testing "query-export-roots with :published-table eligibility returns nil (derived)"
+ (let [table-spec (spec/spec-for-model-key :model/Table)]
+ (is (nil? (spec/query-export-roots table-spec)))))
+
+ (testing "query-export-roots with :parent-table eligibility returns nil (derived)"
+ (let [field-spec (spec/spec-for-model-key :model/Field)
+ segment-spec (spec/spec-for-model-key :model/Segment)]
+ (is (nil? (spec/query-export-roots field-spec)))
+ (is (nil? (spec/query-export-roots segment-spec))))))
diff --git a/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj b/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj
new file mode 100644
index 00000000000..a59d73abfba
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/remote_sync/transforms_test.clj
@@ -0,0 +1,404 @@
+(ns metabase-enterprise.remote-sync.transforms-test
+ "Tests for transform sync functionality in remote-sync."
+ (:require
+ [clojure.string :as str]
+ [clojure.test :refer :all]
+ [java-time.api :as t]
+ [metabase-enterprise.remote-sync.impl :as impl]
+ [metabase-enterprise.remote-sync.models.remote-sync-object :as sync-object]
+ [metabase-enterprise.remote-sync.source.protocol :as source.p]
+ [metabase-enterprise.remote-sync.test-helpers :as test-helpers]
+ [metabase.collections.models.collection :as collection]
+ [metabase.events.core :as events]
+ [metabase.search.core :as search]
+ [metabase.test :as mt]
+ [metabase.test.fixtures :as fixtures]
+ [metabase.util :as u]
+ [toucan2.core :as t2]))
+
+(use-fixtures :once (fixtures/initialize :db))
+
+#_{:clj-kondo/ignore [:metabase/validate-deftest]}
+(use-fixtures :each (fn [f]
+ (mt/with-dynamic-fn-redefs [search/reindex! (constantly nil)]
+ (test-helpers/clean-remote-sync-state f))))
+
+(deftest transform-event-creates-sync-object-when-setting-enabled-test
+ (testing "Creating a transform creates a RemoteSyncObject entry when remote-sync-transforms is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms true
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Test Transform" :collection_id coll-id}]
+ (events/publish-event! :event/transform-create {:object transform})
+ (is (t2/exists? :model/RemoteSyncObject
+ :model_type "Transform"
+ :model_id (:id transform))
+ "Transform should be tracked when remote-sync-transforms is enabled"))))))
+
+(deftest transform-event-ignored-when-setting-disabled-test
+ (testing "Transform events are ignored when remote-sync-transforms is disabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Test Transform" :collection_id coll-id}]
+ (events/publish-event! :event/transform-create {:object transform})
+ (is (not (t2/exists? :model/RemoteSyncObject
+ :model_type "Transform"
+ :model_id (:id transform)))
+ "Transform should NOT be tracked when remote-sync-transforms is disabled"))))))
+
+(deftest transform-event-updates-sync-object-test
+ (testing "Updating a transform updates the RemoteSyncObject entry when setting is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms true
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Test Transform" :collection_id coll-id}]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "Transform"
+ :model_id (:id transform)
+ :model_name "Test Transform"
+ :model_collection_id coll-id
+ :status "synced"
+ :status_changed_at (t/offset-date-time)})
+ (events/publish-event! :event/transform-update {:object transform})
+ (let [entry (t2/select-one :model/RemoteSyncObject
+ :model_type "Transform"
+ :model_id (:id transform))]
+ (is (= "update" (:status entry))
+ "Transform should have 'update' status after modification")))))))
+
+(deftest transform-tag-event-creates-sync-object-test
+ (testing "Creating a transform tag creates a RemoteSyncObject entry when transform sync is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms true
+ remote-sync-enabled true]
+ (mt/with-temp [:model/TransformTag tag {:name "Test Tag"}]
+ (events/publish-event! :event/transform-tag-create {:object tag})
+ (is (t2/exists? :model/RemoteSyncObject
+ :model_type "TransformTag"
+ :model_id (:id tag))
+ "TransformTag should be tracked in RemoteSyncObject"))))))
+
+(deftest transform-tag-event-ignored-when-disabled-test
+ (testing "TransformTag events are ignored when transform sync is disabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/TransformTag tag {:name "Test Tag"}]
+ (events/publish-event! :event/transform-tag-create {:object tag})
+ (is (not (t2/exists? :model/RemoteSyncObject
+ :model_type "TransformTag"
+ :model_id (:id tag)))
+ "TransformTag should NOT be tracked when sync is disabled"))))))
+
+(deftest transforms-namespace-collection-tracked-when-setting-enabled-test
+ (testing "Transforms-namespace collection creates RemoteSyncObject entry when setting is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms true
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection coll {:name "Transforms Collection" :namespace collection/transforms-ns :location "/"}]
+ (events/publish-event! :event/collection-create {:object coll :user-id (mt/user->id :rasta)})
+ (is (t2/exists? :model/RemoteSyncObject
+ :model_type "Collection"
+ :model_id (:id coll))
+ "Transforms-namespace collection should be tracked when setting is enabled"))))))
+
+(deftest transforms-namespace-collection-ignored-when-setting-disabled-test
+ (testing "Transforms-namespace collection is NOT tracked when setting is disabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection coll {:name "Transforms Collection" :namespace collection/transforms-ns :location "/"}]
+ (events/publish-event! :event/collection-create {:object coll :user-id (mt/user->id :rasta)})
+ (is (not (t2/exists? :model/RemoteSyncObject
+ :model_type "Collection"
+ :model_id (:id coll)))
+ "Transforms-namespace collection should NOT be tracked when setting is disabled"))))))
+
+(deftest enable-transform-sync-marks-all-transforms-test
+ (testing "Enabling transform sync marks all existing transforms, tags, and collections for initial sync"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Existing Transform" :collection_id coll-id}
+ :model/TransformTag tag {:name "Existing Tag"}]
+ (is (zero? (t2/count :model/RemoteSyncObject :model_type [:in ["Transform" "TransformTag"]]))
+ "Should have no transform tracking entries initially")
+ (impl/sync-transform-tracking! true)
+ (is (t2/exists? :model/RemoteSyncObject
+ :model_type "Collection"
+ :model_id coll-id
+ :status "create")
+ "Transforms-namespace collection should be marked for initial sync")
+ (is (t2/exists? :model/RemoteSyncObject
+ :model_type "Transform"
+ :model_id (:id transform)
+ :status "create")
+ "Transform should be marked for initial sync")
+ (is (t2/exists? :model/RemoteSyncObject
+ :model_type "TransformTag"
+ :model_id (:id tag)
+ :status "create")
+ "TransformTag should be marked for initial sync"))))))
+
+(deftest disable-transform-sync-removes-all-tracking-test
+ (testing "Disabling transform sync removes all transform-related tracking entries"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Test Transform" :collection_id coll-id}
+ :model/TransformTag tag {:name "Test Tag"}]
+ (mt/with-temporary-setting-values [remote-sync-transforms true]
+ (t2/insert! :model/RemoteSyncObject
+ [{:model_type "Collection" :model_id coll-id :model_name "Transforms Collection" :status "synced" :status_changed_at (t/offset-date-time)}
+ {:model_type "Transform" :model_id (:id transform) :model_name "Test Transform" :status "synced" :status_changed_at (t/offset-date-time)}
+ {:model_type "TransformTag" :model_id (:id tag) :model_name "Test Tag" :status "synced" :status_changed_at (t/offset-date-time)}])
+ (is (= 3 (t2/count :model/RemoteSyncObject :model_type [:in ["Collection" "Transform" "TransformTag"]]
+ :model_id [:in [coll-id (:id transform) (:id tag)]]))
+ "Should have 3 tracking entries")
+ (impl/sync-transform-tracking! false)
+ (is (zero? (t2/count :model/RemoteSyncObject :model_type [:in ["Transform" "TransformTag"]]))
+ "Transform and TransformTag tracking entries should be removed")
+ (is (zero? (t2/count :model/RemoteSyncObject :model_type "Collection" :model_id coll-id))
+ "Transforms-namespace collection tracking entry should be removed")))))))
+
+(deftest transforms-included-in-dirty-check-when-enabled-test
+ (testing "Transforms are included in dirty check when setting is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms true
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Dirty Transform" :collection_id coll-id}]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "Transform"
+ :model_id (:id transform)
+ :model_name "Dirty Transform"
+ :status "update"
+ :status_changed_at (t/offset-date-time)})
+ (is (sync-object/dirty?)
+ "Should report dirty state when transform has changes"))))))
+
+(deftest transforms-excluded-from-dirty-check-when-disabled-test
+ (testing "Transforms are excluded from dirty check when setting is disabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/Collection {coll-id :id} {:name "Transforms Collection" :namespace collection/transforms-ns}
+ :model/Transform transform {:name "Dirty Transform" :collection_id coll-id}]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "Transform"
+ :model_id (:id transform)
+ :model_name "Dirty Transform"
+ :status "update"
+ :status_changed_at (t/offset-date-time)})
+ (is (not (sync-object/dirty?))
+ "Should NOT report dirty state when transform sync is disabled"))))))
+
+(deftest transform-tags-included-in-dirty-check-when-enabled-test
+ (testing "TransformTags are included in dirty check when transform sync is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms true
+ remote-sync-enabled true]
+ (mt/with-temp [:model/TransformTag tag {:name "Dirty Tag"}]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "TransformTag"
+ :model_id (:id tag)
+ :model_name "Dirty Tag"
+ :status "update"
+ :status_changed_at (t/offset-date-time)})
+ (is (sync-object/dirty?)
+ "Should report dirty state when transform tag has changes"))))))
+
+(deftest transform-tags-excluded-from-dirty-check-when-disabled-test
+ (testing "TransformTags are excluded from dirty check when transform sync is disabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-transforms false
+ remote-sync-enabled true]
+ (mt/with-temp [:model/TransformTag tag {:name "Dirty Tag"}]
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "TransformTag"
+ :model_id (:id tag)
+ :model_name "Dirty Tag"
+ :status "update"
+ :status_changed_at (t/offset-date-time)})
+ (is (not (sync-object/dirty?))
+ "Should NOT report dirty state when transform sync is disabled"))))))
+
+(deftest export-includes-transforms-namespace-collections-test
+ (testing "Export includes transforms-namespace collections when setting is enabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-type :read-write
+ remote-sync-transforms true
+ remote-sync-enabled true]
+ (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "export" :initiated_by (mt/user->id :rasta)})]
+ (mt/with-temp [:model/Collection {coll-id :id coll-eid :entity_id} {:name "Transforms Collection" :namespace collection/transforms-ns :entity_id "transforms-coll-xxx" :location "/"}
+ :model/Transform transform {:name "Export Transform" :collection_id coll-id}]
+ (let [saved-coll (t2/select-one :model/Collection :id coll-id)]
+ (is (= :transforms (:namespace saved-coll))
+ (str "Collection should have transforms namespace, got: " (:namespace saved-coll)))
+ (is (= "/" (:location saved-coll))
+ (str "Collection should have root location, got: " (:location saved-coll))))
+ (let [found-colls (t2/select-fn-set :entity_id :model/Collection :namespace (name collection/transforms-ns))]
+ (is (contains? found-colls coll-eid)
+ (str "Query should find the collection. Found: " found-colls ", expected to contain: " coll-eid)))
+ (t2/insert! :model/RemoteSyncObject
+ [{:model_type "Collection" :model_id coll-id :model_name "Transforms Collection" :status "create" :status_changed_at (t/offset-date-time)}
+ {:model_type "Transform" :model_id (:id transform) :model_name "Export Transform" :model_collection_id coll-id :status "create" :status_changed_at (t/offset-date-time)}])
+ (let [mock-source (test-helpers/create-mock-source)
+ result (impl/export! (source.p/snapshot mock-source) task-id "Test export")]
+ (is (= :success (:status result))
+ (str "Export should succeed. Result: " result))
+ (let [files-after-export (get @(:files-atom mock-source) "main")]
+ (is (some #(str/includes? % coll-eid)
+ (keys files-after-export))
+ "Export should include the transforms-namespace collection itself")))))))))
+
+(deftest export-excludes-transforms-when-setting-disabled-test
+ (testing "Export excludes transforms when setting is disabled"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-type :read-write
+ remote-sync-transforms false
+ remote-sync-enabled true]
+ (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "export" :initiated_by (mt/user->id :rasta)})]
+ (mt/with-temp [:model/Collection {rs-coll-id :id rs-coll-eid :entity_id} {:name "Remote Synced" :is_remote_synced true :entity_id "remote-synced-xxxx" :location "/"}]
+ (let [saved-coll (t2/select-one :model/Collection :id rs-coll-id)]
+ (is (true? (:is_remote_synced saved-coll))
+ (str "Collection should have is_remote_synced true, got: " (:is_remote_synced saved-coll)))
+ (is (= "/" (:location saved-coll))
+ (str "Collection should have root location, got: " (:location saved-coll))))
+ (let [found-colls (t2/select-fn-set :entity_id :model/Collection :is_remote_synced true :location "/")]
+ (is (contains? found-colls rs-coll-eid)
+ (str "Query should find the collection. Found: " found-colls ", expected to contain: " rs-coll-eid)))
+ (t2/insert! :model/RemoteSyncObject
+ {:model_type "Collection" :model_id rs-coll-id :model_name "Remote Synced" :status "create" :status_changed_at (t/offset-date-time)})
+ (let [mock-source (test-helpers/create-mock-source)
+ result (impl/export! (source.p/snapshot mock-source) task-id "Test export")]
+ (is (= :success (:status result))
+ (str "Export should succeed. Result: " result))
+ (let [files-after-export (get @(:files-atom mock-source) "main")]
+ (is (not (some #(str/includes? % "transforms/") (keys files-after-export)))
+ "Export should NOT include transform files when setting is disabled")))))))))
+
+(defn- generate-transform-yaml
+ "Generates YAML content for a transform."
+ [entity-id name]
+ (format "name: %s
+description: null
+entity_id: %s
+collection_id: null
+created_at: '2024-08-28T09:46:18.671622Z'
+creator_id: [email protected]
+source:
+ type: query
+ query:
+ database: 1
+ type: query
+ query:
+ source-table: 1
+target:
+ type: table
+ name: test_output
+ schema: PUBLIC
+serdes/meta:
+- id: %s
+ label: %s
+ model: Transform
+"
+ name entity-id entity-id (str/replace (u/lower-case-en name) #"\s+" "_")))
+
+(defn- generate-transforms-namespace-collection-yaml
+ "Generates YAML content for a transforms-namespace collection."
+ [entity-id name]
+ (format "name: %s
+description: null
+entity_id: %s
+slug: %s
+created_at: '2024-08-28T09:46:18.671622Z'
+archived: false
+type: null
+parent_id: null
+personal_owner_id: null
+namespace: transforms
+authority_level: null
+serdes/meta:
+- id: %s
+ label: %s
+ model: Collection
+archive_operation_id: null
+archived_directly: null
+is_sample: false
+"
+ name entity-id (str/replace (u/lower-case-en name) #"\s+" "_")
+ entity-id (str/replace (u/lower-case-en name) #"\s+" "_")))
+
+(deftest import-transforms-from-transforms-namespace-collection-test
+ (testing "Import brings in transforms located inside a transforms-namespace collection"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-enabled true]
+ (mt/with-model-cleanup [:model/Transform :model/Collection]
+ (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "import" :initiated_by (mt/user->id :rasta)})
+ coll-entity-id "transforms-coll-xxx"
+ transform-entity-id "test-transform-xxxxx"
+ test-files {"main" {(str "collections/" coll-entity-id "_transforms/" coll-entity-id "_transforms.yaml")
+ (generate-transforms-namespace-collection-yaml coll-entity-id "Transforms")
+ (str "collections/" coll-entity-id "_transforms/transforms/" transform-entity-id "_test_transform.yaml")
+ (generate-transform-yaml transform-entity-id "Test Transform")}}
+ mock-source (test-helpers/create-mock-source :initial-files test-files)
+ result (impl/import! (source.p/snapshot mock-source) task-id)]
+ (is (= :success (:status result)))
+ (is (t2/exists? :model/Collection :entity_id coll-entity-id :namespace "transforms")
+ "Transforms-namespace collection should be imported")
+ (is (t2/exists? :model/Transform :entity_id transform-entity-id)
+ "Transform should be imported from transforms-namespace collection")))))))
+
+(deftest archived-transforms-namespace-collection-excluded-from-export-test
+ (testing "When a transforms-namespace collection is archived, it and its children are excluded from export"
+ (mt/with-premium-features #{:transforms}
+ (mt/with-temporary-setting-values [remote-sync-type :read-write
+ remote-sync-transforms true
+ remote-sync-enabled true]
+ (let [task-id (t2/insert-returning-pk! :model/RemoteSyncTask {:sync_task_type "export" :initiated_by (mt/user->id :rasta)})]
+ (mt/with-temp [:model/Collection {coll-id :id coll-eid :entity_id}
+ {:name "Transforms Collection"
+ :namespace collection/transforms-ns
+ :entity_id "arch-transforms-collx"
+ :location "/"}
+ :model/Transform {transform-id :id transform-eid :entity_id}
+ {:name "Child Transform"
+ :collection_id coll-id
+ :entity_id "child-transform-xxxxx"}]
+ (t2/insert! :model/RemoteSyncObject
+ [{:model_type "Collection"
+ :model_id coll-id
+ :model_name "Transforms Collection"
+ :status "synced"
+ :status_changed_at (t/offset-date-time)}
+ {:model_type "Transform"
+ :model_id transform-id
+ :model_name "Child Transform"
+ :model_collection_id coll-id
+ :status "synced"
+ :status_changed_at (t/offset-date-time)}])
+ (let [initial-files {"main" {(str "collections/" coll-eid "_transforms_collection/" coll-eid "_transforms_collection.yaml")
+ (generate-transforms-namespace-collection-yaml coll-eid "Transforms Collection")
+ (str "collections/" coll-eid "_transforms_collection/transforms/" transform-eid "_child_transform.yaml")
+ (generate-transform-yaml transform-eid "Child Transform")}}
+ mock-source (test-helpers/create-mock-source :initial-files initial-files)]
+ (is (some #(str/includes? % coll-eid) (keys (get @(:files-atom mock-source) "main"))))
+ (is (some #(str/includes? % transform-eid) (keys (get @(:files-atom mock-source) "main"))))
+ (t2/update! :model/Collection coll-id {:archived true})
+ (events/publish-event! :event/collection-update
+ {:object (t2/select-one :model/Collection :id coll-id)
+ :user-id (mt/user->id :rasta)})
+ (is (= "delete" (:status (t2/select-one :model/RemoteSyncObject :model_type "Collection" :model_id coll-id))))
+ (is (= "delete" (:status (t2/select-one :model/RemoteSyncObject :model_type "Transform" :model_id transform-id))))
+ (let [result (impl/export! (source.p/snapshot mock-source) task-id "Test export")]
+ (is (= :success (:status result)))
+ (let [files-after-export (get @(:files-atom mock-source) "main")]
+ (is (not (some #(str/includes? % coll-eid) (keys files-after-export))))
+ (is (not (some #(str/includes? % transform-eid) (keys files-after-export)))))))))))))
diff --git a/enterprise/backend/test/metabase_enterprise/serialization/v2/extract_test.clj b/enterprise/backend/test/metabase_enterprise/serialization/v2/extract_test.clj
index 0e4a110fcfe..783346bde1e 100644
--- a/enterprise/backend/test/metabase_enterprise/serialization/v2/extract_test.clj
+++ b/enterprise/backend/test/metabase_enterprise/serialization/v2/extract_test.clj
@@ -2291,6 +2291,11 @@
{_field-id :id}
{:name "Some Field" :table_id table-id}
+ :model/Collection
+ {coll-id :id coll-eid :entity_id}
+ {:name "Transform Collection"
+ :namespace :transforms}
+
;; Create tags for associations
:model/TransformTag
{hourly-tag-id :id
@@ -2314,6 +2319,7 @@
{:name "Test Transform"
:description "A test transform for serialization"
:entity_id "2HzIFwJ6720JAx07UMavl"
+ :collection_id coll-id
:source {:query {:database db-id
:type "query"
:query {:source-table table-id}}
@@ -2364,8 +2370,9 @@
(is (= [hourly-tag-eid custom-tag-eid daily-tag-eid] tag-ids))
(is (= [0 1 2] positions))))
- (testing "dependencies include source table and tags"
+ (testing "dependencies include collection, source table, and tags"
(let [deps (set (serdes/dependencies ser))]
+ (is (contains? deps [{:model "Collection" :id coll-eid}]))
(is (contains? deps [{:model "Database" :id "My Database"}
{:model "Table" :id "Schemaless Table"}]))
(is (contains? deps [{:model "TransformTag" :id hourly-tag-eid}]))
diff --git a/frontend/src/metabase-types/api/remote-sync.ts b/frontend/src/metabase-types/api/remote-sync.ts
index 4b05ef7e58a..39cd955d4f5 100644
--- a/frontend/src/metabase-types/api/remote-sync.ts
+++ b/frontend/src/metabase-types/api/remote-sync.ts
@@ -12,7 +12,10 @@ export type RemoteSyncEntityModel =
| "snippet"
| "table"
| "field"
- | "segment";
+ | "segment"
+ | "transform"
+ | "transformtag"
+ | "transformjob";
export type RemoteSyncEntityStatus =
| "create"
diff --git a/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx b/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx
index e3072ac4db4..999168f0d73 100644
--- a/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx
+++ b/frontend/src/metabase/nav/components/search/SearchBar/SearchBar.styled.tsx
@@ -40,6 +40,7 @@ export const SearchInputContainer = styled.div<{
}
return css`
background-color: var(--mb-color-background-primary);
+
&:hover {
background-color: var(--mb-color-background-secondary);
}
diff --git a/src/metabase/collections/core.clj b/src/metabase/collections/core.clj
index ddd32c70e53..1e6330072b5 100644
--- a/src/metabase/collections/core.clj
+++ b/src/metabase/collections/core.clj
@@ -9,16 +9,18 @@
(p/import-vars
[metabase.collections.models.collection
- remote-synced-collection
- remote-synced-collection?
check-for-remote-sync-update
check-non-remote-synced-dependencies
check-remote-synced-dependents
+ create-library-collection!
+ descendant-ids
library-collection
library-data-collection-type
location-path
- create-library-collection!
- moving-into-remote-synced?
moving-from-remote-synced?
+ moving-into-remote-synced?
non-remote-synced-dependencies
- shared-tenant-collection?])
+ remote-synced-collection
+ remote-synced-collection?
+ shared-tenant-collection?
+ transforms-ns])
diff --git a/src/metabase/collections/models/collection.clj b/src/metabase/collections/models/collection.clj
index d16386e600c..8b2c2b190c1 100644
--- a/src/metabase/collections/models/collection.clj
+++ b/src/metabase/collections/models/collection.clj
@@ -1989,8 +1989,12 @@
[:= :collection_id id]
[:= :is_published true]
(when skip-archived [:= :archived_at nil])]})]
- {["Table" table-id] {"Collection" id}}))]
- (merge child-colls dashboards cards documents timelines tables)))
+ {["Table" table-id] {"Collection" id}}))
+ ;; Transforms don't have an archived column, so we don't filter by skip-archived
+ transforms (when config/ee-available?
+ (into {} (for [transform-id (t2/select-pks-set :model/Transform {:where [:= :collection_id id]})]
+ {["Transform" transform-id] {"Collection" id}})))]
+ (merge child-colls dashboards cards documents timelines tables transforms)))
(defmethod serdes/storage-path "Collection" [coll {:keys [collections]}]
(let [parental (get collections (:entity_id coll))]
diff --git a/src/metabase/remote_sync/core.clj b/src/metabase/remote_sync/core.clj
index c2712d7838e..823a74882e1 100644
--- a/src/metabase/remote_sync/core.clj
+++ b/src/metabase/remote_sync/core.clj
@@ -18,3 +18,12 @@
metabase-enterprise.remote-sync.core
[_table]
true)
+
+(defenterprise transforms-editable?
+ "Returns if transforms can be edited.
+
+ Returns false if remote-sync is enabled and remote-sync-type is :read-only.
+ Always true on OSS."
+ metabase-enterprise.remote-sync.core
+ []
+ true)
diff --git a/test_resources/serialization_baseline/transforms/2HzIFwJ6720JAx07UMavl_test_transform.yaml b/test_resources/serialization_baseline/collections/transforms/2HzIFwJ6720JAx07UMavl_test_transform.yaml
similarity index 100%
rename from test_resources/serialization_baseline/transforms/2HzIFwJ6720JAx07UMavl_test_transform.yaml
rename to test_resources/serialization_baseline/collections/transforms/2HzIFwJ6720JAx07UMavl_test_transform.yaml