Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Checker module (#70866)

dpsuttond4a861b3c928
diff --git a/.claude/skills/serdes-workflow/SKILL.md b/.claude/skills/serdes-workflow/SKILL.md
new file mode 100644
index 00000000000..dcc82bee984
--- /dev/null
+++ b/.claude/skills/serdes-workflow/SKILL.md
@@ -0,0 +1,76 @@
+---
+name: serdes-workflow
+description: Export content from a running Metabase instance, validate with checkers, edit YAML, and import back. Use when the user wants to export, import, or run the full serdes round-trip workflow.
+---
+
+# Serdes Workflow Skill
+
+End-to-end workflow for exporting Metabase content as YAML, validating it, editing it, and importing it back.
+
+## Prerequisites
+
+All commands require the `:ee` alias since export, import, and checkers live in enterprise code.
+
+You need `MB_DB_CONNECTION_URI` set to point at the target Metabase Postgres database when importing. The running dev instance's connection string is in the worktree environment (check the nREPL or startup output).
+
+## Export
+
+```bash
+clojure -M:run:ee export /path/to/export-dir
+```
+
+Flags:
+- `-c 123,456` - export only specific collection IDs
+- `-f` - include field values
+- `-e` - continue on error
+
+This writes YAML files in the serdes directory layout:
+```
+databases/<db-name>/<db-name>.yaml
+databases/<db-name>/schemas/<schema>/tables/<table>/<table>.yaml
+databases/<db-name>/schemas/<schema>/tables/<table>/fields/<field>.yaml
+collections/<eid>_<slug>/cards/<eid>_<slug>.yaml
+```
+
+## Validate
+
+Always run both checkers after any modification. They skip full Metabase startup (~8s each).
+
+### Structural checker (fast, schema-based)
+```bash
+clojure -M:run:ee --mode checker --checker structural --export /path/to/export-dir
+```
+Validates YAML shapes against Malli schemas. Catches typos, missing required fields, wrong types.
+
+### Cards checker (query validation)
+```bash
+clojure -M:run:ee --mode checker --checker cards --export /path/to/export-dir
+```
+Validates that card queries resolve correctly against exported metadata (tables, fields, FKs). Uses MLv2 under the hood.
+
+Additional cards checker flags:
+- `--lenient` - fabricate metadata on demand (when no database schema files exist)
+- `--manifest /path/to/manifest.yaml` - write manifest of all referenced entities (lenient mode only)
+- `--output /path/to/results.edn` - write detailed results to file
+
+## Edit YAML
+
+See the **serdes-yaml-edit** skill for guidance on editing YAML files. The critical rule: **run both checkers after every edit before proceeding**.
+
+## Import
+
+```bash
+MB_DB_CONNECTION_URI="postgres://user:pass@localhost:5432/metabase_dbname" \
+ clojure -M:run:ee import /path/to/export-dir -e
+```
+
+The `-e` flag continues on error. The `MB_DB_CONNECTION_URI` must point at the same Postgres database the running Metabase instance uses - without it, import defaults to H2 which is wrong.
+
+## Full Demo Sequence
+
+1. Export: `clojure -M:run:ee export /tmp/metabase-export`
+2. Validate baseline: run both checkers
+3. Edit YAML files (see serdes-yaml-edit skill)
+4. Re-validate: run both checkers again
+5. Import: `MB_DB_CONNECTION_URI=... clojure -M:run:ee import /tmp/metabase-export -e`
+6. Verify in browser
diff --git a/.claude/skills/serdes-yaml-edit/SKILL.md b/.claude/skills/serdes-yaml-edit/SKILL.md
new file mode 100644
index 00000000000..1d6413ecddd
--- /dev/null
+++ b/.claude/skills/serdes-yaml-edit/SKILL.md
@@ -0,0 +1,243 @@
+---
+name: serdes-yaml-edit
+description: Edit Metabase serdes YAML files (cards, dashboards, databases) with correct portable references and structural conventions. Use when modifying exported YAML content.
+---
+
+# Serdes YAML Edit Skill
+
+## Golden Rule
+
+**Run both checkers after every edit.** No exceptions. Do not batch multiple edits before validating.
+
+```bash
+clojure -M:run:ee --mode checker --checker structural --export /path/to/export-dir
+clojure -M:run:ee --mode checker --checker cards --export /path/to/export-dir
+```
+
+If either checker fails, fix the issue before making further edits.
+
+## Portable References
+
+Serdes YAML uses portable references instead of integer IDs. This is the most important concept for editing.
+
+### Database references
+String name: `"Sample Database"`
+
+### Table references
+Array of `[database, schema, table]`:
+```yaml
+table_id:
+- Sample Database
+- PUBLIC
+- ACCOUNTS
+```
+
+### Field references
+Array of `[database, schema, table, field]`:
+```yaml
+id:
+- Sample Database
+- PUBLIC
+- ACCOUNTS
+- EMAIL
+```
+
+### Field refs in queries
+```yaml
+field_ref:
+- field
+- - Sample Database
+ - PUBLIC
+ - ACCOUNTS
+ - EMAIL
+- null
+```
+The outer array is `[field, <field-path>, <options>]`. The options are usually `null`.
+
+### Card references
+Entity ID string (21 characters): `"Qk5TgsNx4ubXIUtsQmT8G"`
+
+### Source table in queries
+```yaml
+dataset_query:
+ database: Sample Database
+ query:
+ source-table:
+ - Sample Database
+ - PUBLIC
+ - ACCOUNTS
+ type: query
+```
+
+## Safe Edits (low risk)
+
+These rarely break validation:
+- `name` - card/dashboard display name
+- `description` - card/dashboard description
+- `display` - visualization type (table, bar, line, pie, etc.)
+- `visualization_settings` - chart configuration
+- `archived` - true/false
+- `collection_id` - move to a different collection (use entity_id of target collection, or null for root)
+
+## Structural Edits (must match schema)
+
+These must use valid portable refs and will be caught by the cards checker if wrong:
+- `dataset_query` - the query definition
+- `result_metadata` - column metadata (must match the query's output columns)
+- `table_id` - must reference a table that exists in the export
+- `database_id` - must reference a database that exists in the export
+
+## Editing result_metadata
+
+Each entry in `result_metadata` describes an output column. When changing a query's source table or fields, you must update result_metadata to match. Each field entry needs at minimum:
+- `name` - column name (e.g., `EMAIL`)
+- `base_type` - Metabase type (e.g., `type/Text`, `type/Integer`, `type/DateTime`)
+- `display_name` - human-readable name
+- `field_ref` - portable field reference
+- `id` - portable field path
+- `table_id` - portable table path
+- `source` - usually `fields`
+
+## Common Operations
+
+### Rename a card
+Change `name:` at the top level. Safe, no ref changes needed.
+
+### Change a card's source table
+Update all of these consistently:
+1. `table_id` - top-level
+2. `dataset_query.query.source-table` - in the query
+3. `result_metadata` - every field entry's `id`, `field_ref`, and `table_id`
+
+### Add a filter to a structured query
+```yaml
+dataset_query:
+ database: Sample Database
+ query:
+ source-table:
+ - Sample Database
+ - PUBLIC
+ - ORDERS
+ filter:
+ - ">"
+ - - field
+ - - Sample Database
+ - PUBLIC
+ - ORDERS
+ - TOTAL
+ - null
+ - 100
+ type: query
+```
+
+### Change visualization type
+```yaml
+display: bar # was: table
+```
+Valid types: `table`, `bar`, `line`, `pie`, `scalar`, `row`, `area`, `combo`, `scatter`, `funnel`, `map`, `pivot`, `progress`, `gauge`, `waterfall`
+
+## Looking Up Valid References
+
+The export directory IS the reference catalog. When you need to find the correct name for a database, table, or field, look it up directly from the export.
+
+### List valid databases
+```bash
+ls databases/
+```
+Each entry is a database name (directory for serdes format, .yaml for concise format).
+
+### List valid tables for a database
+```bash
+ls databases/<db-name>/schemas/<schema>/tables/
+```
+Example: `ls databases/Sample\ Database/schemas/PUBLIC/tables/` shows `ACCOUNTS`, `ORDERS`, `PRODUCTS`, etc.
+
+### List valid fields for a table
+```bash
+ls databases/<db-name>/schemas/<schema>/tables/<table>/fields/
+```
+Example: `ls databases/Sample\ Database/schemas/PUBLIC/tables/PRODUCTS/fields/` shows `CATEGORY.yaml`, `TITLE.yaml`, `PRICE.yaml`, etc. The filename (minus `.yaml`) is the field name.
+
+### Find card entity IDs
+Card filenames encode the entity ID: `<entity-id>_<slug>.yaml`. The entity ID is the part before the first underscore (21 characters).
+
+## Self-Healing Loop
+
+When a checker reports errors, follow this loop:
+
+1. **Read the error message** - it tells you what's wrong and often suggests the fix
+2. **Look up the correct value** from the export directory (see "Looking Up Valid References" above)
+3. **Fix the YAML**
+4. **Re-run both checkers**
+5. **Repeat until clean**
+
+Do not guess at fixes. Always look up the correct value from the export.
+
+## Understanding Checker Errors
+
+### Structural checker errors
+
+The structural checker validates YAML shape against Malli schemas. Common errors:
+
+**Missing required key with typo suggestion:**
+```
+Missing required key 'name' - found 'nameee' which may be a typo
+```
+Fix: rename the typo'd key back to the correct name. The checker tells you what it expected and what it found.
+
+**Wrong type:**
+```
+'archived' should be a boolean, got: "yes"
+```
+Fix: use `true`/`false`, not strings.
+
+**Unknown key:**
+```
+Unknown key 'foobar' in card
+```
+Fix: remove the key, or check if it's a typo of a known key.
+
+### Cards checker errors
+
+The cards checker validates that queries resolve against exported metadata. Error types:
+
+**UNRESOLVED REFERENCES:**
+```
+UNRESOLVED REFERENCES:
+ - field: Sample Database.PUBLIC.PRODUCTS.CATEGORYYY
+```
+The dotted path tells you exactly which reference failed. Look up the correct value:
+- Last segment is the field name - check `databases/.../tables/<table>/fields/`
+- Third segment is the table name - check `databases/.../schemas/<schema>/tables/`
+- First segment is the database name - check `databases/`
+
+**ERROR (query construction failed):**
+```
+ERROR: Error creating query from legacy query: Invalid output: ...
+```
+Usually means the query structure is malformed. This often follows unresolved references - fix the refs first, then re-check.
+
+**ERROR (nil name):**
+```
+ERROR: Invalid output: {:name ["should be a string, got: nil"]}
+```
+The card is missing its `name` field (likely a structural issue that also affects the cards checker).
+
+### Exit codes
+
+- Exit 0: all checks passed
+- Exit 1: one or more failures
+
+## File Naming Convention
+
+Card files: `collections/<collection-path>/cards/<entity-id>_<slug>.yaml`
+
+The filename slug should match the card name (lowercase, underscored). If you rename a card, consider renaming the file to match, though import works on entity_id not filename.
+
+## What NOT to Edit
+
+- `entity_id` - this is the identity of the object, changing it creates a new object on import
+- `serdes/meta` - internal serdes metadata, leave it alone
+- `created_at` - timestamp, no reason to change
+- `creator_id` - email reference, leave it
+- `metabase_version` - informational, leave it
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.edn
index 50d076bf85c..abca4ff91ff 100644
--- a/.clj-kondo/config/modules/config.edn
+++ b/.clj-kondo/config/modules/config.edn
@@ -1290,6 +1290,7 @@
metabase.models.interface
metabase.models.resolution
metabase.models.serialization
+ metabase.models.serialization.resolve
metabase.models.util.spec-update
metabase.models.visualization-settings}
:uses #{api
@@ -2741,6 +2742,18 @@
:model/QueryCache
:model/QueryExecution}}
+ enterprise/checker
+ {:team "Graphy"
+ ;; todo: add ability to prevent usage (transitive even) of app db
+ :api #{}
+ :uses #{lib
+ models
+ query-processor
+ sql-tools
+ util
+ enterprise/dependencies}
+ :model-imports #{:model/Database}}
+
enterprise/cloud-add-ons
{:team "Cloud"
:api #{metabase-enterprise.cloud-add-ons.api}
@@ -2836,7 +2849,8 @@
enterprise/dependencies
{:team "Graphy"
- :api #{metabase-enterprise.dependencies.api
+ :api #{metabase-enterprise.dependencies.analysis
+ metabase-enterprise.dependencies.api
metabase-enterprise.dependencies.core
metabase-enterprise.dependencies.init
metabase-enterprise.dependencies.models.dependency}
diff --git a/CLAUDE.md b/CLAUDE.md
index 1558c62ae77..480b696abc0 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -26,6 +26,11 @@ For detailed guidance on writing and reviewing code and documentation, see the s
- **[docs-write](.claude/skills/docs-write/SKILL.md)** - Documentation writing with Metabase style guide
- **[docs-review](.claude/skills/docs-review/SKILL.md)** - Documentation review checklist
+## Serialization
+
+- **[serdes-workflow](.claude/skills/serdes-workflow/SKILL.md)** - Export, validate, and import Metabase content via serdes
+- **[serdes-yaml-edit](.claude/skills/serdes-yaml-edit/SKILL.md)** - Edit exported YAML files with correct portable references
+
## Frontend
- **[analytics-events](.claude/skills/analytics-events/SKILL.md)** - Add product analytics events to track user interactions
diff --git a/enterprise/backend/src/metabase_enterprise/checker/README.md b/enterprise/backend/src/metabase_enterprise/checker/README.md
new file mode 100644
index 00000000000..7d2fd200201
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/README.md
@@ -0,0 +1,350 @@
+# Checker Module
+
+Validates serdes YAML exports without a running Metabase instance or database connection.
+
+Given two directories — database schemas and exported entities — the checker
+resolves all references, runs query validation, and reports what's broken.
+
+## Quick Start
+
+```bash
+# Dev mode
+clj -M:dev:drivers:ee -m metabase.core.bootstrap \
+ --mode checker \
+ --export /path/to/export \
+ --schema-dir /path/to/databases
+
+# Built jar
+java -jar metabase.jar --mode checker \
+ --export /path/to/export --schema-dir /path/to/databases
+```
+
+Exit code 0 when all checks pass, 1 when any fail.
+
+Example call against representations with edits intentionall introducing errors:
+
+```
+❯ clj -M:dev:drivers:ee -m metabase.core.bootstrap --mode checker --export /Users/dan/projects/work/representations/examples/v1 --schema-dir /Users/dan/projects/work/yaml-checked-files-v1/exports/sqlite-based/databases
+[lots of logging removed]
+Semantic Check Results
+=====================
+Total entities: 79
+ OK: 76
+ Errors: 0
+ Unresolved refs: 1
+ Native SQL errors: 2
+ Issues: 0
+
+Failures:
+---------
+
+=== Daily Order Summary [7XqqzlsonSq4KFyOvtrAm] ===
+ Kind: transform
+ UNRESOLVED REFERENCES:
+ NATIVE SQL ERRORS:
+ - {:type :syntax-error}
+ SQL: SELECT
+ DATE_TRUNC('day', CREATED_AT) AS order_date
+ COUT(*) AS order_count,
+ SU(TOTAL) AS total_revenue
+FROM ORDERS
+GROUP BY 1
+ Status: NATIVE SQL ERRORS
+
+=== Native Query - Card and Snippet References [HiBFSt0BNx5s5MxVDLLKB] ===
+ Card ID: 94
+ Tables: Sample Database.PUBLIC.ORDERS
+ Fields: Sample Database.PUBLIC.ORDERS.CREATED_AT
+ Source Cards: Basic Aggregations
+ NATIVE SQL ERRORS:
+ - {:type :missing-column, :name "created_attt", :source-entity-type :card, :source-entity-id "h5F2EjHsRd73Dqqh8sAtd"}
+ - {:type :missing-column, :name "cooount", :source-entity-type :card, :source-entity-id "h5F2EjHsRd73Dqqh8sAtd"}
+ Status: NATIVE SQL ERRORS
+
+=== Product Analysis Report [dOc1PrOdAnAlYsIsRpTx2] ===
+ Kind: document
+ UNRESOLVED REFERENCES:
+ - document-link: Q_jD-f-9clKLFZ2TfUG2g
+ Status: MISSING REFS
+
+❯ echo $?
+1
+```
+
+## What It Validates
+
+### Query validation (via `deps.analysis/check-entity`)
+
+- **MBQL**: bad field/table/card references, broken joins
+- **Native SQL**: missing columns, syntax errors, bad table references
+- **Transforms**: query validation plus duplicate output column detection
+- Source attribution on errors (which table or card caused the problem)
+
+### Structural checks
+
+- `collection_id`, `dashboard_id`, `document_id`, `parent_id` point to entities of the correct kind
+- Dashboard layout: card refs exist, tab refs match, grid positions in bounds
+- Document content: embedded card refs and entity links resolve
+- Transform database refs exist in the schema
+- Duplicate `entity_id`s across files
+
+### Dependency tracking
+
+Each card result includes a `:refs` map showing what it touches:
+
+- `:tables` — database tables referenced
+- `:fields` — specific fields used
+- `:source-cards` — other cards used as data sources
+- `:snippets` — native query snippets used
+- `:measures`, `:metrics`, `:segments` — referenced by entity-id
+
+These propagate transitively through source-card chains.
+
+## CLI Options
+
+```
+--export PATH Path to serdes export directory (collections/)
+--schema-dir PATH Path to database schema directory (databases/)
+--output PATH Write raw results to a file
+--errors-only Output only errors to stdout (concise format)
+--help Show help
+```
+
+`--errors-only` outputs one block per failing entity:
+
+```
+card: Number of Orders (entity_id: 8EdazRgPwfxdiltp7NCjS)
+ unresolved field: Sample Database.PUBLIC.ORDERS.STARTED_AT
+transform: Bad Transform (entity_id: 6Fdv3rO4bB5xyXusrVEGS)
+ unresolved field: zomato..zomato.deliveryyyy
+```
+
+## Running Tests
+
+```bash
+clj -X:dev:test:ee:ee-dev :module enterprise/checker
+```
+
+Tests include unit tests (in-memory sources), integration tests (YAML fixtures),
+and e2e tests against the baseline export at `test_resources/serialization_baseline`.
+
+## REPL Usage
+
+```clojure
+(require '[metabase-enterprise.checker.semantic :as checker])
+
+;; Check the baseline export
+(def results
+ (:results (checker/check "test_resources/serialization_baseline"
+ "test_resources/serialization_baseline/databases")))
+
+(checker/summarize-results results)
+;; => {:total 80, :ok 80, :errors 0, :unresolved 0, :native-errors 0, :issues 0}
+
+;; Inspect specific entities
+(select-keys results ["wpW9JO5MsHQd0-7WrR7gN"
+ "kBjQ5VXJ5z3vYSW72J6qa"
+ "dOc1PrOdAnAlYsIsRpTx2"])
+;; =>
+;; {"wpW9JO5MsHQd0-7WrR7gN"
+;; {:card-id 67,
+;; :name "Joins - Strategies and Compound Conditions",
+;; :entity-id "wpW9JO5MsHQd0-7WrR7gN",
+;; :refs {:tables ["Sample Database.PUBLIC.PRODUCTS"
+;; "Sample Database.PUBLIC.REVIEWS"
+;; "Sample Database.PUBLIC.ORDERS"
+;; "Sample Database.PUBLIC.PEOPLE"],
+;; :fields ["Sample Database.PUBLIC.ORDERS.ID"
+;; "Sample Database.PUBLIC.ORDERS.USER_ID"
+;; ...19 fields across 4 tables...]}},
+;;
+;; "kBjQ5VXJ5z3vYSW72J6qa"
+;; {:card-id 6,
+;; :name "Metric, Measure, and Segment References",
+;; :entity-id "kBjQ5VXJ5z3vYSW72J6qa",
+;; :refs {:tables ["Sample Database.PUBLIC.ORDERS"],
+;; :source-cards ["Total Revenue"],
+;; :measures ["DBap2523KuN4Lt-cYrUjF"],
+;; :metrics ["IW8kbqZVaMxdGtCM2F4U6"],
+;; :segments ["OaFXAvEzPvu9ZXW5PUMRa"]}},
+;;
+;; "dOc1PrOdAnAlYsIsRpTx2"
+;; {:name "Product Analysis Report",
+;; :entity-id "dOc1PrOdAnAlYsIsRpTx2",
+;; :kind :document}}
+
+;; Interactive iteration on a single card
+(def ctx (checker/setup "test_resources/serialization_baseline"
+ "test_resources/serialization_baseline/databases"))
+(checker/check-one ctx "HiBFSt0BNx5s5MxVDLLKB" :verbose true)
+```
+
+## Architecture
+
+```
+ SchemaSource ──┐ ┌── query validation (deps.analysis)
+ (databases, │ │
+ tables, ├─→ Store ─→ Provider ─→ MetadataProvider
+ fields) │ (IDs, (lib │
+ │ cache) adapter) └── structural checks
+ AssetsSource ──┘ (collection refs, dashboard
+ (cards, layout, document links)
+ snippets,
+ transforms,
+ segments)
+```
+
+1. **Sources** provide entity data. `SchemaSource` resolves databases/tables/fields
+ from `--schema-dir`. `AssetsSource` resolves cards/snippets/transforms/segments
+ from `--export`. Both load YAML lazily.
+2. **Store** combines both sources with a file index. Knows what entities exist,
+ loads them on demand, assigns synthetic integer IDs (lib requires ints), caches
+ everything.
+3. **Provider** adapts the store into a `MetadataProvider` that `lib/query`
+ understands. Converts raw YAML to lib metadata format, resolves portable refs
+ to integer IDs, normalizes MBQL. Owns a mutable `current-db` atom set per
+ entity check.
+4. **Validation** runs query checks via `deps.analysis/check-entity` and
+ structural checks on entity relationships.
+
+### On-demand indexing
+
+The schema directory can contain 500k+ field files. To avoid parsing them all
+at startup:
+
+- **Databases** (~40): indexed at startup by reading only the `name:` field
+ from each database YAML via fast regex (no full parse).
+- **Tables** (~22k): enumerated on demand by listing directories. Assigned
+ lightweight IDs from directory paths via `ensure-table-id!` (no YAML parse).
+ Full YAML is parsed only when table data is actually needed.
+- **Fields** (~500k): resolved on demand from `<table>/fields/` directories.
+ Never pre-indexed.
+- **Segments/measures**: indexed from the export dir only (walks table
+ directories for `segments/` and `measures/` subdirs). Not indexed in the
+ schema dir.
+
+Directory names are slugified (e.g. `analytics_data_warehouse` for
+"Analytics Data Warehouse"). A `db-name->dir` mapping built from database
+YAMLs handles the translation. Schema and table directory names match their
+real names.
+
+This brings setup time from ~64s to ~100ms.
+
+### Running without an app DB
+
+The checker runs the full QP compilation pipeline (template tag expansion,
+snippet inlining, card ref resolution) but without a running Metabase instance.
+Two things make this possible:
+
+- **`qp.i/*skip-middleware-because-app-db-access*`** is bound to `true`, which
+ skips middleware that queries the app DB for access control. Currently skipped:
+ - **Impersonation** — connection role swapping based on user permissions
+ - **Database routing** — routing queries to alternative destination databases
+
+ These are security middleware, not semantic middleware. They control *who runs
+ where*, not the shape of the query. Skipping them is safe for validation.
+
+- **`mu.fn/*enforce*`** is bound to `false`, disabling Malli output schema
+ enforcement so `lib/query` accepts valid MBQL clauses that the schemas don't
+ fully enumerate.
+
+This means native SQL validation resolves through `{{#card-ref}}` and
+`{{snippet: Name}}` template tags using real data from the store, catching
+missing columns with source attribution back to the originating card or snippet.
+
+### Namespaces
+
+- **`checker.source`** — `SchemaSource` and `AssetsSource` protocols
+- **`checker.format.serdes`** — serdes directory walking, YAML extraction, file index building, `SerdesSource` (implements both protocols)
+- **`checker.store`** — combines sources + index, ID registry, entity caches, lazy loading
+- **`checker.provider`** — adapts store to lib `MetadataProvider`, data conversion, MBQL normalization, reference resolution
+- **`checker.semantic`** — entity validation, result formatting, top-level API (`check`, `setup`, `check-one`)
+- **`checker.cli`** — CLI entrypoint and argument parsing
+
+## Performance
+
+- **Setup**: ~100ms (reads ~40 database YAMLs)
+- **Per entity**: ~1s average, cached after first access per database
+- **First entity per database**: ~5-8s (table directory listing + ID assignment)
+- **Complex native SQL**: 5-20s per entity (SQLGlot parsing bottleneck)
+- **Full run** (177 entities, 40 databases, 22k tables): ~90s
+
+### Parallelism
+
+The architecture is ready for multi-threaded entity checking:
+- The store (atom) is thread-safe
+- Sources are stateless/immutable
+- Each thread needs its own `provider` (owns mutable `current-db`)
+- Dynamic bindings (`mu.fn/*enforce*`, `qp.i/*skip-middleware*`) are per-thread
+
+To parallelize: create a provider per thread, replace the `for` in
+`check-entities` with `pmap` or a thread pool.
+
+## Roadmap
+
+### Dependency graph
+
+The checker computes forward edges per entity (what it depends on). The next
+step is building a bidirectional dependency graph using
+`metabase.graph.core/InMemoryGraph`.
+
+The `dependencies` module already has the right pieces:
+
+- **`calculation.clj`** — `upstream-deps:card`, `upstream-deps:transform`, etc.
+ extract forward edges from resolved lib queries. These just call
+ `lib/all-source-table-ids` etc., which work fine against our provider.
+- **`analysis.clj`** — `check-entity` validates individual entities. The checker
+ already uses this.
+- **`metabase.graph.core`** — generic `Graph` protocol with `InMemoryGraph`,
+ `transitive`, `find-cycle`, etc.
+
+The plan:
+
+1. Call `calculation.clj`'s `upstream-deps:*` functions against the checker's
+ resolved entities (or align our `extract-refs-from-query` output to match)
+2. Collect all edges into an adjacency map, wrap in `graph/in-memory`
+3. Invert the map for the reverse graph
+4. Return both from `checker/check`
+
+This enables "who uses this snippet?", "what breaks if I rename this table?",
+cycle detection via `graph/find-cycle`, and full impact analysis — all offline.
+
+### Decouple dependencies module from app DB
+
+`deps.analysis` and `deps.calculation` are clean — they operate on
+MetadataProviders and lib queries with no toucan2 dependency. But
+`deps.core` and `deps.models.dependency` are tightly coupled to the app DB
+(toucan2, `:model/Dependency` table, etc.).
+
+Currently the checker imports `deps.analysis` directly, avoiding the toucan2
+dependency chain. For the graph work, we'll also need `deps.calculation`
+(also clean) and `metabase.graph.core` (also clean).
+
+The goal is that the "pure analysis" side of the dependencies module
+(`analysis.clj`, `calculation.clj`) stays usable without an app DB connection,
+while the "storage" side (`models/dependency.clj`, `events.clj`) stays in leaf
+namespaces that the checker never touches. This mirrors the pattern in
+`models.serialization` where we separated the resolve protocols from the
+DB-backed implementations.
+
+### Merge native errors into bad-refs
+
+`:native-errors` (from SQL parsing) and `:bad-refs` (from MBQL analysis) are
+both query issues found through different code paths. Consider merging them
+into a single `:query-issues` key. The other result keys (`:error` for fatal
+failures, `:unresolved` for missing schema refs) are genuinely different
+categories and should stay separate.
+
+### Table-type template tag validation
+
+Native queries with `{{table_one}}`-style template tags (type: table) currently
+get placeholder substitution — column validation is skipped because the actual
+table is unknown. The template tag contains a `table-id` that could be resolved
+via the provider to substitute the real table name, enabling full column
+validation against the schema.
+
+### Schema mismatch false positives
+
+`sql-tools/validate-query` defaults unqualified tables to `default-schema`,
+which returns `"public"` for schema-less databases where tables have `nil`
+schema. Root cause is in `sql-tools`, not the checker.
diff --git a/enterprise/backend/src/metabase_enterprise/checker/cli.clj b/enterprise/backend/src/metabase_enterprise/checker/cli.clj
new file mode 100644
index 00000000000..e1fc45d62e5
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/cli.clj
@@ -0,0 +1,126 @@
+(ns metabase-enterprise.checker.cli
+ "CLI entrypoint for the checker module.
+
+ Invoked from metabase.core.bootstrap:
+ java -jar metabase.jar --mode checker --export /path/to/export --schema-dir /path/to/schemas"
+ (:require
+ [clojure.java.io :as io]
+ [clojure.string :as str]
+ [clojure.tools.cli :as cli]
+ [metabase-enterprise.checker.semantic :as checker]))
+
+(set! *warn-on-reflection* true)
+
+#_{:clj-kondo/ignore [:discouraged-var]}
+(def output!
+ "Alias for println so can suppress warning in one place"
+ println)
+
+(defn- fail!
+ "Print error message to stderr and exit with code 1."
+ [& messages]
+ (binding [*out* *err*]
+ (doseq [msg messages]
+ (output! msg)))
+ (flush)
+ (System/exit 1))
+
+(defn- validate-directory!
+ "Validate that path exists and is a directory. Exits on failure."
+ [path]
+ (let [f (io/file path)]
+ (cond
+ (not (.exists f))
+ (fail! (str "Directory does not exist: " path))
+
+ (not (.isDirectory f))
+ (fail! (str "Not a directory: " path)))))
+
+(defn- run-checker
+ "Run the semantic checker."
+ [export-dir {:keys [output errors-only schema-dir]}]
+ (when-not schema-dir
+ (fail! "Missing --schema-dir option"))
+ (validate-directory! schema-dir)
+ (let [{:keys [results]} (try
+ (checker/check export-dir schema-dir)
+ (catch clojure.lang.ExceptionInfo e
+ (fail! (.getMessage e))))
+ summary (checker/summarize-results results)
+ failures (filter #(not= :ok (checker/result-status (second %))) results)]
+ (if errors-only
+ ;; Errors-only mode: just errors to stdout, nothing else
+ (doseq [entry (sort-by (comp :name second) failures)
+ :let [error-str (checker/format-error entry)]
+ :when error-str]
+ #_{:clj-kondo/ignore [:discouraged-var]}
+ (output! error-str))
+ ;; Normal mode: full output
+ (do
+ ;; Write to output file if specified
+ (when output
+ (spit output (pr-str results))
+ (output! "Results written to:" output))
+ ;; Print summary
+ (output! "Semantic Check Results")
+ (output! "=====================")
+ (output! "Total entities:" (:total summary))
+ (output! " OK:" (:ok summary))
+ (output! " Errors:" (:errors summary))
+ (output! " Unresolved refs:" (:unresolved summary))
+ (output! " Native SQL errors:" (:native-errors summary))
+ (output! " Issues:" (:issues summary))
+ ;; Print failures
+ (when (seq failures)
+ (output! "\nFailures:")
+ (output! "---------")
+ (doseq [entry (sort-by (comp :name second) failures)]
+ (output!)
+ (output! (checker/format-result entry))))))
+ ;; Exit with appropriate code
+ (flush)
+ (System/exit (if (zero? (+ (:errors summary) (:unresolved summary)
+ (:native-errors summary) (:issues summary)))
+ 0
+ 1))))
+
+(def ^:private cli-spec
+ "CLI options for checker mode."
+ [[nil "--mode MODE" "Mode (handled by bootstrap, included here for completeness)"]
+ [nil "--export PATH" "Path to serdes export directory"]
+ [nil "--schema-dir PATH" "Path to database schema directory"]
+ [nil "--output PATH" "Path to output file for results"]
+ [nil "--errors-only" "Output only errors to stdout (concise format for LLM consumption)"]
+ ["-h" "--help" "Show this help"]])
+
+(defn- usage [summary]
+ (str "Usage: java -jar metabase.jar --mode checker [options]\n\n"
+ "Options:\n"
+ summary))
+
+(defn entrypoint
+ "Main entrypoint for checker mode. Receives raw args. Intended to enter from
+ metabase.core.bootstrap/run-standalone-mode so that it can skip loading everything that
+ metabase.core.core/entrypoint loads. This entrypoint owns the process and will call System/exit."
+ [args]
+ (let [{:keys [options errors summary]} (cli/parse-opts args cli-spec)
+ {:keys [export help]} options]
+ (cond
+ help
+ (do (output! (usage summary))
+ (System/exit 0))
+
+ errors
+ (fail! (str "Error parsing arguments:\n" (str/join "\n" errors))
+ ""
+ (usage summary))
+
+ (not export)
+ (fail! "Missing --export option"
+ ""
+ (usage summary))
+
+ :else
+ (do
+ (validate-directory! export)
+ (run-checker export options)))))
diff --git a/enterprise/backend/src/metabase_enterprise/checker/format/serdes.clj b/enterprise/backend/src/metabase_enterprise/checker/format/serdes.clj
new file mode 100644
index 00000000000..accb84b0378
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/format/serdes.clj
@@ -0,0 +1,564 @@
+(ns metabase-enterprise.checker.format.serdes
+ "Serdes export format — directory structure, path parsing, entity loading.
+
+ Directory structure (on disk, names may be slugified):
+ databases/<db-dir>/<db-dir>.yaml
+ databases/<db-dir>/schemas/<schema-dir>/tables/<table-dir>/<table-dir>.yaml
+ databases/<db-dir>/schemas/<schema-dir>/tables/<table-dir>/fields/<field>.yaml
+ databases/<db-dir>/tables/<table-dir>/<table-dir>.yaml # schema-less
+ databases/<db-dir>/tables/<table-dir>/fields/<field>.yaml # schema-less
+ collections/<entity-id>_<slug>/<entity-id>_<slug>.yaml
+ collections/.../cards/<entity-id>_<slug>.yaml
+ collections/.../dashboards/<entity-id>_<slug>.yaml
+
+ Directory names may differ from entity names (e.g. `analytics_data_warehouse`
+ for \"Analytics Data Warehouse\", or `orders` for \"ORDERS\" in H2).
+
+ ## Lazy resolution cache
+
+ The `SerdesSource` maintains a single lazily-populated cache atom that maps
+ real entity names (from YAML) to file locations:
+
+ {db-name ; real name, e.g. \"Analytics Data Warehouse\"
+ {schema ; real name or nil for schema-less
+ ::not-indexed ; schema exists but tables not yet walked
+ | {table-name ; real name from YAML
+ {:table-file \"path/to/table.yaml\" ; path string to the table YAML
+ :fields ::not-indexed ; fields not yet walked for this table
+ | {field-name \"path\" ...}}}}} ; real name → field YAML path
+
+ Two levels of laziness:
+ 1. Schema → tables: realized when any table in that schema is resolved
+ (via `resolve-table`, `resolve-field`, `fields-for-table`) or when
+ `tables-for-database` is called. Walks the schema's `tables/` directory
+ and reads each table's `name:` from YAML via fast regex.
+ 2. Table → fields: realized when `resolve-field` or `fields-for-table` is
+ called for that specific table. Walks the table's `fields/` directory
+ and reads each field's `name:` via fast regex. Other tables in the same
+ schema are unaffected.
+
+ The `db-name->dir` map (built at startup from ~40 database YAMLs) handles the
+ database name → directory name translation. Schema and table name → directory
+ name translation is handled by case-insensitive directory matching during
+ schema indexing."
+ (:require
+ [clojure.java.io :as io]
+ [clojure.string :as str]
+ [metabase-enterprise.checker.source :as source]
+ [metabase.util.yaml :as yaml])
+ (:import
+ (java.io File)
+ (java.util Locale)))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; YAML Loading
+;;; ===========================================================================
+
+(defn load-yaml
+ "Load and parse a YAML file."
+ [path]
+ (yaml/parse-string (slurp path)))
+
+(defn quick-extract
+ "Extract a field from YAML using regex (fast) with full parse fallback."
+ [file-path field-name pattern]
+ (try
+ (let [content (slurp file-path)]
+ (if-let [[_ value] (re-find pattern content)]
+ (str/trim value)
+ (get (yaml/parse-string content) (keyword field-name))))
+ (catch Exception _ nil)))
+
+(defn extract-entity-id
+ "Extract the entity id from a yaml file without parsing it. Looking for a top level entity_id at start of line."
+ [file-path]
+ (quick-extract file-path "entity_id" #"(?m)^entity_id:\s*(\S+)"))
+
+(defn extract-model
+ "Extract the serdes model from a YAML file (e.g., \"Collection\", \"Card\", \"Dashboard\").
+ Looks for the model field inside serdes/meta to avoid matching embedded refs."
+ [file-path]
+ (try
+ (let [content (slurp file-path)]
+ (when-let [[_ model] (re-find #"(?m)^serdes/meta:\s*\n\s*- id:.*\n\s+(?:label:.*\n\s+)?model:\s*(\S+)" content)]
+ (str/trim model)))
+ (catch Exception _ nil)))
+
+(defn- read-yaml-name
+ "Read the `name:` field from a YAML file using regex (fast).
+ Falls back to `fallback` if the file doesn't exist or can't be read."
+ [^File yaml-file ^String fallback]
+ (if (.exists yaml-file)
+ (or (quick-extract (.getPath yaml-file) "name" #"(?m)^name:\s*(.+)")
+ fallback)
+ fallback))
+
+(defn- db-dir
+ "Resolve a real database name to its directory on disk."
+ ^File [^File databases-dir db-name->dir db-name]
+ (let [dir-name (get db-name->dir db-name db-name)]
+ (io/file databases-dir dir-name)))
+
+(defn- list-table-dirs
+ "Return a seq of table directory Files for a database.
+ Walks schemas/<schema>/tables/<table>/ and tables/<table>/ directories."
+ [^File databases-dir db-name->dir-map db-name]
+ (let [^File db (db-dir databases-dir db-name->dir-map db-name)
+ result (volatile! [])]
+ (let [^File schemas-dir (io/file db "schemas")]
+ (when (.isDirectory schemas-dir)
+ (doseq [^File schema-d (.listFiles schemas-dir)
+ :when (.isDirectory schema-d)]
+ (let [^File tables-dir (io/file schema-d "tables")]
+ (when (.isDirectory tables-dir)
+ (doseq [^File td (.listFiles tables-dir)
+ :when (.isDirectory td)]
+ (vswap! result conj td)))))))
+ (let [^File tables-dir (io/file db "tables")]
+ (when (.isDirectory tables-dir)
+ (doseq [^File td (.listFiles tables-dir)
+ :when (.isDirectory td)]
+ (vswap! result conj td))))
+ @result))
+
+(defn- index-segments-and-measures
+ "Walk a table directory's segments/ and measures/ subdirectories.
+ Reads entity_id from each YAML (only a handful exist).
+ Returns a seq of {:kind :ref :file} entries."
+ [^File table-dir]
+ (into []
+ (mapcat
+ (fn [[subdir kind]]
+ (let [^File d (io/file table-dir subdir)]
+ (when (.isDirectory d)
+ (keep (fn [^File f]
+ (when (and (.isFile f) (str/ends-with? (.getName f) ".yaml"))
+ (when-let [eid (extract-entity-id (.getPath f))]
+ {:kind kind :ref eid :file (.getPath f)})))
+ (.listFiles d))))))
+ [["segments" :segment] ["measures" :measure]]))
+
+(defn- index-schema-dir
+ "Walk a databases directory and index databases, segments, and measures.
+ Reads ~40 database YAMLs for real names, plus a handful of segment/measure YAMLs
+ for entity_ids. Tables and fields are resolved on demand from the filesystem.
+
+ When `include-segments-measures?` is true, also walks table directories
+ to find segment and measure YAMLs (for export dirs). Set to false for
+ schema-only dirs to avoid walking thousands of table directories.
+
+ Returns {:entries [...]
+ :db-name->dir {real-db-name dir-name}}."
+ [schema-dir & {:keys [include-segments-measures?]}]
+ (let [schema-dir (io/file schema-dir)
+ db-name->dir (volatile! {})
+ entries (volatile! [])]
+ (doseq [^File db-dir (.listFiles schema-dir)
+ :when (.isDirectory db-dir)]
+ (let [dir-name (.getName db-dir)
+ db-yaml (io/file db-dir (str dir-name ".yaml"))
+ db-name (read-yaml-name db-yaml dir-name)]
+ (vswap! db-name->dir assoc db-name dir-name)
+ (vswap! entries conj {:kind :database :ref db-name
+ :file (if (.exists db-yaml) (.getPath db-yaml) (.getPath db-dir))})
+ ;; Walk table directories for segments and measures (export dirs only)
+ (when include-segments-measures?
+ (doseq [^File table-dir (list-table-dirs schema-dir @db-name->dir db-name)]
+ (vswap! entries into (index-segments-and-measures table-dir))))))
+ {:entries @entries
+ :db-name->dir @db-name->dir}))
+
+;;; ===========================================================================
+;;; File Walking - Build index of all entities
+;;; ===========================================================================
+
+(def ^:private model->kind
+ "Map serdes model names to index kinds."
+ {"Card" :card
+ "Dashboard" :dashboard
+ "Collection" :collection
+ "Document" :document
+ "Measure" :measure
+ "Segment" :segment
+ "NativeQuerySnippet" :snippet
+ "Transform" :transform})
+
+(defn- index-collections-tree
+ "Walk the collections/ directory and index all entities by their serdes model.
+ Returns entries for cards, dashboards, and collections."
+ [export-dir]
+ (let [collections-dir (io/file export-dir "collections")]
+ (when (.isDirectory collections-dir)
+ (for [^File file (file-seq collections-dir)
+ :when (.isFile file)
+ :let [path (.getPath file)]
+ :when (str/ends-with? (.getName file) ".yaml")
+ :let [model (extract-model path)
+ kind (get model->kind model)]
+ :when kind
+ :let [entity-id (extract-entity-id path)]
+ :when entity-id]
+ {:kind kind :ref entity-id :file path}))))
+
+(defn build-file-index
+ "Build index of all entity files in an export directory.
+ Returns `{:index {kind {ref file-path}} :db-name->dir {real-name dir-name}}`
+ where index also has `:duplicates` for refs appearing in multiple files."
+ [export-dir]
+ (let [databases-dir (io/file export-dir "databases")
+ schema-result (when (.isDirectory databases-dir)
+ (index-schema-dir databases-dir :include-segments-measures? true))
+ entries (concat
+ (:entries schema-result)
+ (index-collections-tree export-dir))
+ ;; Group by [kind ref] to find duplicates
+ by-key (reduce (fn [m {:keys [kind ref file]}]
+ (update m [kind ref] (fnil conj []) file))
+ {}
+ entries)
+ ;; Build the index (first file wins) and collect duplicates
+ index (reduce-kv (fn [idx [kind ref] files]
+ (assoc-in idx [kind ref] (first files)))
+ {}
+ by-key)
+ dupes (into []
+ (keep (fn [[[kind ref] files]]
+ (when (> (count files) 1)
+ {:kind kind :ref ref :files files})))
+ by-key)]
+ {:index (cond-> index
+ (seq dupes) (assoc :duplicates dupes))
+ :db-name->dir (:db-name->dir schema-result)}))
+
+(defn build-database-dir-index
+ "Build index of databases from a schema directory.
+ Tables and fields are resolved on demand.
+
+ Returns `{:index {:database {name file-path}} :db-name->dir {real-name dir-name}}`."
+ [databases-dir]
+ (let [{:keys [entries db-name->dir]} (index-schema-dir databases-dir)]
+ {:index (reduce (fn [idx {:keys [kind ref file]}]
+ (assoc-in idx [kind ref] file))
+ {}
+ entries)
+ :db-name->dir db-name->dir}))
+
+(defn index-stats
+ "Get statistics about a file index.
+ Note: tables and fields are resolved on demand and won't appear in the index
+ for schema-dir sources. They will be 0 for `build-database-dir-index` results."
+ [index]
+ {:databases (count (:database index))
+ :tables (count (:table index))
+ :fields (count (:field index))
+ :measures (count (:measure index))
+ :segments (count (:segment index))
+ :cards (count (:card index))
+ :dashboards (count (:dashboard index))
+ :collections (count (:collection index))
+ :database-names (vec (keys (:database index)))})
+
+;;; ===========================================================================
+;;; MetadataSource Implementation
+;;;
+;;; Returns raw YAML data - the checker handles conversion to lib format.
+;;; This keeps format knowledge here, lib knowledge in checker.
+;;; ===========================================================================
+
+(defn- table-real-name
+ "Read the real table name from its YAML file via regex. Falls back to dir name."
+ [^File table-dir]
+ (let [dir-name (.getName table-dir)
+ ^File yaml (io/file table-dir (str dir-name ".yaml"))]
+ (read-yaml-name yaml dir-name)))
+
+(defn- field-real-name
+ "Read the real field name from its YAML file via regex. Falls back to filename without extension."
+ [^File field-yaml]
+ (let [fname (.getName field-yaml)
+ fallback (str/replace fname #"\.yaml$" "")]
+ (read-yaml-name field-yaml fallback)))
+
+(defn- read-yaml-schema
+ "Read the `schema:` field from a YAML file using regex (fast).
+ Returns nil if not found or if the value is `null`."
+ [^File yaml-file]
+ (when (.exists yaml-file)
+ (let [v (quick-extract (.getPath yaml-file) "schema" #"(?m)^schema:\s*(\S+)")]
+ (when (and v (not= v "null"))
+ v))))
+
+(defn- index-schema-tables
+ "Walk a single schema's tables directory and return:
+ {:real-schema-name string-or-nil ; the real schema name from YAML (may differ from dir name)
+ :tables {real-table-name {:table-file \"path\", :fields ::not-indexed} ...}}
+ Reads real table names from YAMLs via fast regex.
+ Fields are NOT indexed here — they are indexed lazily per-table."
+ [^File tables-dir]
+ (when (.isDirectory tables-dir)
+ (let [real-schema (volatile! nil)
+ tables (into {}
+ (keep (fn [^File td]
+ (when (.isDirectory td)
+ (let [table-name (table-real-name td)
+ dir-name (.getName td)
+ ^File yaml (io/file td (str dir-name ".yaml"))]
+ (when (.exists yaml)
+ (when-not @real-schema
+ (vreset! real-schema (read-yaml-schema yaml)))
+ [table-name {:table-file (.getPath yaml) :fields ::not-indexed}])))))
+ (.listFiles tables-dir))]
+ {:real-schema-name @real-schema
+ :tables tables})))
+
+(defn- index-table-fields
+ "Walk a table's fields/ directory and return a map of:
+ {real-field-name File ...}
+ Reads real field names from YAMLs via fast regex.
+ Skips FieldValues and FieldUserSettings files."
+ [table-file-path]
+ (let [^File table-dir (.getParentFile (io/file table-file-path))
+ ^File fields-dir (io/file table-dir "fields")]
+ (if (.isDirectory fields-dir)
+ (into {}
+ (keep (fn [^File f]
+ (let [fname (.getName f)]
+ (when (and (.isFile f)
+ (str/ends-with? fname ".yaml")
+ (not (str/includes? fname "___")))
+ [(field-real-name f) (.getPath f)]))))
+ (.listFiles fields-dir))
+ {})))
+
+(defn- resolve-schema-name
+ "Resolve a real schema name to the key used in the model.
+
+ Schema names are the one entity type with no canonical YAML file — the directory
+ name is the filesystem key, and the real name comes from the `schema:` field in
+ table YAMLs. These may differ (e.g. dir=`public`, real=`PUBLIC` for H2).
+
+ Strategy:
+ 1. Exact match in the model's :schemas
+ 2. Case-insensitive match (handles H2-style uppercasing)
+ 3. nil for schema-less databases
+
+ To change schema resolution in the future, modify this function."
+ [model db-name schema]
+ (let [schemas (:schemas (get model db-name))
+ lower (fn [s] (.toLowerCase (str s) Locale/US))]
+ (cond
+ (contains? schemas schema) schema
+ schema (let [lowered (lower schema)]
+ (some (fn [k] (when (and (string? k) (= (lower k) lowered)) k))
+ (keys schemas)))
+ :else (when (contains? schemas nil) nil))))
+
+(defn- ensure-schema-indexed!
+ "Ensure a schema's tables are indexed in the model. No-op if already done.
+
+ Resolves the requested schema name to a model key (possibly case-insensitive),
+ walks the table directories, reads real names from YAMLs, and stores the results.
+ If the real schema name (from the table YAML `schema:` field) differs from the
+ directory name, the model entry is re-keyed under the real name."
+ [^File databases-dir db-name->dir db-name schema model]
+ (let [model-key (resolve-schema-name @model db-name schema)]
+ (when (= ::not-indexed (get-in @model [db-name :schemas model-key]))
+ (let [^File db (db-dir databases-dir db-name->dir db-name)
+ ^File tables-dir (if model-key
+ (io/file db "schemas" model-key "tables")
+ (io/file db "tables"))
+ {:keys [real-schema-name tables]} (index-schema-tables tables-dir)
+ tables (or tables {})
+ ;; The real schema name (from YAML) is the canonical key.
+ ;; Falls back to model-key (dir name) or the requested name.
+ final-key (or real-schema-name model-key schema)]
+ (swap! model (fn [m]
+ (cond-> m
+ ;; Remove old key if re-keying
+ (and model-key (not= model-key final-key))
+ (update-in [db-name :schemas] dissoc model-key)
+ ;; Store under canonical name
+ true
+ (assoc-in [db-name :schemas final-key] tables))))))))
+
+(defn- ensure-fields-indexed!
+ "Ensure a table's fields are indexed in the model. No-op if already done."
+ [^File databases-dir db-name->dir db-name schema table-name model]
+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema model)
+ (when (= ::not-indexed (get-in @model [db-name :schemas schema table-name :fields]))
+ (when-let [table-file (get-in @model [db-name :schemas schema table-name :table-file])]
+ (let [field-map (index-table-fields table-file)]
+ (swap! model assoc-in [db-name :schemas schema table-name :fields] field-map)))))
+
+(defn- list-schemas
+ "List schema directory names for a database. Returns seq of strings, or [nil] for schema-less."
+ [^File databases-dir db-name->dir db-name]
+ (let [^File db (db-dir databases-dir db-name->dir db-name)
+ ^File schemas-dir (io/file db "schemas")
+ ^File tables-dir (io/file db "tables")]
+ (cond-> []
+ (.isDirectory schemas-dir)
+ (into (map #(.getName ^File %))
+ (filter #(.isDirectory ^File %) (.listFiles schemas-dir)))
+ (.isDirectory tables-dir)
+ (conj nil))))
+
+(defn- init-schema-model
+ "Build the initial schema model with db-file paths and ::not-indexed for each schema.
+
+ Schema model shape:
+ {db-name {:db-file \"path/to/db.yaml\"
+ :schemas {schema ::not-indexed
+ | {table {:table-file \"path\" :fields ::not-indexed | {field \"path\"}}}}}}
+
+ Two levels of laziness:
+ 1. Schema → tables: populated when the schema is first accessed
+ 2. Table → fields: populated when that table's fields are first accessed"
+ [^File databases-dir db-name->dir db-file-index]
+ (atom
+ (into {}
+ (for [[db-name _dir-name] db-name->dir]
+ [db-name {:db-file (get db-file-index db-name)
+ :schemas (into {}
+ (for [schema (list-schemas databases-dir db-name->dir db-name)]
+ [schema ::not-indexed]))}]))))
+
+(deftype SerdesSource [databases-dir assets-index db-name->dir
+ ;; See init-schema-model for shape documentation
+ schema-model]
+ source/SchemaSource
+ (resolve-database [_ db-name]
+ (when-let [file (get-in @schema-model [db-name :db-file])]
+ (load-yaml file)))
+
+ (resolve-table [_ [db schema table-name :as _table-path]]
+ (ensure-schema-indexed! databases-dir db-name->dir db schema schema-model)
+ (when-let [path (get-in @schema-model [db :schemas schema table-name :table-file])]
+ (load-yaml path)))
+
+ (resolve-field [_ [db schema table-name field :as _field-path]]
+ (ensure-fields-indexed! databases-dir db-name->dir db schema table-name schema-model)
+ (when-let [path (get-in @schema-model [db :schemas schema table-name :fields field])]
+ (load-yaml path)))
+
+ (fields-for-table [_ [db schema table-name :as _table-path]]
+ (ensure-fields-indexed! databases-dir db-name->dir db schema table-name schema-model)
+ (when-let [entry (get-in @schema-model [db :schemas schema table-name])]
+ (let [fields (:fields entry)]
+ (when (map? fields)
+ (into #{} (map (fn [field-name] [db schema table-name field-name])) (keys fields))))))
+
+ (all-field-paths [_]
+ (doseq [[db-name {:keys [schemas]}] @schema-model
+ [schema tables] schemas
+ :when (= tables ::not-indexed)]
+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema schema-model))
+ (doseq [[db-name {:keys [schemas]}] @schema-model
+ [schema tables] schemas
+ :when (map? tables)
+ [table-name entry] tables
+ :when (= (:fields entry) ::not-indexed)]
+ (ensure-fields-indexed! databases-dir db-name->dir db-name schema table-name schema-model))
+ (into #{}
+ (for [[db {:keys [schemas]}] @schema-model
+ [schema tables] schemas
+ :when (map? tables)
+ [table-name {:keys [fields]}] tables
+ :when (map? fields)
+ field-name (keys fields)]
+ [db schema table-name field-name])))
+
+ (all-database-names [_]
+ (keys @schema-model))
+
+ (all-table-paths [_]
+ (doseq [[db-name {:keys [schemas]}] @schema-model
+ [schema _] schemas]
+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema schema-model))
+ (into []
+ (for [[db {:keys [schemas]}] @schema-model
+ [schema tables] schemas
+ :when (map? tables)
+ table-name (keys tables)]
+ [db schema table-name])))
+
+ (tables-for-database [_ db-name]
+ (doseq [[schema _] (:schemas (get @schema-model db-name))]
+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema schema-model))
+ (into []
+ (for [[schema tables] (:schemas (get @schema-model db-name))
+ :when (map? tables)
+ table-name (keys tables)]
+ [db-name schema table-name])))
+
+ source/AssetsSource
+ (resolve-card [_ entity-id]
+ (when-let [file (get-in assets-index [:card entity-id])]
+ (load-yaml file)))
+
+ (resolve-snippet [_ entity-id]
+ (when-let [file (get-in assets-index [:snippet entity-id])]
+ (load-yaml file)))
+
+ (resolve-transform [_ entity-id]
+ (when-let [file (get-in assets-index [:transform entity-id])]
+ (load-yaml file)))
+
+ (resolve-segment [_ entity-id]
+ (when-let [file (get-in assets-index [:segment entity-id])]
+ (load-yaml file)))
+
+ (resolve-dashboard [_ entity-id]
+ (when-let [file (get-in assets-index [:dashboard entity-id])]
+ (load-yaml file)))
+
+ (resolve-collection [_ entity-id]
+ (when-let [file (get-in assets-index [:collection entity-id])]
+ (load-yaml file)))
+
+ (resolve-document [_ entity-id]
+ (when-let [file (get-in assets-index [:document entity-id])]
+ (load-yaml file)))
+
+ (resolve-measure [_ entity-id]
+ (when-let [file (get-in assets-index [:measure entity-id])]
+ (load-yaml file))))
+
+(defn make-source
+ "Create a MetadataSource for a serdes export directory.
+ The databases-dir for field resolution is export-dir/databases."
+ [export-dir]
+ (let [databases-dir (io/file export-dir "databases")
+ {:keys [index db-name->dir]} (build-file-index export-dir)
+ db-files (:database index)
+ assets-index (dissoc index :database)]
+ (->SerdesSource databases-dir assets-index db-name->dir
+ (init-schema-model databases-dir db-name->dir db-files))))
+
+(defn make-database-source
+ "Create a MetadataSource for a directory that IS the databases directory.
+ The directory should contain database subdirectories directly (e.g., `Sample Database/`).
+ This source only resolves databases, tables, and fields — not cards."
+ [databases-dir]
+ (let [{:keys [index db-name->dir]} (build-database-dir-index databases-dir)
+ databases-dir (io/file databases-dir)
+ db-files (:database index)]
+ (->SerdesSource databases-dir {} db-name->dir
+ (init-schema-model databases-dir db-name->dir db-files))))
+
+(defn source-index
+ "Get the assets index from a SerdesSource. Returns a flat map of
+ {[kind ref] file-path} for cards, dashboards, segments, etc."
+ [^SerdesSource source]
+ (.-assets-index source))
+
+;;; ===========================================================================
+;;; Enumeration
+;;; ===========================================================================
+
+(defn all-card-ids
+ "Get all card entity-ids from source."
+ [^SerdesSource source]
+ (keys (:card (.-assets-index source))))
diff --git a/enterprise/backend/src/metabase_enterprise/checker/provider.clj b/enterprise/backend/src/metabase_enterprise/checker/provider.clj
new file mode 100644
index 00000000000..c28a94b6a50
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/provider.clj
@@ -0,0 +1,465 @@
+(ns metabase-enterprise.checker.provider
+ "Adapts a checker store into a lib MetadataProvider.
+
+ The store (checker.store) knows what entities exist and can load them.
+ This namespace converts that raw data into the lib metadata format that
+ `lib/query` and `deps.analysis/check-entity` expect.
+
+ Responsibilities:
+ - Convert raw YAML entity maps → lib metadata maps (keywordize types,
+ resolve FK targets, normalize MBQL operators, resolve portable refs)
+ - Implement `MetadataProvider` by delegating to the store
+ - Assign sentinel IDs for unresolved refs so queries can still be
+ constructed and `lib/find-bad-refs` reports issues with context
+
+ Public API:
+ - `(make-provider store)` — wrap a store in a MetadataProvider
+ - `(card-metadata store data)` — convert a card, collecting resolution failures
+ - `(transform-metadata store data)` — convert a transform, collecting resolution failures"
+ (:require
+ [metabase-enterprise.checker.store :as store]
+ [metabase.lib.core :as lib]
+ [metabase.lib.metadata.protocols :as lib.metadata.protocols]
+ [metabase.lib.schema.metadata :as lib.schema.metadata]
+ [metabase.models.serialization.resolve :as resolve]
+ [potemkin.types :as p.types]))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; Settable database protocol
+;;; ===========================================================================
+
+(p.types/defprotocol+ ISettableDatabase
+ "Protocol for providers that can switch which database they report."
+ (set-database! [this db-name]
+ "Set the database name (string) this provider should return from `(database)`.")
+ (clear-database! [this]
+ "Clear the current database. `(database)` will return nil until `set-database!` is called again."))
+
+;;; ===========================================================================
+;;; Reference resolution
+;;;
+;;; Resolution functions take the store and a `!failures` atom (or nil).
+;;; When a ref can't be resolved, a failure map is conj'd onto the atom.
+;;; ===========================================================================
+
+(defn- resolve-ref
+ "Try to resolve `ref` of `kind`. Returns integer ID or nil.
+ On failure, appends failure-info to `!failures`.
+ When `sentinel?` is true, assigns a sentinel ID for unresolved refs."
+ [store kind ref !failures failure-info & {:keys [sentinel?]}]
+ (when ref
+ (or (store/ref->id store kind ref)
+ (let [load-fn (case kind
+ :database store/load-database!
+ :table store/load-table!
+ :field store/load-field!
+ :card store/load-card!
+ :snippet store/load-snippet!
+ :transform store/load-transform!
+ :segment store/load-segment!
+ :dashboard store/load-dashboard!
+ :collection store/load-collection!
+ :document store/load-document!
+ :measure store/load-measure!)
+ found? (some? (load-fn store ref))]
+ (if found?
+ (store/get-or-assign! store kind ref)
+ (do (when !failures (swap! !failures conj failure-info))
+ (when sentinel?
+ (store/get-or-assign! store kind ref))))))))
+
+(defn- resolve-field-path
+ ([store !failures field-path]
+ (resolve-ref store :field field-path !failures {:type :field :path field-path}))
+ ([store !failures field-path sentinel?]
+ (resolve-ref store :field field-path !failures {:type :field :path field-path} :sentinel? sentinel?)))
+
+(defn- resolve-table-path
+ ([store !failures table-path]
+ (resolve-ref store :table table-path !failures {:type :table :path table-path}))
+ ([store !failures table-path sentinel?]
+ (resolve-ref store :table table-path !failures {:type :table :path table-path} :sentinel? sentinel?)))
+
+(defn- resolve-db-name [store !failures db-name]
+ (resolve-ref store :database db-name !failures {:type :database :name db-name}))
+
+(defn- resolve-card-entity-id [store !failures entity-id]
+ (resolve-ref store :card entity-id !failures {:type :card :entity-id entity-id}))
+
+;;; ---------------------------------------------------------------------------
+;;; SerdesImportResolver — bridges store-based resolution into import-mbql
+;;; ---------------------------------------------------------------------------
+
+(defn- make-import-resolver
+ [store !failures & {:keys [sentinel?]}]
+ (reify resolve/SerdesImportResolver
+ (import-fk [_ eid model]
+ #_{:clj-kondo/ignore [:case-symbol-test]}
+ (case model
+ Card (resolve-card-entity-id store !failures eid)
+ Segment (resolve-ref store :segment eid !failures {:type :segment :entity-id eid})
+ Measure (resolve-ref store :measure eid !failures {:type :measure :entity-id eid})
+ NativeQuerySnippet (resolve-ref store :snippet eid !failures {:type :snippet :entity-id eid})
+ (do (when !failures
+ (swap! !failures conj {:type :unknown :model model :entity-id eid}))
+ nil)))
+ (import-fk-keyed [_ portable model field]
+ (case [model field]
+ [:model/Database :name] (resolve-db-name store !failures portable)
+ (do (when !failures
+ (swap! !failures conj {:type :keyed-lookup :model model :field field :value portable}))
+ nil)))
+ (import-user [_ _email] nil)
+ (import-table-fk [_ path] (resolve-table-path store !failures path sentinel?))
+ (import-field-fk [_ path] (resolve-field-path store !failures path sentinel?))))
+
+;;; ===========================================================================
+;;; Data conversion — raw YAML entity maps → lib metadata maps
+;;; ===========================================================================
+
+(defn- ->database-metadata [data]
+ {:lib/type :metadata/database
+ :id (:id data)
+ :name (:name data)
+ :engine (keyword (:engine data))
+ :dbms-version (:dbms_version data)
+ :features #{:foreign-keys :nested-queries :expressions :native-parameters
+ :basic-aggregations :standard-deviation-aggregations
+ :expression-aggregations :left-join :right-join :inner-join :full-join}
+ :settings (:settings data)})
+
+(defn- ->table-metadata [data]
+ {:lib/type :metadata/table
+ :id (:id data)
+ :name (:name data)
+ :display-name (:display_name data)
+ :schema (:schema data)
+ :db-id (:db_id data)
+ :active (if (contains? data :active) (:active data) true)
+ :visibility-type (some-> (:visibility_type data) keyword)})
+
+(defn- ->field-metadata
+ [resolver data]
+ (cond-> {:lib/type :metadata/column
+ :id (:id data)
+ :table-id (:table_id data)
+ :name (:name data)
+ :display-name (:display_name data)
+ :base-type (keyword (:base_type data))
+ :effective-type (or (some-> (:effective_type data) keyword)
+ (keyword (:base_type data)))
+ :semantic-type (some-> (:semantic_type data) keyword)
+ :database-type (:database_type data)
+ :active (if (contains? data :active) (:active data) true)
+ :visibility-type (some-> (:visibility_type data) keyword)
+ :position (:position data)}
+ (and resolver (vector? (:fk_target_field_id data)))
+ (as-> m (if-let [fk-id (resolve/import-field-fk resolver (:fk_target_field_id data))]
+ (assoc m :fk-target-field-id fk-id)
+ m))))
+
+(defn- convert-result-metadata-column
+ "Resolve portable refs in a result_metadata column map.
+ Portable refs appear as path vectors in :id, :table_id, :fk_target_field_id,
+ and as MBQL field refs in :field_ref."
+ [resolver col]
+ (let [resolve-or-drop (fn [col k resolve-fn]
+ (let [v (get col k)]
+ (if-not (vector? v)
+ col
+ (if-let [id (resolve-fn resolver v)]
+ (assoc col k id)
+ (dissoc col k)))))]
+ (-> col
+ (resolve-or-drop :id resolve/import-field-fk)
+ (resolve-or-drop :table_id resolve/import-table-fk)
+ (resolve-or-drop :fk_target_field_id resolve/import-field-fk)
+ (cond-> (:field_ref col)
+ (update :field_ref (fn [ref]
+ (let [resolved (resolve/import-mbql resolver ref)]
+ (if (and (vector? resolved) (= :field (first resolved)) (nil? (second resolved)))
+ ::drop
+ resolved)))))
+ (as-> c (if (= ::drop (:field_ref c)) (dissoc c :field_ref) c)))))
+
+(defn- convert-dataset-query
+ "Convert a dataset query from serdes format, resolving paths to IDs.
+
+ Pipeline:
+ 1. lib/normalize — convert YAML strings to keywords (operators, types, etc.)
+ 2. resolve/import-mbql — resolve portable refs (path vectors) to integer IDs
+ Uses a sentinel resolver so unresolved refs get IDs for better error reporting."
+ [resolver query]
+ (when query
+ (try
+ (let [query (lib/normalize nil query)
+ db-name (:database query)
+ db-id (when (string? db-name)
+ (resolve/import-fk-keyed resolver db-name :model/Database :name))
+ query (cond-> query db-id (assoc :database db-id))
+ query (resolve/import-mbql resolver query)]
+ query)
+ (catch Exception _
+ (let [db-name (:database query)
+ db-id (when (string? db-name)
+ (resolve/import-fk-keyed resolver db-name :model/Database :name))]
+ (cond-> query db-id (assoc :database db-id)))))))
+
+;;; ---------------------------------------------------------------------------
+;;; High-level entity conversion (public — used by semantic.clj)
+;;; ---------------------------------------------------------------------------
+
+(declare compute-result-metadata)
+
+(defn card-metadata
+ "Convert a cached card entity to lib metadata. Returns the metadata map with
+ `:provider/resolution-failures` and `:provider/missing-database` keys when resolution fails."
+ [store data]
+ (let [!failures (atom [])
+ ;; Sentinel resolver: unresolved refs get fake integer IDs so queries can still be constructed and
+ ;; lib/find-bad-refs reports issues with context. We use the same resolver for both the query and metadata
+ ;; (table_id, FK targets, result_metadata). If we ever need to distinguish "soft failure" (metadata)
+ ;; from "must-build" (query), create a second resolver with :sentinel? false and use it for the metadata
+ ;; resolution calls below.
+ resolver (make-import-resolver store !failures :sentinel? true)
+ table-id (when-let [t (:table_id data)]
+ (if (vector? t)
+ (resolve/import-table-fk resolver t)
+ t))
+ db-name (get-in data [:dataset_query :database])
+ db-id (when (string? db-name)
+ (resolve/import-fk-keyed resolver db-name :model/Database :name))
+ dataset-query (convert-dataset-query resolver (:dataset_query data))
+ result-metadata (if-let [cols (seq (:result_metadata data))]
+ (->> cols
+ (map (partial convert-result-metadata-column resolver))
+ (lib/normalize [:sequential ::lib.schema.metadata/lib-or-legacy-column]))
+ (when (and dataset-query db-id)
+ (compute-result-metadata store dataset-query)))]
+ (cond-> {:lib/type :metadata/card
+ :id (:id data)
+ :name (:name data)
+ :type (some-> (:type data) keyword)
+ :dataset-query dataset-query
+ :result-metadata result-metadata
+ :archived (:archived data)}
+ db-id (assoc :database-id db-id)
+ (not db-id) (assoc :provider/missing-database db-name)
+ table-id (assoc :table-id table-id)
+ (seq @!failures) (assoc :provider/resolution-failures (vec (distinct @!failures))))))
+
+(defn transform-metadata
+ "Convert a cached transform entity to lib metadata with its source query resolved."
+ [store data]
+ (let [!failures (atom [])
+ resolver (make-import-resolver store !failures :sentinel? true)
+ query (convert-dataset-query resolver (get-in data [:source :query]))]
+ (cond-> {:lib/type :metadata/transform
+ :id (:id data)
+ :name (:name data)
+ :source (assoc (:source data) :query query)}
+ (seq @!failures) (assoc :provider/resolution-failures (vec (distinct @!failures))))))
+
+(defn- ->segment-metadata
+ [store data]
+ (let [!failures (atom [])
+ resolver (make-import-resolver store !failures :sentinel? true)
+ definition (convert-dataset-query resolver (:definition data))]
+ (cond-> {:lib/type :metadata/segment
+ :id (:id data)
+ :name (:name data)
+ :definition definition}
+ (seq @!failures) (assoc :provider/resolution-failures (vec (distinct @!failures))))))
+
+;;; ===========================================================================
+;;; MetadataProvider
+;;; ===========================================================================
+
+(deftype SourceMetadataProvider
+ ;; The store knows what entities exist, loads raw YAML data, assigns integer IDs, and caches.
+ ;; The provider adapts the store into the MetadataProvider interface that lib/query expects,
+ ;; converting raw YAML maps to lib metadata format along the way.
+ ;;
+ ;; `current-db` is an atom holding the database name (string) to return from `(database)`.
+ ;; Must be set via `set-database!` before each entity check and unset afterward.
+ [store current-db]
+ ISettableDatabase
+ (set-database! [_this db-name]
+ (reset! current-db db-name))
+ (clear-database! [_this]
+ (reset! current-db nil))
+
+ lib.metadata.protocols/MetadataProvider
+ (database [_this]
+ (let [db-name @current-db]
+ (when (nil? db-name)
+ (throw (ex-info "No current database set on provider. Call set-database! before querying." {})))
+ (when-let [data (store/load-database! store db-name)]
+ (->database-metadata data))))
+
+ (metadatas [_this {:keys [lib/type id table-id name]}]
+ (case type
+ :metadata/table
+ (vec
+ (if id
+ (for [tid id
+ :let [path (store/id->ref store :table tid)]
+ :when path
+ :let [data (store/load-table! store path)]
+ :when data]
+ (->table-metadata data))
+ ;; Only return tables for the current database — uses lightweight
+ ;; path-derived metadata, no YAML parsing
+ (let [db-name @current-db]
+ (for [path (store/tables-for-database store db-name)
+ :let [data (store/ensure-table-id! store path)]
+ :when data]
+ (->table-metadata data)))))
+
+ :metadata/column
+ (vec
+ (cond
+ id
+ (for [fid id
+ :let [path (store/id->ref store :field fid)]
+ :when path
+ :let [data (store/load-field! store path)]
+ :when data]
+ (->field-metadata (make-import-resolver store nil) data))
+
+ table-id
+ (let [table-path (store/id->ref store :table table-id)]
+ (when table-path
+ (for [fpath (store/fields-for-table store table-path)
+ :let [data (store/load-field! store fpath)]
+ :when data]
+ (->field-metadata (make-import-resolver store nil) data))))
+
+ :else
+ (for [path (store/all-field-paths store)
+ :let [data (store/load-field! store path)]
+ :when data]
+ (->field-metadata (make-import-resolver store nil) data))))
+
+ :metadata/card
+ (vec
+ (if id
+ (for [cid id
+ :let [eid (store/id->ref store :card cid)]
+ :when eid
+ :let [data (store/load-card! store eid)]
+ :when data]
+ (card-metadata store data))
+ (for [eid (store/all-card-ids store)
+ :let [data (store/load-card! store eid)]
+ :when data]
+ (card-metadata store data))))
+
+ :metadata/native-query-snippet
+ (vec
+ (cond
+ id
+ (for [sid id
+ :let [eid (store/id->ref store :snippet sid)]
+ :when eid
+ :let [data (store/load-snippet! store eid)]
+ :when data]
+ {:lib/type :metadata/native-query-snippet
+ :id (:id data)
+ :name (:name data)
+ :content (:content data)})
+
+ name
+ (for [eid (store/all-refs store :snippet)
+ :let [data (store/load-snippet! store eid)]
+ :when data
+ :when (contains? name (:name data))]
+ {:lib/type :metadata/native-query-snippet
+ :id (:id data)
+ :name (:name data)
+ :content (:content data)})
+
+ :else []))
+
+ :metadata/transform
+ (vec
+ (if id
+ (for [tid id
+ :let [eid (store/id->ref store :transform tid)]
+ :when eid
+ :let [data (store/load-transform! store eid)]
+ :when data]
+ (transform-metadata store data))
+ (for [eid (store/all-refs store :transform)
+ :let [data (store/load-transform! store eid)]
+ :when data]
+ (transform-metadata store data))))
+
+ :metadata/segment
+ (vec
+ (if id
+ (for [sid id
+ :let [eid (store/id->ref store :segment sid)]
+ :when eid
+ :let [data (store/load-segment! store eid)]
+ :when data]
+ (->segment-metadata store data))
+ (for [eid (store/all-refs store :segment)
+ :let [data (store/load-segment! store eid)]
+ :when data]
+ (->segment-metadata store data))))
+
+ nil))
+
+ (setting [_this _key] nil)
+
+ lib.metadata.protocols/CachedMetadataProvider
+ (cached-metadatas [this type ids]
+ (lib.metadata.protocols/metadatas this {:lib/type type :id (set ids)}))
+ (store-metadata! [_this _obj] nil)
+ (cached-value [_this _k not-found] not-found)
+ (cache-value! [_this _k _v] nil)
+ (has-cache? [_this] true)
+ (clear-cache! [_this] nil))
+
+(declare make-provider)
+
+(defn- compute-result-metadata
+ [store dataset-query]
+ (try
+ (let [provider (make-provider store)
+ db-id (:database dataset-query)
+ db-name (when (integer? db-id) (store/id->ref store :database db-id))]
+ (when db-name
+ (set-database! provider db-name))
+ (let [query (lib/query provider dataset-query)]
+ (mapv #(select-keys % [:name :base-type :effective-type :semantic-type :display-name])
+ (lib/returned-columns query))))
+ (catch Exception _ nil)))
+
+;;; ===========================================================================
+;;; Public API
+;;; ===========================================================================
+
+(defn make-provider
+ "Create a MetadataProvider backed by a store.
+ The store holds sources, index, ID registry, and entity caches —
+ see `checker.store/make-store`."
+ [store]
+ (->SourceMetadataProvider store (atom nil)))
+
+(defn database-name-for-entity
+ "Determine the database name for an entity from its raw YAML data.
+ Cards have it in dataset_query.database, transforms in source.query.database."
+ [store kind data]
+ (let [db-ref (case kind
+ :card (get-in data [:dataset_query :database])
+ :transform (or (get-in data [:source :query :database])
+ (:source_database_id data))
+ nil)]
+ (cond
+ (string? db-ref) db-ref
+ (integer? db-ref) (store/id->ref store :database db-ref)
+ :else nil)))
diff --git a/enterprise/backend/src/metabase_enterprise/checker/semantic.clj b/enterprise/backend/src/metabase_enterprise/checker/semantic.clj
new file mode 100644
index 00000000000..8a9c656aac9
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/semantic.clj
@@ -0,0 +1,780 @@
+(ns metabase-enterprise.checker.semantic
+ "Entity validation — the checker's main entry point.
+
+ Creates a store (checker.store) from sources, wraps it in a MetadataProvider
+ (checker.provider), then validates every entity:
+ - Query validation via deps.analysis/check-entity (MBQL + native SQL)
+ - Structural checks (collection_id, dashboard layout, document links)
+
+ Public API:
+ - `(check export-dir schema-dir)` — check all entities, returns results map
+ - `(setup export-dir schema-dir)` + `(check-one ctx entity-id)` — REPL workflow"
+ (:require
+ [clojure.string :as str]
+ [metabase-enterprise.checker.format.serdes :as serdes]
+ [metabase-enterprise.checker.provider :as provider]
+ [metabase-enterprise.checker.store :as store]
+ [metabase-enterprise.dependencies.analysis :as deps.analysis]
+ [metabase.lib.core :as lib]
+ [metabase.lib.metadata :as lib.metadata]
+ [metabase.query-processor.interface :as qp.i]
+ ;; sql-tools.init registers multimethod implementations needed by deps.native-validation
+ [metabase.sql-tools.init]
+ [metabase.util.malli.fn :as mu.fn]))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; Reference resolution — resolve portable refs to IDs, collecting failures
+;;;
+;;; Resolution functions take the store and a `!failures` atom (or nil).
+;;; When a ref can't be resolved, a failure map is conj'd onto the atom
+;;; for the caller to inspect. The failure maps look like:
+;;; {:type :field :path ["DB" "schema" "table" "field"]}
+;;; {:type :table :path ["DB" "schema" "table"]}
+;;; {:type :database :name "DB"}
+;;; ===========================================================================
+;;; Entity loading by kind
+;;; ===========================================================================
+
+(def ^:private kind->load-fn
+ {:dashboard store/load-dashboard!
+ :collection store/load-collection!
+ :document store/load-document!
+ :transform store/load-transform!
+ :segment store/load-segment!
+ :measure store/load-measure!
+ :snippet store/load-snippet!})
+
+;;; ===========================================================================
+;;; Common entity checks — apply to any entity type
+;;;
+;;; These validations run on every entity (cards, dashboards, collections,
+;;; metrics, segments, etc.). Add new cross-entity checks here.
+;;; ===========================================================================
+
+(defn- validate-collection-id
+ "Validate that `collection_id` in `data` points to a known collection.
+ Returns a failure map or nil."
+ [store data]
+ (when-let [coll-id (:collection_id data)]
+ (let [kind (store/index-kind-of store coll-id)]
+ (cond
+ (nil? kind)
+ {:type :collection :entity-id coll-id
+ :message (str "collection_id " coll-id " not found")}
+
+ (not= :collection kind)
+ {:type :collection :entity-id coll-id
+ :message (str "collection_id " coll-id " points to a " (name kind) ", not a collection")}))))
+
+(defn- validate-dashboard-id
+ "Validate that `dashboard_id` in `data` points to a known dashboard.
+ Returns a failure map or nil."
+ [store data]
+ (when-let [dash-id (:dashboard_id data)]
+ (let [kind (store/index-kind-of store dash-id)]
+ (cond
+ (nil? kind)
+ {:type :dashboard :entity-id dash-id
+ :message (str "dashboard_id " dash-id " not found")}
+
+ (not= :dashboard kind)
+ {:type :dashboard :entity-id dash-id
+ :message (str "dashboard_id " dash-id " points to a " (name kind) ", not a dashboard")}))))
+
+(defn- validate-parent-id
+ "Validate that `parent_id` on a collection points to a known collection.
+ Returns a failure map or nil."
+ [store data]
+ (when-let [parent-id (:parent_id data)]
+ (let [kind (store/index-kind-of store parent-id)]
+ (cond
+ (nil? kind)
+ {:type :collection :entity-id parent-id
+ :message (str "parent_id " parent-id " not found")}
+
+ (not= :collection kind)
+ {:type :collection :entity-id parent-id
+ :message (str "parent_id " parent-id " points to a " (name kind) ", not a collection")}))))
+
+(defn- validate-document-id
+ "Validate that `document_id` in `data` points to a known document.
+ Returns a failure map or nil."
+ [store data]
+ (when-let [doc-id (:document_id data)]
+ (let [kind (store/index-kind-of store doc-id)]
+ (cond
+ (nil? kind)
+ {:type :document :entity-id doc-id
+ :message (str "document_id " doc-id " not found")}
+
+ (not= :document kind)
+ {:type :document :entity-id doc-id
+ :message (str "document_id " doc-id " points to a " (name kind) ", not a document")}))))
+
+(defn- validate-container-exclusion
+ "Validate that a card doesn't have both dashboard_id and document_id set.
+ Returns a failure map or nil."
+ [_store data]
+ (when (and (:dashboard_id data) (:document_id data))
+ {:type :container-conflict
+ :message "card has both dashboard_id and document_id set — only one container allowed"}))
+
+(defn- validate-container-collection-match
+ "Validate that a card nested under a dashboard or document has the same
+ collection_id as its container. Returns a failure map or nil."
+ [store data]
+ (when-let [container-id (or (:dashboard_id data) (:document_id data))]
+ (let [container-kind (if (:dashboard_id data) :dashboard :document)
+ load-fn (kind->load-fn container-kind)
+ container-data (when load-fn (load-fn store container-id))]
+ (when container-data
+ (let [container-coll (:collection_id container-data)
+ card-coll (:collection_id data)]
+ (when (and card-coll container-coll
+ (not= card-coll container-coll))
+ {:type :container-collection-mismatch
+ :message (str "card collection_id " card-coll
+ " doesn't match its " (name container-kind)
+ "'s collection_id " container-coll)}))))))
+
+(defn- check-common
+ "Run checks that apply to any entity (card, dashboard, metric, etc.).
+ Returns a vector of failure maps, or empty vector if all checks pass."
+ [store data]
+ (filterv some?
+ [(validate-collection-id store data)
+ (validate-dashboard-id store data)
+ (validate-document-id store data)
+ (validate-parent-id store data)
+ (validate-container-exclusion store data)
+ (validate-container-collection-match store data)]))
+
+;;; ===========================================================================
+;;; Dashboard-specific checks
+;;;
+;;; Validates dashboard internal consistency:
+;;; - dashcard card_id refs point to known cards
+;;; - dashcard dashboard_tab_id refs point to tabs in this dashboard
+;;; - dashcard grid positions are within bounds (24-column grid)
+;;; ===========================================================================
+
+(def ^:private grid-width
+ "Dashboard grid width in columns."
+ 24)
+
+(defn- validate-dashcard-card-ref
+ "Validate that a dashcard's card_id points to a known card.
+ Virtual cards (headings, text) have null card_id — that's fine."
+ [store dashcard]
+ (let [card-id (:card_id dashcard)]
+ (when (and card-id (not (store/exists? store :card card-id)))
+ {:type :dashcard-card-ref
+ :dashcard-entity-id (:entity_id dashcard)
+ :entity-id card-id
+ :message (str "dashcard " (:entity_id dashcard) " references card " card-id " which is not in the export")})))
+
+(defn- validate-dashcard-tab-ref
+ "Validate that a dashcard's dashboard_tab_id references a tab in this dashboard.
+ dashboard_tab_id is [dashboard-entity-id, tab-entity-id]."
+ [tab-entity-ids dashcard]
+ (when-let [tab-ref (:dashboard_tab_id dashcard)]
+ (let [tab-eid (if (vector? tab-ref) (second tab-ref) tab-ref)]
+ (when (and tab-eid (not (contains? tab-entity-ids tab-eid)))
+ {:type :dashcard-tab-ref
+ :dashcard-entity-id (:entity_id dashcard)
+ :entity-id tab-eid
+ :message (str "dashcard " (:entity_id dashcard) " references tab " tab-eid " which is not in this dashboard")}))))
+
+(defn- validate-dashcard-grid
+ "Validate that a dashcard's grid position is within bounds."
+ [dashcard]
+ (let [col (or (:col dashcard) 0)
+ row (or (:row dashcard) 0)
+ size-x (or (:size_x dashcard) 1)
+ size-y (or (:size_y dashcard) 1)
+ eid (:entity_id dashcard)]
+ (cond
+ (neg? col)
+ {:type :dashcard-grid :dashcard-entity-id eid
+ :message (str "dashcard " eid " has negative col: " col)}
+
+ (> (+ col size-x) grid-width)
+ {:type :dashcard-grid :dashcard-entity-id eid
+ :message (str "dashcard " eid " extends beyond grid: col=" col " + size_x=" size-x " > " grid-width)}
+
+ (< size-x 1)
+ {:type :dashcard-grid :dashcard-entity-id eid
+ :message (str "dashcard " eid " has invalid size_x: " size-x)}
+
+ (< size-y 1)
+ {:type :dashcard-grid :dashcard-entity-id eid
+ :message (str "dashcard " eid " has invalid size_y: " size-y)}
+
+ (neg? row)
+ {:type :dashcard-grid :dashcard-entity-id eid
+ :message (str "dashcard " eid " has negative row: " row)})))
+
+(defn- check-dashboard
+ "Run dashboard-specific semantic checks on loaded dashboard data.
+ Returns a vector of failure maps."
+ [store data]
+ (let [tabs (:tabs data)
+ dashcards (:dashcards data)
+ tab-entity-ids (into #{} (keep :entity_id) tabs)]
+ (into []
+ (keep identity)
+ (mapcat (fn [dc]
+ [(validate-dashcard-card-ref store dc)
+ (validate-dashcard-tab-ref tab-entity-ids dc)
+ (validate-dashcard-grid dc)])
+ dashcards))))
+
+;;; ===========================================================================
+;;; Document-specific checks
+;;;
+;;; Documents contain ProseMirror content with embedded card references.
+;;; cardEmbed nodes have attrs.id which is a serdes portable ref:
+;;; [{:model "Card" :id "entity-id"}]
+;;; ===========================================================================
+
+(defn- extract-card-embeds
+ "Walk a ProseMirror content tree and extract card entity-ids from cardEmbed nodes."
+ [node]
+ (when (map? node)
+ (let [embeds (when (= "cardEmbed" (:type node))
+ (let [refs (get-in node [:attrs :id])]
+ (keep (fn [ref]
+ (when (and (map? ref) (= "Card" (:model ref)))
+ (:id ref)))
+ (if (sequential? refs) refs [refs]))))
+ child-embeds (mapcat extract-card-embeds (:content node))]
+ (concat embeds child-embeds))))
+
+(defn- extract-entity-links
+ "Walk a ProseMirror content tree and extract entity references from link hrefs.
+ Links like /dashboard/ABC or /question/ABC reference entities by entity_id."
+ [node]
+ (when (map? node)
+ (let [;; Check marks on text nodes for links
+ mark-links (when-let [marks (:marks node)]
+ (keep (fn [mark]
+ (when (= "link" (:type mark))
+ (when-let [href (get-in mark [:attrs :href])]
+ (when-let [[_ kind eid] (re-matches #"/(dashboard|question|card|collection)/(.+)" (str href))]
+ {:kind (case kind
+ "dashboard" :dashboard
+ "question" :card
+ "card" :card
+ "collection" :collection)
+ :entity-id eid}))))
+ marks))
+ child-links (mapcat extract-entity-links (:content node))]
+ (concat mark-links child-links))))
+
+(defn- check-document
+ "Run document-specific semantic checks. Validates embedded card refs and entity links."
+ [store data]
+ (let [doc (:document data)
+ card-eids (extract-card-embeds doc)
+ links (extract-entity-links doc)]
+ (into []
+ cat
+ [;; Check card embeds
+ (keep (fn [card-eid]
+ (when-not (store/exists? store :card card-eid)
+ {:type :document-card-ref
+ :entity-id card-eid
+ :message (str "document embeds card " card-eid " which is not in the export")}))
+ card-eids)
+ ;; Check entity links in href attributes
+ (keep (fn [{:keys [kind entity-id]}]
+ (when-not (store/exists? store kind entity-id)
+ {:type :document-link
+ :entity-id entity-id
+ :message (str "document links to " (name kind) " " entity-id " which is not in the export")}))
+ links)])))
+
+;;; ===========================================================================
+;;; deps.analysis helpers
+;;; ===========================================================================
+
+(defn- partition-errors
+ "Split a set of check-entity errors into :bad-refs (MBQL issues) and
+ :native-errors (SQL validation issues like :missing-column, :syntax-error)."
+ [errors]
+ (let [{native true, mbql false} (group-by #(contains? #{:missing-column :syntax-error :missing-table-alias} (:type %))
+ errors)]
+ {:bad-refs (vec mbql)
+ :native-errors (when (seq native) (set native))}))
+
+(defn- humanize-source-entity
+ "Replace synthetic :source-entity-id integers with portable refs (entity-ids
+ or path strings) so error output is meaningful."
+ [store error]
+ (if-let [src-id (:source-entity-id error)]
+ (let [kind (:source-entity-type error)
+ ref (case kind
+ :table (some->> (store/id->ref store :table src-id) (str/join "."))
+ :card (or (store/id->ref store :card src-id) src-id)
+ src-id)]
+ (assoc error :source-entity-id ref))
+ error))
+
+(defn- humanize-errors
+ "Translate all synthetic IDs in a set of errors back to portable refs."
+ [store errors]
+ (when errors
+ (into (empty errors) (map #(humanize-source-entity store %)) errors)))
+
+;;; ===========================================================================
+;;; Transform-specific checks
+;;;
+;;; Transforms have a source query (native or MBQL) and a target table.
+;;; We validate the database ref and the source query via deps.analysis.
+;;; ===========================================================================
+
+(defn- check-transform
+ "Run transform-specific semantic checks via deps.analysis/check-entity.
+ Loads the transform from the store (which caches and assigns IDs), then
+ delegates query validation to deps.analysis.
+ Returns a result map with :unresolved, :native-errors, :bad-refs, :error, :sql keys."
+ [store provider entity-id]
+ (let [data (store/load-transform! store entity-id)]
+ (if (not= "query" (get-in data [:source :type]))
+ ;; Non-query transforms (e.g. python) — no query to validate
+ {:unresolved []}
+ ;; Query-based transforms — validate via deps.analysis
+ (let [db-name (or (get-in data [:source :query :database])
+ (:source_database_id data))
+ unresolved (atom [])
+ transform-id (:id data)
+ sql (get-in data [:source :query :stages 0 :native])]
+ ;; Check database ref
+ (when (and db-name (not (store/exists? store :database db-name)))
+ (swap! unresolved conj {:type :database :name db-name
+ :message (str "transform references database " db-name " which is not in the schema")}))
+ ;; Resolution failures from converting transform metadata
+ (let [transform-meta (provider/transform-metadata store data)]
+ (swap! unresolved into (or (:provider/resolution-failures transform-meta) []))
+ ;; Validate via deps.analysis
+ (let [errors (try
+ (deps.analysis/check-entity provider :transform transform-id)
+ (catch Exception e
+ #{{:type :validation-exception-error :message (.getMessage e)}}))
+ {:keys [bad-refs native-errors]} (partition-errors errors)
+ native-errors (humanize-errors store native-errors)
+ bad-refs (mapv #(humanize-source-entity store %) bad-refs)]
+ (cond-> {:unresolved @unresolved}
+ (seq bad-refs) (assoc :bad-refs bad-refs)
+ (seq native-errors) (assoc :native-errors native-errors)
+ sql (assoc :sql sql))))))))
+
+;;; ===========================================================================
+;;; Non-card entity checking — load from files, run common + type-specific checks
+;;; ===========================================================================
+
+(defn- id->path
+ "Convert an integer ID back to a human-readable path string for error messages."
+ [store id id-type]
+ (case id-type
+ :table (some->> (store/id->ref store :table id) (str/join "."))
+ :field (some->> (store/id->ref store :field id) (str/join "."))
+ :card (when-let [eid (store/id->ref store :card id)]
+ (or (:name (store/cached-entity store :card eid)) eid))
+ :database (store/id->ref store :database id)
+ nil))
+
+(declare extract-refs-from-card)
+
+(defn- merge-refs
+ "Merge multiple ref maps, concatenating and deduplicating each key."
+ [& ref-maps]
+ (reduce (fn [acc m]
+ (reduce-kv (fn [a k v]
+ (if (seq v)
+ (update a k (fn [existing]
+ (vec (distinct (into (or existing []) v)))))
+ a))
+ acc m))
+ {} ref-maps))
+
+(declare extract-entity-refs-from-raw-data)
+
+(defn- extract-refs-from-query
+ ([store query] (extract-refs-from-query store query nil #{}))
+ ([store query provider visited]
+ (let [table-ids (lib/all-source-table-ids query)
+ card-ids (lib/all-source-card-ids query)
+ cols (try (lib/returned-columns query) (catch Exception _ nil))
+ tables (mapv #(id->path store % :table) table-ids)
+ fields (when cols (->> cols (keep :id) (mapv #(id->path store % :field))))
+ cards (mapv #(id->path store % :card) card-ids)
+ transitive (when provider
+ (for [cid card-ids :when (not (visited cid))]
+ (extract-refs-from-card store provider cid (conj visited cid))))
+ direct {:tables (vec (remove nil? tables))
+ :fields (vec (remove nil? fields))
+ :source-cards (vec (remove nil? cards))}]
+ (apply merge-refs direct transitive))))
+
+(defn- extract-refs-from-card [store provider card-id visited]
+ (try
+ (when-let [card (lib.metadata/card provider card-id)]
+ (let [;; Get the raw card data from the store for entity ref extraction
+ eid (store/id->ref store :card card-id)
+ raw-data (when eid (store/cached-entity store :card eid))
+ entity-refs (when raw-data (extract-entity-refs-from-raw-data raw-data))
+ query-refs (when-let [dq (:dataset-query card)]
+ (extract-refs-from-query store (lib/query provider dq) provider visited))]
+ (merge-refs (or query-refs {}) entity-refs)))
+ (catch Exception _ {})))
+
+(defn- extract-entity-refs-from-raw-data
+ "Extract snippet, measure, metric, and segment references from raw card data.
+ These are extracted from the YAML structure before resolution, since the
+ resolved lib query normalizes them away."
+ [data]
+ (let [stages (get-in data [:dataset_query :stages])]
+ (reduce
+ (fn [acc stage]
+ (let [;; Snippets from template-tags
+ snippets (->> (vals (:template-tags stage))
+ (filter #(= "snippet" (get % :type)))
+ (keep :snippet-name))
+ ;; Measures and metrics from aggregation clauses
+ agg-refs (->> (:aggregation stage)
+ (filter vector?)
+ (keep (fn [clause]
+ (let [tag (first clause)]
+ (case (str tag)
+ "measure" {:kind :measures :ref (last clause)}
+ "metric" {:kind :metrics :ref (last clause)}
+ nil)))))
+ ;; Segments from filter clauses
+ seg-refs (->> (:filters stage)
+ (filter vector?)
+ (keep (fn [clause]
+ (when (= "segment" (str (first clause)))
+ {:kind :segments :ref (last clause)}))))]
+ (cond-> acc
+ (seq snippets) (update :snippets into snippets)
+ (seq agg-refs) (as-> a (reduce (fn [m {:keys [kind ref]}]
+ (update m kind (fnil conj []) ref))
+ a agg-refs))
+ (seq seg-refs) (update :segments into (map :ref seg-refs)))))
+ {}
+ (or stages []))))
+
+;;; ===========================================================================
+;;; Type-specific checks
+;;; ===========================================================================
+
+(defn- check-card-specific
+ "Card-specific checks: query validation via deps.analysis + ref extraction."
+ [store provider data]
+ (let [card-id (:id data)
+ card-meta (provider/card-metadata store data)
+ resolution-failures (:provider/resolution-failures card-meta)
+ errors (when-not (:provider/missing-database card-meta)
+ (try
+ (deps.analysis/check-entity provider :card card-id)
+ (catch Exception e
+ #{{:type :validation-exception-error :message (.getMessage e)}})))
+ {:keys [bad-refs native-errors]} (partition-errors errors)
+ native-errors (humanize-errors store native-errors)
+ bad-refs (mapv #(humanize-source-entity store %) bad-refs)
+ query-refs (try
+ (let [card (lib.metadata/card provider card-id)
+ query (lib/query provider (:dataset-query card))]
+ (extract-refs-from-query store query provider #{card-id}))
+ (catch Exception _ nil))
+ entity-refs (extract-entity-refs-from-raw-data data)
+ refs (when query-refs
+ (cond-> query-refs
+ (seq (:snippets entity-refs)) (assoc :snippets (vec (distinct (:snippets entity-refs))))
+ (seq (:measures entity-refs)) (assoc :measures (vec (:measures entity-refs)))
+ (seq (:metrics entity-refs)) (assoc :metrics (vec (:metrics entity-refs)))
+ (seq (:segments entity-refs)) (assoc :segments (vec (:segments entity-refs)))))]
+ {:card-id card-id
+ :unresolved (into (or resolution-failures [])
+ (when (:provider/missing-database card-meta)
+ [{:type :database :name (:provider/missing-database card-meta)}]))
+ :refs refs
+ :bad-refs bad-refs
+ :native-errors native-errors
+ :error (when (:provider/missing-database card-meta)
+ (str "Unknown database: " (:provider/missing-database card-meta)))}))
+
+;;; ===========================================================================
+;;; Unified entity checking
+;;; ===========================================================================
+
+(defn- check-entity
+ "Check a single entity by kind. Runs common checks on all entities,
+ plus type-specific checks (query validation for cards/transforms,
+ structural checks for dashboards/documents).
+ Returns [entity-id result-map]."
+ [store provider kind entity-id]
+ (try
+ (let [load-fn (or (kind->load-fn kind) store/load-card!)
+ data (load-fn store entity-id)
+ _ (provider/set-database! provider (provider/database-name-for-entity store kind data))
+ common (check-common store data)
+ type-specific (case kind
+ :card (check-card-specific store provider data)
+ :dashboard {:unresolved (check-dashboard store data)}
+ :document {:unresolved (check-document store data)}
+ :transform (check-transform store provider entity-id)
+ {})
+ all-unresolved (into common (:unresolved type-specific))
+ result (cond-> {:name (or (:name data) entity-id)
+ :entity-id entity-id
+ :unresolved all-unresolved
+ :native-errors (:native-errors type-specific)
+ :bad-refs (:bad-refs type-specific)
+ :error (:error type-specific)
+ :sql (:sql type-specific)}
+ (:card-id type-specific) (assoc :card-id (:card-id type-specific))
+ (:refs type-specific) (assoc :refs (:refs type-specific))
+ (not= kind :card) (assoc :kind kind))]
+ [entity-id result])
+ (catch Exception e
+ [entity-id {:name entity-id
+ :entity-id entity-id
+ :kind (when (not= kind :card) kind)
+ :error (.getMessage e)}])
+ (finally
+ (provider/clear-database! provider))))
+
+(defn- check-duplicate-entity-ids
+ "Check for duplicate entity_ids in the index. Returns a map of
+ synthetic key → result for each duplicate group."
+ [index]
+ (when-let [dupes (:duplicates index)]
+ (into {}
+ (for [{:keys [kind ref files]} dupes]
+ [(str "duplicate:" ref)
+ {:name (str "Duplicate " (name kind) " entity_id: " ref)
+ :entity-id ref
+ :kind kind
+ :error (str "entity_id " ref " appears in " (count files) " files: "
+ (str/join ", " files))}]))))
+
+(def ^:private checked-kinds
+ "Entity kinds that the checker validates, in processing order."
+ [:card :dashboard :collection :document :measure :segment :transform])
+
+(defn check-entities
+ "Check all entities: cards, dashboards, collections, documents, transforms, etc.
+ Returns `{entity-id result-map}`. See the Results processing section for the
+ result-map shape.
+
+ `schema-source` — a SchemaSource for resolving databases/tables/fields
+ `assets-source` — an AssetsSource for resolving cards/snippets/transforms/segments
+ `index` — a file index (see store/make-store)
+ `entity-ids` — optional seq of entity-ids to check (defaults to all entities)"
+ ([schema-source assets-source index]
+ (check-entities schema-source assets-source index nil))
+ ([schema-source assets-source index entity-ids]
+ (binding [mu.fn/*enforce* false
+ qp.i/*skip-middleware-because-app-db-access* true]
+ (let [store (store/make-store schema-source assets-source index)
+ provider (provider/make-provider store)
+ results (if entity-ids
+ ;; Check specific entities — look up their kind
+ (into {}
+ (for [eid entity-ids
+ :let [kind (or (store/index-kind-of store eid) :card)]]
+ (check-entity store provider kind eid)))
+ ;; Check all entities by kind
+ (into {}
+ (for [kind checked-kinds
+ eid (store/all-refs store kind)]
+ (check-entity store provider kind eid))))
+ dupes (check-duplicate-entity-ids index)]
+ (merge results dupes)))))
+
+;;; ===========================================================================
+;;; Results processing — pure functions on result data
+;;;
+;;; Result map shape (per entity):
+;;;
+;;; :name string — entity name
+;;; :entity-id string — portable entity ID
+;;; :card-id int? — synthetic integer ID (cards only)
+;;; :kind keyword? — :dashboard, :transform, etc. (non-card entities)
+;;; :error string? — fatal error message (e.g. unknown database)
+;;; :unresolved [map] — unresolved reference failures
+;;; each {:type kw, :path vec?, :entity-id str?,
+;;; :name str?, :message str?}
+;;; :bad-refs [map] — query issues from deps.analysis
+;;; (e.g. {:type :validation-exception-error, :message str})
+;;; :native-errors #{map} — native SQL errors from deps.analysis
+;;; (e.g. {:type :missing-column, :name str,
+;;; :source-entity-type kw?, :source-entity-id str?})
+;;; :refs map? — extracted references for display (cards only)
+;;; {:tables [str], :fields [str], :source-cards [str]}
+;;; :sql string? — source SQL (transforms with native queries)
+;;;
+;;; Summary map shape (from summarize-results):
+;;;
+;;; :total int — total entities checked
+;;; :ok int — entities with no issues
+;;; :errors int — entities with fatal errors
+;;; :unresolved int — entities with unresolved references
+;;; :native-errors int — entities with native SQL errors
+;;; :issues int — entities with bad refs / query issues
+;;; ===========================================================================
+
+(defn result-status
+ "Compute the status of a single entity result.
+ Returns :error, :unresolved, :native-errors, :issues, or :ok (checked in priority order)."
+ [result]
+ (cond
+ (:error result) :error
+ (seq (:unresolved result)) :unresolved
+ (seq (:native-errors result)) :native-errors
+ (seq (:bad-refs result)) :issues
+ :else :ok))
+
+(defn summarize-results
+ "Summarize check results into counts by status."
+ [results]
+ (let [by-status (group-by (comp result-status second) results)]
+ {:total (count results)
+ :ok (count (get by-status :ok))
+ :errors (count (get by-status :error))
+ :unresolved (count (get by-status :unresolved))
+ :native-errors (count (get by-status :native-errors))
+ :issues (count (get by-status :issues))}))
+
+(defn results-by-status
+ "Group results by status. Returns map of status → seq of [entity-id result]."
+ [results]
+ (group-by (comp result-status second) results))
+
+(defn format-result
+ "Format a single entity result as a human-readable string."
+ [[entity-id result]]
+ (let [lines (atom [(str "=== " (:name result) " [" entity-id "] ===")
+ (if-let [kind (:kind result)]
+ (str " Kind: " (name kind))
+ (str " Card ID: " (:card-id result)))])
+ {:keys [tables fields source-cards]} (:refs result)]
+ (when (seq tables)
+ (swap! lines conj (str " Tables: " (str/join ", " tables))))
+ (when (seq fields)
+ (swap! lines conj (str " Fields: " (str/join ", " fields))))
+ (when (seq source-cards)
+ (swap! lines conj (str " Source Cards: " (str/join ", " source-cards))))
+ (when-let [unresolved (:unresolved result)]
+ (swap! lines conj " UNRESOLVED REFERENCES:")
+ (doseq [{:keys [type path entity-id name message]} unresolved]
+ (swap! lines conj (str " - " (clojure.core/name type) ": "
+ (or (some->> path (str/join ".")) entity-id name message)))))
+ (when (seq (:native-errors result))
+ (swap! lines conj " NATIVE SQL ERRORS:")
+ (doseq [err (:native-errors result)]
+ (swap! lines conj (str " - " (pr-str err))))
+ (when-let [sql (:sql result)]
+ (swap! lines conj (str " SQL: " sql))))
+ (swap! lines conj (case (result-status result)
+ :error (str " ERROR: " (:error result))
+ :unresolved " Status: MISSING REFS"
+ :native-errors " Status: NATIVE SQL ERRORS"
+ :issues (str " Status: ISSUES FOUND\n"
+ (str/join "\n" (map #(str " - " (pr-str %)) (:bad-refs result))))
+ :ok " Status: OK"))
+ (str/join "\n" @lines)))
+
+(defn format-error
+ "Format a single card error concisely for LLM consumption.
+ Returns nil for :ok results. Only includes actionable error information."
+ [[entity-id result]]
+ (let [status (result-status result)]
+ (when (not= :ok status)
+ (let [kind-label (if-let [kind (:kind result)] (name kind) "card")
+ lines (atom [(str kind-label ": " (:name result) " (entity_id: " entity-id ")")])]
+ (when-let [unresolved (seq (:unresolved result))]
+ (doseq [{:keys [type path entity-id name message]} unresolved]
+ (swap! lines conj (str " unresolved " (clojure.core/name type) ": "
+ (or (some->> path (str/join ".")) entity-id name message)))))
+ (when (seq (:bad-refs result))
+ (doseq [ref (:bad-refs result)]
+ (swap! lines conj (str " bad ref: " (pr-str ref)))))
+ (when (seq (:native-errors result))
+ (doseq [err (:native-errors result)]
+ (swap! lines conj (str " sql error: " (pr-str (dissoc err :source-entity-type :source-entity-id)))))
+ (when-let [sql (:sql result)]
+ (swap! lines conj (str " sql: " sql))))
+ (when (:error result)
+ (swap! lines conj (str " error: " (:error result))))
+ (str/join "\n" @lines)))))
+
+(defn- make-sources-and-index
+ "Build schema and assets sources from `schema-dir` and `export-dir`,
+ and a merged assets index (cards, dashboards, segments, etc.)."
+ [export-dir schema-dir]
+ (let [schema-source (serdes/make-database-source schema-dir)
+ assets-source (serdes/make-source export-dir)
+ index (serdes/source-index assets-source)]
+ {:schema-source schema-source :assets-source assets-source :index index}))
+
+(defn check
+ "Check entities in an export directory against database schemas.
+
+ `export-dir` — directory containing serdes-exported entities (collections/)
+ `schema-dir` — directory containing serdes-exported database schemas
+ `entity-ids` — optional seq of entity-ids to check (defaults to all entities)
+
+ Returns `{:results results-map}` where results-map is `{entity-id result-map}`.
+ See the Results processing section above for the result-map and summary-map shapes."
+ ([export-dir schema-dir]
+ (let [{:keys [schema-source assets-source index]} (make-sources-and-index export-dir schema-dir)]
+ {:results (check-entities schema-source assets-source index)}))
+ ([export-dir schema-dir entity-ids]
+ (let [{:keys [schema-source assets-source index]} (make-sources-and-index export-dir schema-dir)]
+ {:results (check-entities schema-source assets-source index entity-ids)})))
+
+(defn setup
+ "Create a store and provider for REPL iteration.
+ Returns {:store store :provider provider :index index}.
+
+ Usage:
+ (def ctx (setup \"/path/to/export\" \"/path/to/schemas\"))
+ (check-one ctx \"entity-id\")
+ (check-one ctx \"entity-id\" :verbose true)"
+ [export-dir schema-dir]
+ (let [{:keys [schema-source assets-source index]} (make-sources-and-index export-dir schema-dir)
+ store (binding [mu.fn/*enforce* false]
+ (store/make-store schema-source assets-source index))
+ provider (provider/make-provider store)]
+ {:store store :provider provider :index index}))
+
+(defn check-one
+ "Check a single entity by entity-id using a pre-built context from [[setup]].
+ Looks up the entity's kind from the store. Returns the result map.
+ With :verbose true, also prints formatted output."
+ [{:keys [store provider]} entity-id & {:keys [verbose]}]
+ (binding [mu.fn/*enforce* false
+ qp.i/*skip-middleware-because-app-db-access* true]
+ (let [kind (or (store/index-kind-of store entity-id) :card)
+ [_ result] (check-entity store provider kind entity-id)]
+ (when verbose
+ #_{:clj-kondo/ignore [:discouraged-var]}
+ (println (format-result [entity-id result])))
+ result)))
+
+(comment
+ ;; REPL workflow:
+ (check "/Users/dan/projects/work/stats-remote-sync"
+ "/tmp/metadata/metadata/databases")
+ (setup
+ "/Users/dan/projects/work/stats-remote-sync"
+ "/Users/dan/projects/work/stats-remote-sync/databases")
+
+ (check-one (setup "/Users/dan/projects/work/stats-remote-sync"
+ "/tmp/metadata/metadata/databases")
+ "2MNFkdc6EvgmUJU4E0ytF"))
diff --git a/enterprise/backend/src/metabase_enterprise/checker/source.clj b/enterprise/backend/src/metabase_enterprise/checker/source.clj
new file mode 100644
index 00000000000..a6ffee95911
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/source.clj
@@ -0,0 +1,74 @@
+(ns metabase-enterprise.checker.source
+ "Protocols for resolving portable references to entity data.
+
+ Two protocols separate the two concerns:
+
+ - **SchemaSource** resolves database schema references (databases, tables, fields).
+ Backed by the `--schema-dir` directory.
+ - **AssetsSource** resolves serialized entity references (cards, snippets, transforms,
+ segments). Backed by the `--export` directory.
+
+ Portable references:
+ - Database: string name
+ - Table: [db-name schema-name table-name] where schema can be nil
+ - Field: [db-name schema-name table-name field-name]
+ - Card/snippet/transform/segment: entity-id string
+
+ The checker assigns integer IDs; sources just say 'yes this exists, here's the data'
+ or 'no, this reference is unresolved'.")
+
+(set! *warn-on-reflection* true)
+
+(defprotocol SchemaSource
+ "Resolve database schema references to entity data."
+
+ (resolve-database [this db-name]
+ "Resolve database by name. Returns map with :name, :engine, :settings or nil.")
+
+ (resolve-table [this table-path]
+ "Resolve table by [db schema table]. Returns map with :name, :schema, :display-name, etc. or nil.")
+
+ (resolve-field [this field-path]
+ "Resolve field by [db schema table field]. Returns map with :name, :base-type, :semantic-type, etc. or nil.")
+
+ (fields-for-table [this table-path]
+ "Return a set of field paths belonging to the given table path [db schema table].")
+
+ (all-field-paths [this]
+ "Return all known field paths.")
+
+ (all-database-names [this]
+ "Return all known database names.")
+
+ (all-table-paths [this]
+ "Return all known table paths.")
+
+ (tables-for-database [this db-name]
+ "Return table paths belonging to the given database name."))
+
+(defprotocol AssetsSource
+ "Resolve serialized entity references to entity data."
+
+ (resolve-card [this entity-id]
+ "Resolve card by entity-id. Returns map with :name, :dataset-query, :result-metadata, etc. or nil.")
+
+ (resolve-snippet [this entity-id]
+ "Resolve native query snippet by entity-id. Returns map with :name, :content, etc. or nil.")
+
+ (resolve-transform [this entity-id]
+ "Resolve transform by entity-id. Returns map with :name, :source, etc. or nil.")
+
+ (resolve-segment [this entity-id]
+ "Resolve segment by entity-id. Returns map with :name, :definition, etc. or nil.")
+
+ (resolve-dashboard [this entity-id]
+ "Resolve dashboard by entity-id. Returns map with :name, :dashcards, :tabs, etc. or nil.")
+
+ (resolve-collection [this entity-id]
+ "Resolve collection by entity-id. Returns map with :name, :parent_id, etc. or nil.")
+
+ (resolve-document [this entity-id]
+ "Resolve document by entity-id. Returns map with :name, :document, etc. or nil.")
+
+ (resolve-measure [this entity-id]
+ "Resolve measure by entity-id. Returns map with :name, :definition, etc. or nil."))
diff --git a/enterprise/backend/src/metabase_enterprise/checker/store.clj b/enterprise/backend/src/metabase_enterprise/checker/store.clj
new file mode 100644
index 00000000000..9d7b4517eb4
--- /dev/null
+++ b/enterprise/backend/src/metabase_enterprise/checker/store.clj
@@ -0,0 +1,267 @@
+(ns metabase-enterprise.checker.store
+ "The store knows what entities exist and can load them.
+
+ It combines three things:
+ - A **SchemaSource** for resolving databases, tables, and fields
+ - An **AssetsSource** for resolving cards, snippets, transforms, and segments
+ - A **file index** that enumerates all known entities by kind and ref
+
+ On top of that it maintains:
+ - A bidirectional **ID registry** (portable refs ↔ synthetic integer IDs,
+ because lib requires integer IDs)
+ - **Entity caches** (loaded data with :id stamped on)
+
+ Entities are loaded lazily from sources and cached on first access.
+ The store is an atom passed explicitly — no dynamic vars."
+ (:require
+ [metabase-enterprise.checker.source :as source]))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; Store creation
+;;; ===========================================================================
+
+(defn make-store
+ "Create a fresh store for a checking session.
+
+ `schema-source` is a SchemaSource for resolving databases/tables/fields.
+ `assets-source` is an AssetsSource for resolving cards/snippets/transforms/segments.
+ `index` is a file index: `{kind {ref file-path}}` — databases from the schema
+ source, cards/dashboards/collections/transforms/segments/measures from assets."
+ [schema-source assets-source index]
+ (atom {:schema-source schema-source
+ :assets-source assets-source
+ :index index
+ :id-counter 0
+ :ref->id {} ; {kind {ref id}}
+ :id->ref {} ; {kind {id ref}}
+ :entities {}})) ; {kind {ref data}}
+
+;;; ===========================================================================
+;;; Index queries
+;;; ===========================================================================
+
+(defn all-refs
+ "All known refs of a given kind from the file index."
+ [store kind]
+ (keys (get-in @store [:index kind])))
+
+(defn exists?
+ "Is `ref` of `kind` known to the store?
+ Databases are checked via the schema source (not the assets index)."
+ [store kind ref]
+ (if (= kind :database)
+ (contains? (set (source/all-database-names (:schema-source @store))) ref)
+ (contains? (get-in @store [:index kind]) ref)))
+
+(defn index-kind-of
+ "Return the kind of an entity-id in the index, or nil if not found."
+ [store entity-id]
+ (cond
+ (exists? store :collection entity-id) :collection
+ (exists? store :card entity-id) :card
+ (exists? store :dashboard entity-id) :dashboard
+ (exists? store :document entity-id) :document
+ (exists? store :measure entity-id) :measure
+ (exists? store :segment entity-id) :segment
+ (exists? store :snippet entity-id) :snippet
+ (exists? store :transform entity-id) :transform
+ :else nil))
+
+(defn index-file
+ "Get the file path for a ref of a given kind from the index, or nil."
+ [store kind ref]
+ (get-in @store [:index kind ref]))
+
+(defn all-database-names
+ "All database names from the schema source."
+ [store]
+ (source/all-database-names (:schema-source @store)))
+
+(defn all-table-paths
+ "All table paths from the schema source."
+ [store]
+ (source/all-table-paths (:schema-source @store)))
+
+(defn all-field-paths
+ "All field paths from the schema source."
+ [store]
+ (source/all-field-paths (:schema-source @store)))
+
+(defn all-card-ids
+ "All card entity-ids from the file index."
+ [store]
+ (all-refs store :card))
+
+(defn tables-for-database
+ "Table paths belonging to a specific database."
+ [store db-name]
+ (source/tables-for-database (:schema-source @store) db-name))
+
+(defn fields-for-table
+ "Field paths belonging to a specific table path."
+ [store table-path]
+ (source/fields-for-table (:schema-source @store) table-path))
+
+;;; ===========================================================================
+;;; ID registry — assign synthetic integer IDs to portable refs
+;;; ===========================================================================
+
+(defn- next-id! [store]
+ (:id-counter (swap! store update :id-counter inc)))
+
+(defn get-or-assign!
+ "Return the integer ID for `ref` under `kind`, assigning one if needed."
+ [store kind ref]
+ (or (get-in @store [:ref->id kind ref])
+ (let [id (next-id! store)]
+ (swap! store (fn [s]
+ (-> s
+ (assoc-in [:ref->id kind ref] id)
+ (assoc-in [:id->ref kind id] ref))))
+ id)))
+
+(defn id->ref
+ "Look up the portable ref for an integer ID. Returns nil if unknown."
+ [store kind id]
+ (get-in @store [:id->ref kind id]))
+
+(defn ref->id
+ "Look up the integer ID for a portable ref. Returns nil if not yet assigned."
+ [store kind ref]
+ (get-in @store [:ref->id kind ref]))
+
+;;; ===========================================================================
+;;; Entity loading — lazy load from source, cache with assigned IDs
+;;; ===========================================================================
+
+(defn- schema-source [store]
+ (:schema-source @store))
+
+(defn- assets-source [store]
+ (:assets-source @store))
+
+(defn cached-entity
+ "Look up a cached entity by kind and ref. Returns nil if not cached."
+ [store kind ref]
+ (get-in @store [:entities kind ref]))
+
+(defn- cache-entity! [store kind ref data]
+ (swap! store assoc-in [:entities kind ref] data)
+ data)
+
+;; --- Schema entities ---
+
+(defn load-database!
+ "Load and cache a database, assigning it an integer ID. Returns data or nil."
+ [store db-name]
+ (or (cached-entity store :database db-name)
+ (when-let [data (source/resolve-database (schema-source store) db-name)]
+ (let [id (get-or-assign! store :database db-name)]
+ (cache-entity! store :database db-name (assoc data :id id))))))
+
+(defn ensure-table-id!
+ "Ensure a table path has an assigned integer ID and minimal metadata cached.
+ Does NOT parse the table YAML — derives name/schema from the path.
+ Use [[load-table!]] when full table data is needed."
+ [store [db-name schema table-name :as table-path]]
+ (or (cached-entity store :table table-path)
+ (let [id (get-or-assign! store :table table-path)
+ db-id (get-or-assign! store :database db-name)]
+ (cache-entity! store :table table-path
+ {:id id
+ :db_id db-id
+ :name table-name
+ :schema schema}))))
+
+(defn load-table!
+ "Load and cache a table, assigning it and its database integer IDs.
+ Parses the table YAML for full metadata."
+ [store table-path]
+ (let [cached (cached-entity store :table table-path)]
+ (if (and cached (contains? cached :display_name))
+ ;; Already fully loaded
+ cached
+ ;; Load from YAML, merging with any existing minimal cache
+ (when-let [data (source/resolve-table (schema-source store) table-path)]
+ (let [[db-name _ _] table-path
+ id (get-or-assign! store :table table-path)
+ db-id (get-or-assign! store :database db-name)]
+ (cache-entity! store :table table-path (assoc data :id id :db_id db-id)))))))
+
+(defn load-field!
+ "Load and cache a field, assigning it and its table integer IDs."
+ [store field-path]
+ (or (cached-entity store :field field-path)
+ (when-let [data (source/resolve-field (schema-source store) field-path)]
+ (let [[db-name schema table-name _] field-path
+ id (get-or-assign! store :field field-path)
+ table-id (get-or-assign! store :table [db-name schema table-name])]
+ (cache-entity! store :field field-path (assoc data :id id :table_id table-id))))))
+
+;; --- Asset entities ---
+
+(defn load-card!
+ "Load and cache a card, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :card entity-id)
+ (when-let [data (source/resolve-card (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :card entity-id)]
+ (cache-entity! store :card entity-id (assoc data :id id))))))
+
+(defn load-snippet!
+ "Load and cache a snippet, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :snippet entity-id)
+ (when-let [data (source/resolve-snippet (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :snippet entity-id)]
+ (cache-entity! store :snippet entity-id (assoc data :id id))))))
+
+(defn load-transform!
+ "Load and cache a transform, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :transform entity-id)
+ (when-let [data (source/resolve-transform (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :transform entity-id)]
+ (cache-entity! store :transform entity-id (assoc data :id id))))))
+
+(defn load-segment!
+ "Load and cache a segment, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :segment entity-id)
+ (when-let [data (source/resolve-segment (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :segment entity-id)]
+ (cache-entity! store :segment entity-id (assoc data :id id))))))
+
+(defn load-dashboard!
+ "Load and cache a dashboard, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :dashboard entity-id)
+ (when-let [data (source/resolve-dashboard (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :dashboard entity-id)]
+ (cache-entity! store :dashboard entity-id (assoc data :id id))))))
+
+(defn load-collection!
+ "Load and cache a collection, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :collection entity-id)
+ (when-let [data (source/resolve-collection (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :collection entity-id)]
+ (cache-entity! store :collection entity-id (assoc data :id id))))))
+
+(defn load-document!
+ "Load and cache a document, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :document entity-id)
+ (when-let [data (source/resolve-document (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :document entity-id)]
+ (cache-entity! store :document entity-id (assoc data :id id))))))
+
+(defn load-measure!
+ "Load and cache a measure, assigning it an integer ID."
+ [store entity-id]
+ (or (cached-entity store :measure entity-id)
+ (when-let [data (source/resolve-measure (assets-source store) entity-id)]
+ (let [id (get-or-assign! store :measure entity-id)]
+ (cache-entity! store :measure entity-id (assoc data :id id))))))
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj b/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj
index 7af468f8f83..2b6656a89f9 100644
--- a/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj
+++ b/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj
@@ -26,19 +26,18 @@
(deps.native/validate-native-query driver query)
(into #{} (remove :soft?) (lib/find-bad-refs-with-source query))))
-(defmulti check-entity
- "Given a metadata provider, an entity type and an entity id, find any bad refs in that entity."
+(defmulti ^:private -check-entity
+ "Implementation multimethod for [[check-entity]]. Extend this to add support
+ for new entity types."
{:arglists '([metadata-provider entity-type entity-id])}
(fn [_metadata-provider entity-type _entity-id]
entity-type))
-(defmethod check-entity :default
+(defmethod -check-entity :default
[_metadata-provider _entity-type _entity-id]
nil)
-(mu/defmethod check-entity :card :- [:set [:ref ::lib.schema.validate/error-with-source]]
- "Given a `MetadataProvider` and a card ID, analyses the card's query to find any bad refs or other issues.
- Returns any findings with source information, and `nil` for a clean query."
+(mu/defmethod -check-entity :card :- [:set [:ref ::lib.schema.validate/error-with-source]]
[metadata-provider :- ::lib.schema.metadata/metadata-provider
_entity-type
card-id :- ::lib.schema.id/card]
@@ -46,10 +45,7 @@
driver (:engine (lib.metadata/database query))]
(check-query driver query)))
-(mu/defmethod check-entity :transform :- [:set [:ref ::lib.schema.validate/error-with-source]]
- "Given a `MetadataProvider` and a transform ID, analyses the transform's query to find any bad refs or other issues.
-
- Returns any findings with source information, and `nil` for a clean transform."
+(mu/defmethod -check-entity :transform :- [:set [:ref ::lib.schema.validate/error-with-source]]
[metadata-provider :- ::lib.schema.metadata/metadata-provider
_entity-type
transform-id :- ::lib.schema.id/transform]
@@ -67,7 +63,7 @@
(cond-> (check-query driver query)
duplicated-fields (into duplicated-fields))))
-(mu/defmethod check-entity :segment :- [:set [:ref ::lib.schema.validate/error-with-source]]
+(mu/defmethod -check-entity :segment :- [:set [:ref ::lib.schema.validate/error-with-source]]
[metadata-provider :- ::lib.schema.metadata/metadata-provider
_entity-type
segment-id :- ::lib.schema.id/segment]
@@ -76,3 +72,14 @@
(lib/query metadata-provider))
driver (:engine (lib.metadata/database query))]
(check-query driver query)))
+
+(defn check-entity
+ "Check a single entity for bad refs against a MetadataProvider.
+
+ Returns a set of errors with source information, or nil if the entity is clean.
+ Dispatches to entity-type-specific analysis (MBQL bad refs, native SQL validation,
+ transform duplicate columns, etc.).
+
+ Supported entity types: :card, :transform, :segment. Other types return nil."
+ [metadata-provider entity-type entity-id]
+ (-check-entity metadata-provider entity-type entity-id))
diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj b/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj
index d440a2d1b1d..231bc3588a7 100644
--- a/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj
+++ b/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj
@@ -26,10 +26,11 @@
compiled (qp.compile/compile-with-inline-parameters with-params)]
(lib/native-query with-params (:query compiled))))
-(defn- has-card-template-tags?
- "Returns true if the query has any card-type template tags."
+(defn- has-substitutable-template-tags?
+ "Returns true if the query has any card or table template tags that need
+ placeholder substitution before compilation."
[query]
- (some #(= (:type %) :card) (lib/all-template-tags query)))
+ (some #(#{:card :table} (:type %)) (lib/all-template-tags query)))
(def ^:private card-placeholder-prefix
"Prefix used to generate placeholder table names for card references.
@@ -44,9 +45,15 @@
(when (str/starts-with? lower card-placeholder-prefix)
(parse-long (subs lower (count card-placeholder-prefix))))))
+(def ^:private table-placeholder-prefix
+ "Prefix used to generate placeholder table names for table-type template tags.
+ Table tag {{table_one}} becomes mb__validat_table__N in the SQL sent to SQLGlot."
+ "mb__validat_table__")
+
(defn- compile-toplevel-query
- "Compile a query replacing card references with placeholder table names.
+ "Compile a query replacing card and table references with placeholder table names.
Card refs like {{#1}} are replaced with mb__validat_card__1.
+ Table refs like {{table_one}} are replaced with mb__validat_table__N.
Returns the compiled native-only-query, or nil if the original SQL contains
the placeholder prefix (collision guard)."
@@ -54,9 +61,12 @@
(let [stage (first (:stages query))
sql (:native stage)
ttags (:template-tags stage)
- card-tags (into {} (filter #(= (:type (val %)) :card)) ttags)]
- (when-not (str/includes? sql card-placeholder-prefix)
- (let [modified-sql
+ card-tags (into {} (filter #(= (:type (val %)) :card)) ttags)
+ table-tags (into {} (filter #(= (:type (val %)) :table)) ttags)]
+ (when-not (or (str/includes? sql card-placeholder-prefix)
+ (str/includes? sql table-placeholder-prefix))
+ (let [;; Replace card tags
+ sql-after-cards
(reduce (fn [s [tag-name tag]]
(str/replace s
(str "{{" tag-name "}}")
@@ -64,24 +74,40 @@
sql
card-tags)
- non-card-tags
- (into {} (remove #(= (:type (val %)) :card)) ttags)
+ ;; Replace table tags
+ modified-sql
+ (reduce (fn [s [tag-name tag]]
+ (str/replace s
+ (str "{{" tag-name "}}")
+ (str table-placeholder-prefix (:table-id tag))))
+ sql-after-cards
+ table-tags)
+
+ remaining-tags
+ (into {} (remove #(#{:card :table} (:type (val %)))) ttags)
modified-query
(-> query
(assoc-in [:stages 0 :native] modified-sql)
- (assoc-in [:stages 0 :template-tags] non-card-tags))]
+ (assoc-in [:stages 0 :template-tags] remaining-tags))]
(compile-query modified-query)))))
+(defn- table-placeholder?
+ "True if table-name is a table placeholder (e.g. \"mb__validat_table__65478\")."
+ [table-name]
+ (when table-name
+ (str/starts-with? (u/lower-case-en (str table-name)) table-placeholder-prefix)))
+
(defn- table-deps
"Returns the set of table dep maps (e.g. {:table \"products\" :schema \"public\"})
referenced directly in the compiled native query.
- Excludes card placeholder tables (mb__validat_card__N)."
+ Excludes card and table placeholder tables."
[driver compiled]
(into #{}
(filter (fn [dep]
(and (:table dep)
- (not (parse-card-placeholder (:table dep))))))
+ (not (parse-card-placeholder (:table dep)))
+ (not (table-placeholder? (:table dep))))))
(driver/native-query-deps driver compiled)))
(defn- extract-source-entity
@@ -237,10 +263,13 @@
[driver :- :keyword
query :- ::lib.schema/query]
(into #{}
- ;; Strip :unknown source attribution — consumers should not distinguish between
- ;; "we tried and couldn't determine the source" vs "no source info available".
- (map #(cond-> % (= :unknown (:source-entity-type %)) (dissoc :source-entity-type :source-entity-id)))
- (if (has-card-template-tags? query)
+ (comp
+ ;; Strip :unknown source attribution — consumers should not distinguish between
+ ;; "we tried and couldn't determine the source" vs "no source info available".
+ (map #(cond-> % (= :unknown (:source-entity-type %)) (dissoc :source-entity-type :source-entity-id)))
+ ;; Remove errors referencing table placeholder names
+ (remove #(some-> (:name %) (str/starts-with? table-placeholder-prefix))))
+ (if (has-substitutable-template-tags? query)
(if-let [compiled (compile-toplevel-query query)]
(let [errors (validate-with-sources driver compiled true)]
(if (empty? errors)
@@ -254,13 +283,22 @@
errors
(fallback-enrich driver compiled errors))))))
+(defn- has-table-template-tags?
+ "Returns true if the query has any table-type template tags."
+ [query]
+ (some #(= (:type %) :table) (lib/all-template-tags query)))
+
(mu/defn native-result-metadata
- "Compiles a (native) query and calculates its result metadata"
+ "Compiles a (native) query and calculates its result metadata.
+ Returns nil for queries with table-type template tags since the actual
+ table (and therefore columns) is unknown at check time."
[driver :- :keyword
query :- ::lib.schema/query]
- (->> query
- compile-query
- (driver/native-result-metadata driver)))
+ (when-not (has-table-template-tags? query)
+ (let [compiled (if (has-substitutable-template-tags? query)
+ (or (compile-toplevel-query query) (compile-query query))
+ (compile-query query))]
+ (driver/native-result-metadata driver compiled))))
(mu/defn native-query-deps :- [:set
[:or
diff --git a/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj b/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj
index 9f3615724ed..a11a9713aa6 100644
--- a/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj
+++ b/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj
@@ -144,6 +144,9 @@
(log/info (format "Export to '%s' complete!" path) (u/emoji "🚛💨 📦"))
report))
+(comment
+ (v2-dump! "/tmp/serdes" {}))
+
(defn seed-entity-ids!
"Add entity IDs for instances of serializable models that don't already have them.
diff --git a/enterprise/backend/test/metabase_enterprise/checker/e2e_test.clj b/enterprise/backend/test/metabase_enterprise/checker/e2e_test.clj
new file mode 100644
index 00000000000..880e5ba4fe9
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/checker/e2e_test.clj
@@ -0,0 +1,137 @@
+(ns metabase-enterprise.checker.e2e-test
+ "End-to-end test running the checker against the serialization baseline export.
+
+ This exercises the full pipeline: YAML parsing → index building → store →
+ provider → query validation → structural checks → results.
+
+ The baseline at test_resources/serialization_baseline is a real serdes export
+ with cards, dashboards, transforms, snippets, and documents backed by
+ the sample database schema."
+ (:require
+ [clojure.string :as str]
+ [clojure.test :refer :all]
+ [clojure.walk :as walk]
+ [metabase-enterprise.checker.semantic :as checker]))
+
+(set! *warn-on-reflection* true)
+
+(def ^:private export-dir "test_resources/serialization_baseline")
+(def ^:private schema-dir "test_resources/serialization_baseline/databases")
+
+(def ^:private baseline-results
+ "Checker results for the baseline export. Computed once, shared across all tests."
+ (delay (:results (checker/check export-dir schema-dir))))
+
+(defn- failures-report
+ "Format failing results for assertion messages."
+ [results status]
+ (->> results
+ (filter #(= status (checker/result-status (second %))))
+ (map checker/format-error)
+ (str/join "\n")))
+
+(deftest baseline-export-no-errors-test
+ (testing "The serialization baseline export passes all checks with zero errors"
+ (let [results @baseline-results
+ summary (checker/summarize-results results)]
+ (is (=? {:ok pos-int?
+ :errors 0
+ :unresolved 0
+ :native-errors 0
+ :issues 0}
+ summary))
+ (is (> (:total summary) 45)
+ "Should check cards plus non-card entities (dashboards, transforms, etc.)")
+ (is (pos? (:total summary))
+ "Should check at least one entity")
+ (is (zero? (:errors summary))
+ (str "Errors:\n" (failures-report results :error)))
+ (is (zero? (:unresolved summary))
+ (str "Unresolved:\n" (failures-report results :unresolved)))
+ (is (zero? (:native-errors summary))
+ (str "Native SQL errors:\n" (failures-report results :native-errors)))
+ (is (zero? (:issues summary))
+ (str "Issues:\n" (failures-report results :issues))))))
+
+(deftest baseline-export-card-refs-populated-test
+ (testing "Checked cards have populated ref data (tables, fields)"
+ (let [card-results (->> @baseline-results vals (filter :refs))]
+ (is (pos? (count card-results))
+ "At least some cards should have refs")
+ (is (some #(seq (get-in % [:refs :tables])) card-results)
+ "At least one card should reference tables")
+ (is (some #(seq (get-in % [:refs :fields])) card-results)
+ "At least one card should reference fields"))))
+
+;;; ===========================================================================
+;;; Card shape assertions — verify detailed analysis on specific cards
+;;; ===========================================================================
+
+(defn- normalize [x] (walk/postwalk x (cond-> x (sequential? x) set)))
+
+(deftest joins-card-shape-test
+ (testing "MBQL card with 3 joins resolves all tables and fields"
+ (let [result (get @baseline-results "wpW9JO5MsHQd0-7WrR7gN")]
+ (is (=? (normalize
+ {:card-id pos-int?
+ :name "Joins - Strategies and Compound Conditions",
+ :entity-id "wpW9JO5MsHQd0-7WrR7gN",
+ :refs {:tables ["Sample Database.PUBLIC.PRODUCTS"
+ "Sample Database.PUBLIC.REVIEWS"
+ "Sample Database.PUBLIC.ORDERS"
+ "Sample Database.PUBLIC.PEOPLE"],
+ :fields ["Sample Database.PUBLIC.ORDERS.ID"
+ "Sample Database.PUBLIC.ORDERS.USER_ID"
+ "Sample Database.PUBLIC.ORDERS.PRODUCT_ID"
+ "Sample Database.PUBLIC.ORDERS.SUBTOTAL"
+ "Sample Database.PUBLIC.ORDERS.TAX"
+ "Sample Database.PUBLIC.ORDERS.TOTAL"
+ "Sample Database.PUBLIC.ORDERS.DISCOUNT"
+ "Sample Database.PUBLIC.ORDERS.CREATED_AT"
+ "Sample Database.PUBLIC.ORDERS.QUANTITY"
+ "Sample Database.PUBLIC.PRODUCTS.ID"
+ "Sample Database.PUBLIC.PRODUCTS.EAN"
+ "Sample Database.PUBLIC.PRODUCTS.TITLE"
+ "Sample Database.PUBLIC.PRODUCTS.CATEGORY"
+ "Sample Database.PUBLIC.PRODUCTS.VENDOR"
+ "Sample Database.PUBLIC.PRODUCTS.PRICE"
+ "Sample Database.PUBLIC.PRODUCTS.RATING"
+ "Sample Database.PUBLIC.PRODUCTS.CREATED_AT"
+ "Sample Database.PUBLIC.PEOPLE.NAME"
+ "Sample Database.PUBLIC.PEOPLE.STATE"]}})
+ (normalize result)))
+ ;; REVIEWS fields are NOT in the list because the join has `fields: none`
+ ;; — only join condition fields from REVIEWS are used, not selected.
+ ;; The REVIEWS table IS resolved though (checked above).
+ (is (-> result :source-cards empty?)))))
+
+;;;; note: these are all sets here for equality's sake. But they traditionally are vectors
+
+(deftest measure-metric-segment-refs-test
+ (testing "Card with measure, metric, and segment references tracks them all"
+ (let [result (get @baseline-results "kBjQ5VXJ5z3vYSW72J6qa")]
+ (is (=? {:card-id pos-int?,
+ :name "Metric, Measure, and Segment References",
+ :entity-id "kBjQ5VXJ5z3vYSW72J6qa",
+ :refs {:tables ["Sample Database.PUBLIC.ORDERS"],
+ :source-cards ["Total Revenue"],
+ :measures ["DBap2523KuN4Lt-cYrUjF"],
+ :metrics ["IW8kbqZVaMxdGtCM2F4U6"],
+ :segments ["OaFXAvEzPvu9ZXW5PUMRa"]}}
+ result)))))
+
+(deftest transitive-entity-refs-test
+ (testing "Source card reference transitively surfaces the inner card's snippets, measures, segments"
+ (let [;; "Native Query - Card and Snippet References" sources from
+ ;; "Basic Aggregations" AND uses a snippet.
+ snippet-card (get @baseline-results "HiBFSt0BNx5s5MxVDLLKB")]
+ ;; The snippet card itself uses a snippet and sources from Basic Aggregations.
+ ;; Basic Aggregations references ORDERS table — that should appear transitively.
+ (is (=? {:card-id pos-int?
+ :name "Native Query - Card and Snippet References",
+ :entity-id "HiBFSt0BNx5s5MxVDLLKB",
+ :refs {:source-cards ["Basic Aggregations"],
+ :tables ["Sample Database.PUBLIC.ORDERS"],
+ :fields ["Sample Database.PUBLIC.ORDERS.CREATED_AT"],
+ :snippets ["Active Product Filter"]}}
+ snippet-card)))))
diff --git a/enterprise/backend/test/metabase_enterprise/checker/format/serdes_test.clj b/enterprise/backend/test/metabase_enterprise/checker/format/serdes_test.clj
new file mode 100644
index 00000000000..36a5de28947
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/checker/format/serdes_test.clj
@@ -0,0 +1,396 @@
+(ns metabase-enterprise.checker.format.serdes-test
+ "Tests for the serdes format module — YAML extraction, file indexing, and source resolution."
+ (:require
+ [clojure.java.io :as io]
+ [clojure.test :refer [deftest is testing]]
+ [metabase-enterprise.checker.format.serdes :as serdes]
+ [metabase-enterprise.checker.source :as source]
+ [metabase.util.yaml :as yaml]))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; Helpers
+;;; ===========================================================================
+
+(def ^:private fixtures-dir "test_resources/yaml_checks")
+
+(defn- fixtures-path []
+ (let [f (io/file fixtures-dir)]
+ (if (.isAbsolute f)
+ fixtures-dir
+ (.getPath (io/file (System/getProperty "user.dir") fixtures-dir)))))
+
+(defn- with-temp-dir
+ "Create a temp directory, call (f dir-path), then clean up."
+ [f]
+ (let [dir (java.io.File/createTempFile "serdes-test" "")]
+ (.delete dir)
+ (.mkdirs dir)
+ (try
+ (f (.getPath dir))
+ (finally
+ (doseq [fl (reverse (file-seq dir))]
+ (.delete ^java.io.File fl))))))
+
+(defn- write-yaml!
+ "Write a YAML file, creating parent directories as needed."
+ [dir & path-and-data]
+ (let [data (last path-and-data)
+ path-segs (butlast path-and-data)
+ ^java.io.File file (apply io/file dir (map str path-segs))]
+ (.mkdirs (.getParentFile file))
+ (spit file (yaml/generate-string data))
+ (.getPath file)))
+
+;;; ===========================================================================
+;;; YAML extraction tests (pure, no files needed)
+;;; ===========================================================================
+
+(deftest extract-entity-id-from-fixture-test
+ (testing "extract-entity-id reads entity_id from a card YAML"
+ (let [card-file (io/file (fixtures-path) "collections" "cards" "simple-orders_simple_orders.yaml")]
+ (is (= "simple-orders" (serdes/extract-entity-id (.getPath card-file)))))))
+
+(deftest extract-model-from-fixture-test
+ (testing "extract-model reads the serdes model from a card YAML"
+ (let [card-file (io/file (fixtures-path) "collections" "cards" "simple-orders_simple_orders.yaml")]
+ (is (= "Card" (serdes/extract-model (.getPath card-file)))))))
+
+(deftest extract-entity-id-from-temp-file-test
+ (testing "extract-entity-id with various formats"
+ (with-temp-dir
+ (fn [dir]
+ (let [f (write-yaml! dir "test.yaml" {:name "X" :entity_id "xYz123AbC"})]
+ (is (= "xYz123AbC" (serdes/extract-entity-id f))))))))
+
+;;; ===========================================================================
+;;; Index tests — databases are indexed, tables/fields are on-demand
+;;; ===========================================================================
+
+(deftest build-file-index-databases-test
+ (testing "build-file-index indexes databases from fixtures"
+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]
+ (is (contains? (:database index) "Test Database"))
+ (is (contains? (:database index) "SQLite DB")))))
+
+(deftest build-file-index-cards-test
+ (testing "build-file-index indexes cards by entity_id"
+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]
+ (is (contains? (:card index) "simple-orders"))
+ (is (contains? (:card index) "native-orders"))
+ (is (contains? (:card index) "orders-with-products")))))
+
+(deftest build-file-index-no-duplicates-test
+ (testing "fixture index has no duplicates"
+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]
+ (is (nil? (:duplicates index))))))
+
+(deftest build-database-dir-index-test
+ (testing "build-database-dir-index indexes databases and provides db-name->dir mapping"
+ (let [db-dir (str (fixtures-path) "/databases")
+ {:keys [index db-name->dir]} (serdes/build-database-dir-index db-dir)]
+ (is (contains? (:database index) "Test Database"))
+ (is (contains? (:database index) "SQLite DB"))
+ (is (contains? db-name->dir "Test Database"))
+ (is (contains? db-name->dir "SQLite DB"))
+ ;; Tables are NOT in the index — they're resolved on demand
+ (is (nil? (:table index)))
+ ;; Should not contain cards (databases dir has no collections)
+ (is (nil? (:card index))))))
+
+;;; ===========================================================================
+;;; db-name->dir mapping tests
+;;; ===========================================================================
+
+(deftest db-name-to-dir-mapping-test
+ (testing "db-name->dir maps real database names to directory names"
+ (with-temp-dir
+ (fn [dir]
+ ;; Create a database with a slugified directory name but real name in YAML
+ (write-yaml! dir "databases" "my_fancy_db" "my_fancy_db.yaml"
+ {:name "My Fancy DB" :engine "h2"})
+ (let [{:keys [db-name->dir]} (serdes/build-database-dir-index (str dir "/databases"))]
+ (is (= "my_fancy_db" (get db-name->dir "My Fancy DB")))))))
+
+ (testing "db-name->dir includes ALL databases, not just some"
+ (with-temp-dir
+ (fn [dir]
+ (doseq [i (range 10)]
+ (let [slug (str "db_" i)]
+ (write-yaml! dir "databases" slug (str slug ".yaml")
+ {:name (str "Database " i) :engine "h2"})))
+ (let [{:keys [db-name->dir]} (serdes/build-database-dir-index (str dir "/databases"))]
+ (is (= 10 (count db-name->dir)))
+ (doseq [i (range 10)]
+ (is (= (str "db_" i) (get db-name->dir (str "Database " i))))))))))
+
+;;; ===========================================================================
+;;; SchemaSource protocol — on-demand resolution tests
+;;; ===========================================================================
+
+(deftest source-resolves-database-test
+ (testing "SerdesSource resolves databases by name"
+ (let [source (serdes/make-source (fixtures-path))
+ db (source/resolve-database source "Test Database")]
+ (is (some? db))
+ (is (= "Test Database" (:name db))))))
+
+(deftest source-resolves-table-on-demand-test
+ (testing "SerdesSource resolves tables by path on demand (no pre-indexing)"
+ (let [source (serdes/make-source (fixtures-path))
+ table (source/resolve-table source ["Test Database" "public" "orders"])]
+ (is (some? table))
+ (is (= "orders" (:name table))))
+ (testing "schema-less tables"
+ (let [source (serdes/make-source (fixtures-path))
+ table (source/resolve-table source ["SQLite DB" nil "orders"])]
+ (is (some? table))
+ (is (= "orders" (:name table)))))))
+
+(deftest source-resolves-field-on-demand-test
+ (testing "SerdesSource resolves fields by path on demand"
+ (let [source (serdes/make-source (fixtures-path))
+ field (source/resolve-field source ["Test Database" "public" "orders" "id"])]
+ (is (some? field))
+ (is (= "id" (:name field))))
+ (testing "schema-less fields"
+ (let [source (serdes/make-source (fixtures-path))
+ field (source/resolve-field source ["SQLite DB" nil "orders" "id"])]
+ (is (some? field))
+ (is (= "id" (:name field)))))))
+
+(deftest source-resolves-card-test
+ (testing "SerdesSource resolves cards by entity-id"
+ (let [source (serdes/make-source (fixtures-path))
+ card (source/resolve-card source "simple-orders")]
+ (is (some? card))
+ (is (= "Simple Orders" (:name card))))))
+
+(deftest source-returns-nil-for-unknown-test
+ (testing "SerdesSource returns nil for unknown entities"
+ (let [source (serdes/make-source (fixtures-path))]
+ (is (nil? (source/resolve-database source "Nonexistent")))
+ (is (nil? (source/resolve-table source ["DB" "x" "y"])))
+ (is (nil? (source/resolve-field source ["DB" "x" "y" "z"])))
+ (is (nil? (source/resolve-card source "no-such-card"))))))
+
+;;; ===========================================================================
+;;; fields-for-table and all-table-paths — on-demand enumeration
+;;; ===========================================================================
+
+(deftest fields-for-table-test
+ (testing "fields-for-table returns field paths for a table"
+ (let [source (serdes/make-source (fixtures-path))
+ fields (source/fields-for-table source ["Test Database" "public" "orders"])]
+ (is (set? fields))
+ (is (contains? fields ["Test Database" "public" "orders" "id"]))
+ (is (contains? fields ["Test Database" "public" "orders" "total"]))))
+ (testing "fields-for-table works for schema-less tables"
+ (let [source (serdes/make-source (fixtures-path))
+ fields (source/fields-for-table source ["SQLite DB" nil "orders"])]
+ (is (set? fields))
+ (is (contains? fields ["SQLite DB" nil "orders" "id"]))))
+ (testing "fields-for-table returns nil for unknown table"
+ (let [source (serdes/make-source (fixtures-path))]
+ (is (nil? (source/fields-for-table source ["Nope" "x" "y"]))))))
+
+(deftest all-table-paths-test
+ (testing "all-table-paths enumerates all tables across databases"
+ (let [source (serdes/make-source (fixtures-path))
+ tables (source/all-table-paths source)]
+ (is (seq tables))
+ (is (some #(= ["Test Database" "public" "orders"] %) tables))
+ (is (some #(= ["SQLite DB" nil "orders"] %) tables)))))
+
+(deftest all-database-names-test
+ (testing "all-database-names returns database names"
+ (let [source (serdes/make-source (fixtures-path))]
+ (is (= #{"Test Database" "SQLite DB"} (set (source/all-database-names source)))))))
+
+;;; ===========================================================================
+;;; Slugified directory names — critical correctness tests
+;;; ===========================================================================
+
+(deftest slugified-db-dir-resolves-correctly-test
+ (testing "databases with slugified directory names resolve correctly"
+ (with-temp-dir
+ (fn [dir]
+ (let [db-dir (str dir "/databases")]
+ ;; Create a database with slugified dir but real name in YAML
+ (write-yaml! dir "databases" "analytics_data_warehouse" "analytics_data_warehouse.yaml"
+ {:name "Analytics Data Warehouse" :engine "postgres"})
+ ;; Create a table under it
+ (write-yaml! dir "databases" "analytics_data_warehouse" "schemas" "public" "tables" "orders" "orders.yaml"
+ {:name "orders" :schema "public"
+ :serdes/meta [{:id "Analytics Data Warehouse" :model "Database"}
+ {:id "public" :model "Schema"}
+ {:id "orders" :model "Table"}]})
+ ;; Create a field
+ (write-yaml! dir "databases" "analytics_data_warehouse" "schemas" "public" "tables" "orders" "fields" "id.yaml"
+ {:name "id" :base_type "type/Integer"
+ :serdes/meta [{:id "Analytics Data Warehouse" :model "Database"}
+ {:id "public" :model "Schema"}
+ {:id "orders" :model "Table"}
+ {:id "id" :model "Field"}]})
+ (let [source (serdes/make-database-source db-dir)]
+ ;; Database resolves by real name
+ (is (some? (source/resolve-database source "Analytics Data Warehouse")))
+ ;; Table resolves using real db name
+ (is (some? (source/resolve-table source ["Analytics Data Warehouse" "public" "orders"])))
+ ;; Field resolves using real db name
+ (let [field (source/resolve-field source ["Analytics Data Warehouse" "public" "orders" "id"])]
+ (is (some? field))
+ (is (= "id" (:name field))))
+ ;; fields-for-table works
+ (let [fields (source/fields-for-table source ["Analytics Data Warehouse" "public" "orders"])]
+ (is (contains? fields ["Analytics Data Warehouse" "public" "orders" "id"])))))))))
+
+;;; ===========================================================================
+;;; Performance guards — indexing should NOT parse field/table YAMLs
+;;; ===========================================================================
+
+(deftest index-does-not-contain-tables-or-fields-test
+ (testing "the file index only contains databases — tables and fields are resolved on demand"
+ (let [db-dir (str (fixtures-path) "/databases")
+ {:keys [index]} (serdes/build-database-dir-index db-dir)]
+ (is (seq (:database index)) "databases should be in the index")
+ (is (nil? (:table index)) "tables should NOT be in the index")
+ (is (nil? (:field index)) "fields should NOT be in the index"))))
+
+;;; ===========================================================================
+;;; Cache shape and lazy resolution
+;;; ===========================================================================
+
+(def ^:private not-indexed :metabase-enterprise.checker.format.serdes/not-indexed)
+
+(deftest schema-model-shape-test
+ (testing "schema model starts with ::not-indexed for each schema, populates lazily"
+ (with-temp-dir
+ (fn [dir]
+ (write-yaml! dir "databases" "db" "db.yaml" {:name "DB" :engine "h2"})
+ (write-yaml! dir "databases" "db" "schemas" "s1" "tables" "t1" "t1.yaml"
+ {:name "T1" :schema "s1"})
+ (write-yaml! dir "databases" "db" "schemas" "s1" "tables" "t1" "fields" "f1.yaml"
+ {:name "F1" :base_type "type/Integer"})
+ (write-yaml! dir "databases" "db" "schemas" "s2" "tables" "t2" "t2.yaml"
+ {:name "T2" :schema "s2"})
+ (write-yaml! dir "databases" "db" "schemas" "s2" "tables" "t2" "fields" "f2.yaml"
+ {:name "F2" :base_type "type/Text"})
+ (let [source (serdes/make-database-source (str dir "/databases"))]
+ ;; Initial: db-file set, all schemas ::not-indexed
+ (is (=? {"DB" {:db-file string?
+ :schemas {"s1" not-indexed
+ "s2" not-indexed}}}
+ @(.-schema-model source)))
+ ;; After resolving a table in s1: s1 indexed with tables, s2 untouched
+ (source/resolve-table source ["DB" "s1" "T1"])
+ (is (=? {"DB" {:schemas {"s1" {"T1" {:table-file string?
+ :fields not-indexed}}
+ "s2" not-indexed}}}
+ @(.-schema-model source)))
+ ;; After resolving fields for T1: T1 fields indexed, s2 still untouched
+ (source/fields-for-table source ["DB" "s1" "T1"])
+ (is (=? {"DB" {:schemas {"s1" {"T1" {:table-file string?
+ :fields {"F1" string?}}}
+ "s2" not-indexed}}}
+ @(.-schema-model source))))))))
+
+(deftest case-insensitive-schema-resolution-test
+ (testing "schemas with different casing on disk vs YAML resolve correctly"
+ (with-temp-dir
+ (fn [dir]
+ ;; Dir name is lowercase, YAML schema is uppercase (H2 convention)
+ (write-yaml! dir "databases" "db" "db.yaml" {:name "DB" :engine "h2"})
+ (write-yaml! dir "databases" "db" "schemas" "public" "tables" "orders" "orders.yaml"
+ {:name "ORDERS" :schema "PUBLIC"})
+ (write-yaml! dir "databases" "db" "schemas" "public" "tables" "orders" "fields" "id.yaml"
+ {:name "ID" :base_type "type/Integer"})
+ (let [source (serdes/make-database-source (str dir "/databases"))]
+ ;; Resolve by real names (uppercase) — schema re-keys from "public" to "PUBLIC"
+ (is (some? (source/resolve-table source ["DB" "PUBLIC" "ORDERS"])))
+ (is (some? (source/resolve-field source ["DB" "PUBLIC" "ORDERS" "ID"])))
+ (is (=? {"DB" {:schemas {"PUBLIC" {"ORDERS" {:fields {"ID" string?}}}}}}
+ @(.-schema-model source))))))))
+
+;;; ===========================================================================
+;;; Segments and measures — indexed from export dir, NOT schema dir
+;;; ===========================================================================
+
+(deftest export-dir-indexes-segments-and-measures-test
+ (testing "build-file-index (export dir) finds segments and measures under databases/"
+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]
+ (is (contains? (:segment index) "big-orders-segment")
+ "segment from with-schema table should be indexed")
+ (is (contains? (:segment index) "recent-orders-segment")
+ "segment from schema-less table should be indexed")
+ (is (contains? (:measure index) "total-revenue-measure")
+ "measure should be indexed"))))
+
+(deftest schema-dir-does-not-index-segments-test
+ (testing "build-database-dir-index (schema dir) does NOT index segments or measures"
+ (let [db-dir (str (fixtures-path) "/databases")
+ {:keys [index]} (serdes/build-database-dir-index db-dir)]
+ (is (nil? (:segment index)) "schema dir should not contain segments")
+ (is (nil? (:measure index)) "schema dir should not contain measures"))))
+
+(deftest segments-resolve-from-export-source-test
+ (testing "segments can be resolved from the export source"
+ (let [source (serdes/make-source (fixtures-path))]
+ (let [seg (source/resolve-segment source "big-orders-segment")]
+ (is (some? seg))
+ (is (= "Big Orders" (:name seg))))
+ (let [seg (source/resolve-segment source "recent-orders-segment")]
+ (is (some? seg))
+ (is (= "Recent Orders" (:name seg)))))))
+
+(deftest measures-resolve-from-export-source-test
+ (testing "measures can be resolved from the export source"
+ (let [source (serdes/make-source (fixtures-path))
+ m (source/resolve-measure source "total-revenue-measure")]
+ (is (some? m))
+ (is (= "Total Revenue" (:name m))))))
+
+;;; ===========================================================================
+;;; Temp file edge cases
+;;; ===========================================================================
+
+(deftest build-file-index-empty-dir-test
+ (testing "build-file-index on empty directory returns empty index"
+ (with-temp-dir
+ (fn [dir]
+ (let [{:keys [index]} (serdes/build-file-index dir)]
+ (is (nil? (:database index)))
+ (is (nil? (:card index))))))))
+
+(def ^:private card-yaml-template
+ "name: %s\nentity_id: %s\nserdes/meta:\n- id: %s\n model: Card\n")
+
+(deftest build-file-index-detects-duplicates-test
+ (testing "build-file-index reports duplicate entity_ids"
+ (with-temp-dir
+ (fn [dir]
+ ;; Create two cards with the same entity_id in different directories
+ (let [^java.io.File f1 (io/file dir "collections" "a" "cards" "dup.yaml")
+ ^java.io.File f2 (io/file dir "collections" "b" "cards" "dup.yaml")]
+ (.mkdirs (.getParentFile f1))
+ (.mkdirs (.getParentFile f2))
+ (spit f1 (format card-yaml-template "Card A" "dup-id" "dup-id"))
+ (spit f2 (format card-yaml-template "Card B" "dup-id" "dup-id"))
+ (let [{:keys [index]} (serdes/build-file-index dir)]
+ (is (seq (:duplicates index)) "Should detect duplicate entity_ids")
+ (is (= "dup-id" (:ref (first (:duplicates index)))))))))))
+
+(deftest source-index-accessor-test
+ (testing "source-index returns the assets index (no databases — those are in the schema model)"
+ (let [source (serdes/make-source (fixtures-path))
+ index (serdes/source-index source)]
+ (is (map? index))
+ (is (not (contains? index :database)) "databases are in the schema model, not the assets index")
+ (is (contains? index :card)))))
+
+(deftest all-card-ids-test
+ (testing "all-card-ids returns all card entity-ids"
+ (let [source (serdes/make-source (fixtures-path))]
+ (is (>= (count (serdes/all-card-ids source)) 5))
+ (is (contains? (set (serdes/all-card-ids source)) "simple-orders")))))
diff --git a/enterprise/backend/test/metabase_enterprise/checker/semantic_test.clj b/enterprise/backend/test/metabase_enterprise/checker/semantic_test.clj
new file mode 100644
index 00000000000..68064b532a1
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/checker/semantic_test.clj
@@ -0,0 +1,1215 @@
+(ns metabase-enterprise.checker.semantic-test
+ "Tests for the semantic checker.
+
+ Uses real YAML fixture files from test_resources/yaml_checks for most tests.
+ Uses in-memory sources for edge cases like unresolved refs."
+ (:require
+ [clojure.java.io :as io]
+ [clojure.test :refer [deftest is testing]]
+ [metabase-enterprise.checker.format.serdes :as serdes-format]
+ [metabase-enterprise.checker.provider :as provider]
+ [metabase-enterprise.checker.semantic :as checker]
+ [metabase-enterprise.checker.source :as source]
+ [metabase-enterprise.checker.store :as store]
+ [metabase-enterprise.checker.test-helpers :as helpers]
+ [metabase.lib.metadata.protocols :as lib.metadata.protocols]))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; Test Fixture Path
+;;; ===========================================================================
+
+(def ^:private test-fixtures-dir
+ "Path to test fixtures directory (real YAML files)."
+ "test_resources/yaml_checks")
+
+(defn- fixtures-path []
+ (let [f (io/file test-fixtures-dir)]
+ (if (.isAbsolute f)
+ test-fixtures-dir
+ (.getPath (io/file (System/getProperty "user.dir") test-fixtures-dir)))))
+
+(defn- make-test-source []
+ (serdes-format/make-source (fixtures-path)))
+
+(defn- check-all [source]
+ (checker/check-entities source source (serdes-format/source-index source)))
+
+(defn- check-specific [source entity-ids]
+ (checker/check-entities source source (serdes-format/source-index source) entity-ids))
+
+;;; ===========================================================================
+;;; Tests Using Real YAML Files
+;;; ===========================================================================
+
+(deftest source-index-test
+ (testing "Source correctly indexes databases and cards; tables/fields are on-demand"
+ (let [source (make-test-source)
+ index (serdes-format/source-index source)]
+ ;; Databases are in the schema model, not the assets index
+ (is (= #{"Test Database" "SQLite DB"} (set (source/all-database-names source))))
+ ;; Cards are in the assets index
+ (is (= 5 (count (:card index))) "Should have 5 cards")
+ ;; Tables and fields are resolved on demand
+ (is (= 4 (count (source/all-table-paths source))) "Should have 4 tables via source")
+ (is (= 11 (count (source/all-field-paths source))) "Should have 11 fields via source"))))
+
+(deftest simple-mbql-query-test
+ (testing "Simple MBQL query on orders table validates successfully"
+ (let [source (make-test-source)
+ results (check-all source)
+ result (get results "simple-orders")]
+ (is (some? result) "Card should be found")
+ (is (= "Simple Orders" (:name result)) "Card name should match")
+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))
+ (is (empty? (:unresolved result)) "Should not have unresolved refs")
+ (is (empty? (:bad-refs result)) "Should not have bad refs from lib")
+ ;; Check that table refs were extracted
+ (is (seq (get-in result [:refs :tables])) "Should have table refs"))))
+
+(deftest native-query-test
+ (testing "Native SQL query validates successfully"
+ (let [source (make-test-source)
+ results (check-all source)
+ result (get results "native-orders")]
+ (is (some? result) "Card should be found")
+ (is (= "Native Orders" (:name result)) "Card name should match")
+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))
+ (is (empty? (:unresolved result)) "Should not have unresolved refs"))))
+
+(deftest mbql-with-joins-test
+ (testing "MBQL query with joins validates successfully"
+ (let [source (make-test-source)
+ results (check-all source)
+ result (get results "orders-with-products")]
+ (is (some? result) "Card should be found")
+ (is (= "Orders With Products" (:name result)) "Card name should match")
+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))
+ (is (empty? (:unresolved result)) "Should not have unresolved refs")
+ ;; Check that both tables are referenced
+ (let [tables (get-in result [:refs :tables])]
+ (is (some #(re-find #"orders" %) tables) "Should reference orders table")
+ (is (some #(re-find #"products" %) tables) "Should reference products table")))))
+
+(deftest all-cards-checked-test
+ (testing "All cards in fixtures are checked"
+ (let [source (make-test-source)
+ results (check-all source)]
+ (is (>= (count results) 5) "Should check at least 5 cards")
+ (is (contains? results "simple-orders"))
+ (is (contains? results "native-orders"))
+ (is (contains? results "orders-with-products"))
+ (is (contains? results "schemaless-query"))
+ (is (contains? results "orders-with-fk")))))
+
+;;; ===========================================================================
+;;; Schema-less Database Tests
+;;;
+;;; Databases like SQLite don't have schemas, so tables are stored directly
+;;; under databases/<db>/tables/ instead of databases/<db>/schemas/<schema>/tables/
+;;; ===========================================================================
+
+(deftest schemaless-database-index-test
+ (testing "Schema-less databases are indexed correctly with nil schema"
+ (let [source (make-test-source)]
+ (is (= #{"Test Database" "SQLite DB"} (set (source/all-database-names source))))
+ (is (= 4 (count (source/all-table-paths source))) "Should have 4 tables via source"))))
+
+(deftest schemaless-query-test
+ (testing "Query on schema-less database validates successfully"
+ (let [source (make-test-source)
+ results (check-all source)
+ result (get results "schemaless-query")]
+ (is (some? result) "Card should be found")
+ (is (= "Schemaless Query" (:name result)) "Card name should match")
+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))
+ (is (empty? (:unresolved result)) "Should not have unresolved refs")
+ (is (empty? (:bad-refs result)) "Should not have bad refs"))))
+
+;;; ===========================================================================
+;;; FK Resolution Tests
+;;;
+;;; Foreign key references in field metadata and result_metadata must be
+;;; converted from path vectors to integer IDs.
+;;; ===========================================================================
+
+(deftest fk-in-field-metadata-test
+ (testing "FK target in field metadata is resolved to integer ID"
+ (let [source (make-test-source)
+ index (serdes-format/source-index source)
+ provider (provider/make-provider (store/make-store source source index))
+ ;; Get the product_id field which has FK to products.id
+ fields (lib.metadata.protocols/metadatas
+ provider {:lib/type :metadata/column})
+ fk-field (first (filter #(= "product_id" (:name %)) fields))]
+ (is (some? fk-field) "Should find product_id field")
+ (is (= :type/FK (:semantic-type fk-field)) "Should be FK type")
+ (is (pos-int? (:fk-target-field-id fk-field))
+ "FK target should be resolved to positive integer"))))
+
+(deftest fk-in-result-metadata-test
+ (testing "FK target in card result_metadata is resolved to integer ID"
+ (let [source (make-test-source)
+ results (check-all source)
+ result (get results "orders-with-fk")]
+ (is (some? result) "Card should be found")
+ (is (= "Orders With FK" (:name result)) "Card name should match")
+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))
+ (is (empty? (:unresolved result)) "Should not have unresolved refs")
+ (is (empty? (:bad-refs result)) "Should not have bad refs"))))
+
+;;; ===========================================================================
+;;; Tests Using In-Memory Source for Edge Cases
+;;;
+;;; ===========================================================================
+;;; MBQL normalization tests
+;;; ===========================================================================
+
+(deftest template-tag-default-not-keywordized-test
+ (testing "Template tag :default values are not incorrectly keywordized as MBQL operators"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["DB" "PUBLIC" "ORDERS" "CREATED_AT"]
+ {:name "CREATED_AT" :base_type "type/DateTimeWithLocalTZ"
+ :database_type "TIMESTAMP WITH TIME ZONE"}}
+ :cards {"tMpLtAgDeFaUlT00card1"
+ {:name "Field Filter Card"
+ :entity_id "tMpLtAgDeFaUlT00card1"
+ :dataset_query
+ {:database "DB"
+ :stages [{:native "SELECT * FROM ORDERS WHERE {{date_filter}}"
+ :template-tags
+ {"date_filter"
+ {:type "dimension"
+ :dimension ["field" {} ["DB" "PUBLIC" "ORDERS" "CREATED_AT"]]
+ :default "past30days"
+ :display-name "Date Filter"
+ :name "date_filter"
+ :widget-type "date/all-options"}}
+ "lib/type" "mbql.stage/native"}]
+ "lib/type" "mbql/query"}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["tMpLtAgDeFaUlT00card1"])
+ result (get results "tMpLtAgDeFaUlT00card1")]
+ (is (nil? (:error result))
+ (str "Should not error: " (:error result))))))
+
+;;; ===========================================================================
+;;; Create a simple in-memory source to test unresolved references
+;;; without needing temp files.
+;;; ===========================================================================
+
+(deftest unresolved-table-in-card-test
+ (testing "Card referencing nonexistent table is detected"
+ (let [entities {:databases {"Test Database" {:name "Test Database"
+ :engine "h2"}}
+ :tables {} ; No tables!
+ :fields {}
+ :cards {"test-card" {:name "Test Card"
+ :entity_id "test-card"
+ :dataset_query {:database "Test Database"
+ :type "query"
+ :query {:source-table ["Test Database" "public" "orders"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["test-card"])
+ result (get results "test-card")]
+ (is (some? result) "Card should be found")
+ (is (= "Test Card" (:name result)) "Card name should match")
+ (is (seq (:unresolved result)) "Should have unresolved refs")
+ (is (some #(= :table (:type %)) (:unresolved result))
+ "Should have unresolved table ref"))))
+
+(deftest unresolved-database-in-card-test
+ (testing "Card referencing nonexistent database is detected"
+ (let [entities {:databases {} ; No databases!
+ :tables {}
+ :fields {}
+ :cards {"test-card" {:name "Test Card"
+ :entity_id "test-card"
+ :dataset_query {:database "Nonexistent Database"
+ :type "query"
+ :query {:source-table ["Nonexistent Database" "public" "orders"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["test-card"])
+ result (get results "test-card")]
+ (is (some? result) "Card should be found")
+ (is (= "Test Card" (:name result)) "Card name should match")
+ ;; Either has an error about missing database, or unresolved refs
+ (is (or (:error result)
+ (some #(= :database (:type %)) (:unresolved result)))
+ "Should have unresolved database or error"))))
+
+(deftest check-specific-cards-test
+ (testing "Can check specific cards by entity-id"
+ (let [source (make-test-source)
+ results (check-specific source ["simple-orders" "native-orders"])]
+ (is (>= (count results) 2) "Should check at least 2 cards")
+ (is (contains? results "simple-orders"))
+ (is (contains? results "native-orders"))
+ (is (not (contains? results "orders-with-products"))))))
+
+;;; ===========================================================================
+;;; Results Processing Tests
+;;; ===========================================================================
+
+(deftest result-status-test
+ (testing "Result status is computed correctly"
+ (is (= :ok (checker/result-status {})))
+ (is (= :error (checker/result-status {:error "Something went wrong"})))
+ (is (= :unresolved (checker/result-status {:unresolved [{:type :table}]})))
+ (is (= :issues (checker/result-status {:bad-refs [{:type :field}]})))))
+
+(deftest summarize-results-test
+ (testing "Results are summarized correctly"
+ (let [results {"card1" {}
+ "card2" {:error "oops"}
+ "card3" {:unresolved [{:type :table}]}
+ "card4" {:bad-refs [{:type :field}]}}
+ summary (checker/summarize-results results)]
+ (is (= 4 (:total summary)))
+ (is (= 1 (:ok summary)))
+ (is (= 1 (:errors summary)))
+ (is (= 1 (:unresolved summary)))
+ (is (= 1 (:issues summary))))))
+
+;;; ===========================================================================
+;;; Format-error Tests (LLM-friendly error output)
+;;; ===========================================================================
+
+(deftest format-error-returns-nil-for-ok-test
+ (testing "format-error returns nil for OK results"
+ (is (nil? (checker/format-error ["card-1" {}])))
+ (is (nil? (checker/format-error ["card-1" {:refs {:tables ["t"]}}])))))
+
+(deftest format-error-includes-card-identity-test
+ (testing "format-error includes card name and entity-id"
+ (let [output (checker/format-error ["abc123" {:name "My Card" :error "boom"}])]
+ (is (some? output))
+ (is (re-find #"My Card" output))
+ (is (re-find #"abc123" output)))))
+
+(deftest format-error-includes-unresolved-refs-test
+ (testing "format-error includes unresolved field/table refs"
+ (let [output (checker/format-error
+ ["eid" {:name "Bad Card"
+ :unresolved [{:type :field :path ["db" "public" "t" "bad_col"]}
+ {:type :table :path ["db" "public" "missing"]}]}])]
+ (is (re-find #"unresolved field" output))
+ (is (re-find #"bad_col" output))
+ (is (re-find #"unresolved table" output))
+ (is (re-find #"missing" output)))))
+
+(deftest format-error-includes-bad-refs-test
+ (testing "format-error includes bad-refs"
+ (let [output (checker/format-error
+ ["eid" {:name "Issue Card"
+ :bad-refs [{:type :missing-column :name "gone"}]}])]
+ (is (re-find #"bad ref" output))
+ (is (re-find #"missing-column" output)))))
+
+(deftest format-error-includes-error-message-test
+ (testing "format-error includes the error message"
+ (let [output (checker/format-error
+ ["eid" {:name "Error Card" :error "Unknown database: foo"}])]
+ (is (re-find #"error: Unknown database: foo" output)))))
+
+(deftest format-result-non-card-entity-test
+ (testing "format-result shows kind for non-card entities instead of Card ID"
+ (let [output (checker/format-result
+ ["eid" {:name "My Dashboard" :entity-id "eid" :kind :dashboard
+ :unresolved [{:type :collection :entity-id "bad" :message "not found"}]}])]
+ (is (re-find #"Kind: dashboard" output))
+ (is (not (re-find #"Card ID" output))))))
+
+(deftest format-error-non-card-entity-test
+ (testing "format-error uses kind name for non-card entities"
+ (let [output (checker/format-error
+ ["eid" {:name "My Transform" :entity-id "eid" :kind :transform
+ :error "Unknown database: foo"}])]
+ (is (re-find #"^transform:" output))
+ (is (not (re-find #"^card:" output))))))
+
+;;; ===========================================================================
+;;; Sentinel ID Tests — unresolved refs get IDs so queries can be constructed
+;;; ===========================================================================
+
+(deftest sentinel-id-for-unresolved-field-test
+ (testing "Card with unresolved field gets sentinel ID — query builds, field flagged"
+ (let [entities {:databases {"Test DB" {:name "Test DB" :engine "h2"}}
+ :tables {["Test DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["Test DB" "PUBLIC" "ORDERS" "ID"] {:name "ID"
+ :base_type "type/BigInteger"
+ :database_type "BIGINT"
+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}
+ ;; Card references NONEXISTENT_FIELD which doesn't exist
+ :cards {"bad-field" {:name "Bad Field Card"
+ :entity_id "bad-field"
+ :type "question"
+ :dataset_query {:database "Test DB"
+ :type "query"
+ :query {:source-table ["Test DB" "PUBLIC" "ORDERS"]
+ :breakout [[:field
+ ["Test DB" "PUBLIC" "ORDERS" "NONEXISTENT_FIELD"]
+ {:base-type :type/Text}]]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["bad-field"])
+ result (get results "bad-field")]
+ (is (some? result))
+ ;; Should NOT have an :error (query should build successfully with sentinel)
+ (is (nil? (:error result)) (str "Should not blow up: " (:error result)))
+ ;; Should have unresolved refs tracking the bad field
+ (is (seq (:unresolved result)) "Should track unresolved field")
+ (is (some #(= :field (:type %)) (:unresolved result))
+ "Should have unresolved field ref"))))
+
+(deftest sentinel-id-for-unresolved-table-in-join-test
+ (testing "Card with unresolved table in join gets sentinel ID — query builds"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {}
+ ;; Card joins to a nonexistent table
+ :cards {"bad-join" {:name "Bad Join Card"
+ :entity_id "bad-join"
+ :type "question"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "ORDERS"]
+ :joins [{:source-table ["DB" "PUBLIC" "NONEXISTENT"]
+ :alias "n"
+ :condition [:= 1 1]}]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["bad-join"])
+ result (get results "bad-join")]
+ (is (some? result))
+ ;; Should NOT have an :error (query should build with sentinel)
+ (is (nil? (:error result)) (str "Should not blow up: " (:error result)))
+ ;; Should track the unresolved table
+ (is (seq (:unresolved result)) "Should track unresolved table")
+ (is (some #(= :table (:type %)) (:unresolved result))
+ "Should have unresolved table ref"))))
+
+;;; ===========================================================================
+;;; Native SQL validation via deps.analysis
+;;;
+;;; These tests verify that deps.analysis/check-entity is properly invoked
+;;; for native SQL cards (requires sql-tools.init to be loaded).
+;;; ===========================================================================
+
+(deftest native-card-bad-sql-produces-native-errors-test
+ (testing "Card with bad SQL produces native-errors, not validation-exception-error"
+ (let [entities {:databases {"Test DB" {:name "Test DB" :engine "h2"}}
+ :tables {["Test DB" "PUBLIC" "ORDERS"]
+ {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["Test DB" "PUBLIC" "ORDERS" "ID"]
+ {:name "ID" :base_type "type/BigInteger"
+ :database_type "BIGINT"
+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}
+ :cards {"bad-sql" {:name "Bad SQL"
+ :entity_id "bad-sql"
+ :type "question"
+ :dataset_query {:database "Test DB"
+ :type "native"
+ :native {:query "SELLECT * FROM ORDERS"}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["bad-sql"])
+ result (get results "bad-sql")]
+ (is (some? result))
+ (is (or (seq (:native-errors result))
+ (:error result))
+ "Bad SQL should produce native-errors or error, not validation-exception-error")
+ (is (empty? (filter #(= :validation-exception-error (:type %))
+ (:bad-refs result)))
+ "Should not have validation-exception-error in bad-refs"))))
+
+(deftest native-card-valid-sql-no-errors-test
+ (testing "Card with valid native SQL produces no errors"
+ (let [entities {:databases {"Test DB" {:name "Test DB" :engine "h2"}}
+ :tables {["Test DB" "PUBLIC" "ORDERS"]
+ {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["Test DB" "PUBLIC" "ORDERS" "ID"]
+ {:name "ID" :base_type "type/BigInteger"
+ :database_type "BIGINT"
+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}
+ ["Test DB" "PUBLIC" "ORDERS" "TOTAL"]
+ {:name "TOTAL" :base_type "type/Float"
+ :database_type "DOUBLE PRECISION"
+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}
+ :cards {"good-sql" {:name "Good SQL"
+ :entity_id "good-sql"
+ :type "question"
+ :dataset_query {:database "Test DB"
+ :type "native"
+ :native {:query "SELECT ID, TOTAL FROM ORDERS"}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["good-sql"])
+ result (get results "good-sql")]
+ (is (some? result))
+ (is (nil? (:error result)) (str "Should not error: " (:error result)))
+ (is (empty? (:native-errors result)) "Valid SQL should have no native errors")
+ (is (empty? (:bad-refs result)) "Valid SQL should have no bad refs"))))
+
+;;; ===========================================================================
+;;; Computed result-metadata tests
+;;;
+;;; When YAML has no result_metadata, the checker computes it from the query
+;;; so deps.analysis can verify card subquery column references.
+;;; ===========================================================================
+
+(deftest mbql-card-computes-result-metadata-test
+ (testing "MBQL card without result_metadata computes columns from query"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["DB" "PUBLIC" "ORDERS" "ID"]
+ {:name "ID" :base_type "type/BigInteger" :database_type "BIGINT"
+ :table_id ["DB" "PUBLIC" "ORDERS"]}
+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]
+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"
+ :table_id ["DB" "PUBLIC" "ORDERS"]}}
+ ;; Card with MBQL query but no result_metadata
+ :cards {"mbql-card" {:name "MBQL Card"
+ :entity_id "mbql-card"
+ :type "question"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "ORDERS"]}}}
+ ;; Native card referencing the MBQL card's columns
+ "native-card" {:name "Native Ref"
+ :entity_id "native-card"
+ :type "question"
+ :dataset_query {:database "DB"
+ :type "native"
+ :native {:query "SELECT ID FROM {{#1-mbql}}"
+ :template-tags {"#1-mbql" {:type :card
+ :name "#1-mbql"
+ :card-id "mbql-card"}}}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["native-card"])
+ result (get results "native-card")]
+ (is (some? result))
+ ;; ID exists in ORDERS and in the MBQL card's computed result-metadata,
+ ;; so it should NOT be flagged as missing
+ (is (not (some #(and (= :missing-column (:type %))
+ (= "ID" (:name %)))
+ (:native-errors result)))
+ "ID should not be flagged — it exists in the MBQL card's computed columns"))))
+
+(deftest mbql-card-named-aggregation-in-result-metadata-test
+ (testing "Named aggregation columns appear in computed result-metadata"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["DB" "PUBLIC" "ORDERS" "TOTAL"]
+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"
+ :table_id ["DB" "PUBLIC" "ORDERS"]}}
+ ;; Card with named aggregation but no result_metadata
+ :cards {"agg-card" {:name "Agg Card"
+ :entity_id "agg-card"
+ :type "question"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "ORDERS"]
+ :aggregation [[:sum {:name "total_revenue"
+ :display-name "Total Revenue"}
+ [:field
+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]
+ {:base-type :type/Float}]]]}}}
+ ;; Native card referencing the named aggregation column
+ "ref-card" {:name "Ref Card"
+ :entity_id "ref-card"
+ :type "question"
+ :dataset_query {:database "DB"
+ :type "native"
+ :native {:query "SELECT total_revenue FROM {{#1-agg}}"
+ :template-tags {"#1-agg" {:type :card
+ :name "#1-agg"
+ :card-id "agg-card"}}}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["ref-card"])
+ result (get results "ref-card")]
+ (is (some? result))
+ (is (not (some #(and (= :missing-column (:type %))
+ (= "total_revenue" (:name %)))
+ (:native-errors result)))
+ "total_revenue should not be flagged — it's a named aggregation in the source card"))))
+
+(deftest native-card-without-result-metadata-no-false-positives-test
+ (testing "Native card without result_metadata doesn't produce false positive errors"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["DB" "PUBLIC" "ORDERS" "ID"]
+ {:name "ID" :base_type "type/BigInteger" :database_type "BIGINT"
+ :table_id ["DB" "PUBLIC" "ORDERS"]}
+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]
+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"
+ :table_id ["DB" "PUBLIC" "ORDERS"]}}
+ :cards {"sql-card" {:name "SQL Card"
+ :entity_id "sql-card"
+ :type "question"
+ :dataset_query {:database "DB"
+ :type "native"
+ :native {:query "SELECT ID, TOTAL FROM ORDERS"}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["sql-card"])
+ result (get results "sql-card")]
+ (is (some? result))
+ (is (nil? (:error result)))
+ (is (empty? (:bad-refs result)) "Valid native SQL should have no bad refs"))))
+
+;;; ===========================================================================
+;;; Measure and Segment reference tests
+;;; ===========================================================================
+
+(deftest measure-ref-resolved-test
+ (testing "Card referencing a known measure resolves without errors"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Uses Measure"
+ :entity_id "card-1"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]
+ :aggregation [["measure" "mSrAvgProdPrice00008x"]]}}}}
+ :measures {"mSrAvgProdPrice00008x" {:name "Avg Product Price"
+ :entity_id "mSrAvgProdPrice00008x"}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (nil? (:error result)) (str "Should not error: " (:error result)))
+ (is (empty? (filter #(= :measure (:type %)) (or (:unresolved result) [])))
+ "Should not have unresolved measure refs"))))
+
+(deftest measure-ref-missing-test
+ (testing "Card referencing unknown measure is flagged"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Bad Measure Ref"
+ :entity_id "card-1"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]
+ :aggregation [["measure" "xXxNoExIsT0MeAsUrExx1"]]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (some #(= :measure (:type %)) (or (:unresolved result) []))
+ "Should have unresolved measure ref"))))
+
+(deftest segment-ref-resolved-test
+ (testing "Card referencing a known segment resolves without errors"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Uses Segment"
+ :entity_id "card-1"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]
+ :filter ["segment" "aB3kLmN9pQrStUvWxYz1a"]}}}}
+ :segments {"aB3kLmN9pQrStUvWxYz1a" {:name "Known Segment"
+ :entity_id "aB3kLmN9pQrStUvWxYz1a"}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (nil? (:error result)) (str "Should not error: " (:error result)))
+ (is (empty? (filter #(= :segment (:type %)) (or (:unresolved result) [])))
+ "Should not have unresolved segment refs"))))
+
+(deftest segment-ref-missing-test
+ (testing "Card referencing unknown segment is flagged"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Bad Segment Ref"
+ :entity_id "card-1"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]
+ :filter ["segment" "xXxNoExIsT0SeGmEnTx12"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (some #(= :segment (:type %)) (or (:unresolved result) []))
+ "Should have unresolved segment ref"))))
+
+;;; ===========================================================================
+;;; Transitive entity ref tests
+;;; ===========================================================================
+
+(deftest transitive-snippet-refs-test
+ (testing "Card sourcing from another card that uses a snippet surfaces that snippet transitively"
+ ;; Entity-ids must be 21-char NanoIDs to pass portable-id? checks
+ (let [inner-eid "tRaNsInNeRsNiPpEt0001"
+ outer-eid "tRaNsOuTeRsNiPpEt0001"
+ snip-eid "tRaNsSnIpPeTiD0000001"
+ entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :fields {["DB" "PUBLIC" "ORDERS" "ID"] {:name "ID" :base_type "type/BigInteger"
+ :database_type "BIGINT"}}
+ :cards {inner-eid {:name "Inner Card"
+ :entity_id inner-eid
+ :dataset_query {:database "DB"
+ :stages [{:native "SELECT * FROM ORDERS WHERE {{snippet: My Filter}}"
+ :template-tags {"snippet: My Filter"
+ {:type "snippet"
+ :snippet-name "My Filter"
+ :snippet-id snip-eid}}
+ "lib/type" "mbql.stage/native"}]
+ "lib/type" "mbql/query"}
+ :result_metadata [{:name "ID" :base_type "type/BigInteger"}]}
+ outer-eid {:name "Outer Card"
+ :entity_id outer-eid
+ :dataset_query {:database "DB"
+ :stages [{:source-card inner-eid
+ "lib/type" "mbql.stage/mbql"}]
+ "lib/type" "mbql/query"}}}
+ :snippets {snip-eid {:name "My Filter" :content "1=1"}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index [outer-eid inner-eid])
+ outer (get results outer-eid)]
+ (is (= ["Inner Card"] (get-in outer [:refs :source-cards]))
+ "Should list inner card as source")
+ (is (= ["My Filter"] (get-in outer [:refs :snippets]))
+ "Should transitively surface the inner card's snippet"))))
+
+(deftest transitive-measure-refs-test
+ (testing "Card sourcing from another card that uses measures/segments surfaces them transitively"
+ (let [inner-eid "tRaNsInNeRmEaSuRe0001"
+ outer-eid "tRaNsOuTeRmEaSuRe0001"
+ msr-eid "tRaNsMeAsUrEiD0000001"
+ seg-eid "tRaNsSeGmEnTiD0000001"
+ entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {inner-eid {:name "Inner Metrics"
+ :entity_id inner-eid
+ :dataset_query {:database "DB"
+ :stages [{:source-table ["DB" "PUBLIC" "T"]
+ :aggregation [["measure" msr-eid]]
+ :filters [["segment" seg-eid]]
+ "lib/type" "mbql.stage/mbql"}]
+ "lib/type" "mbql/query"}
+ :result_metadata [{:name "count" :base_type "type/Integer"}]}
+ outer-eid {:name "Outer Query"
+ :entity_id outer-eid
+ :dataset_query {:database "DB"
+ :stages [{:source-card inner-eid
+ "lib/type" "mbql.stage/mbql"}]
+ "lib/type" "mbql/query"}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ index (assoc index
+ :measure {msr-eid :memory}
+ :segment {seg-eid :memory})
+ results (checker/check-entities schema assets index [outer-eid inner-eid])
+ outer (get results outer-eid)]
+ (is (= ["Inner Metrics"] (get-in outer [:refs :source-cards])))
+ (is (= [msr-eid] (get-in outer [:refs :measures]))
+ "Should transitively surface the inner card's measure")
+ (is (= [seg-eid] (get-in outer [:refs :segments]))
+ "Should transitively surface the inner card's segment"))))
+
+;;; ===========================================================================
+;;; Collection reference tests
+;;; ===========================================================================
+
+(deftest valid-collection-id-test
+ (testing "Card with valid collection_id passes"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Good Card"
+ :entity_id "card-1"
+ :collection_id "coll-1"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ ;; Index includes the collection
+ index (assoc index :collection {"coll-1" :memory})
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (nil? (:error result)))
+ (is (empty? (:unresolved result)) "Valid collection_id should not produce unresolved refs"))))
+
+(deftest missing-collection-id-test
+ (testing "Card with collection_id pointing to nonexistent entity is flagged"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Bad Collection Card"
+ :entity_id "card-1"
+ :collection_id "nonexistent"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (seq (:unresolved result)) "Missing collection should be flagged")
+ (is (some #(= :collection (:type %)) (:unresolved result))
+ "Should have unresolved collection ref"))))
+
+(deftest collection-id-points-to-card-test
+ (testing "Card with collection_id pointing to another card (not a collection) is flagged"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Card With Bad Collection"
+ :entity_id "card-1"
+ :collection_id "card-2"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}
+ "card-2" {:name "Other Card"
+ :entity_id "card-2"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (seq (:unresolved result)) "collection_id pointing to a card should be flagged")
+ (is (some #(and (= :collection (:type %))
+ (re-find #"card" (or (:message %) "")))
+ (:unresolved result))
+ "Error should mention it points to a card"))))
+
+(deftest nil-collection-id-test
+ (testing "Card with nil collection_id is fine"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Root Card"
+ :entity_id "card-1"
+ :collection_id nil
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (nil? (:error result)))
+ (is (empty? (:unresolved result)) "nil collection_id should be fine"))))
+
+;;; ===========================================================================
+;;; Dashboard ID reference tests
+;;; ===========================================================================
+
+(deftest card-with-missing-dashboard-id-test
+ (testing "Card with dashboard_id pointing to nonexistent dashboard is flagged"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Orphaned Card"
+ :entity_id "card-1"
+ :dashboard_id "nonexistent-dash"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (seq (:unresolved result)) "Missing dashboard_id should be flagged")
+ (is (some #(= :dashboard (:type %)) (:unresolved result))
+ "Should have unresolved dashboard ref"))))
+
+(deftest card-with-valid-dashboard-id-test
+ (testing "Card with dashboard_id pointing to a known dashboard passes"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Dashboard Card"
+ :entity_id "card-1"
+ :dashboard_id "dash-1"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ index (assoc index :dashboard {"dash-1" :memory})
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (nil? (:error result)))
+ (is (empty? (:unresolved result)) "Valid dashboard_id should not produce unresolved refs"))))
+
+(deftest card-with-null-dashboard-id-test
+ (testing "Card with nil dashboard_id is fine"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Standalone Card"
+ :entity_id "card-1"
+ :dashboard_id nil
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (nil? (:error result)))
+ (is (empty? (:unresolved result)) "nil dashboard_id should be fine"))))
+
+(deftest card-with-dashboard-id-pointing-to-card-test
+ (testing "Card with dashboard_id pointing to another card (not a dashboard) is flagged"
+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}
+ :fields {}
+ :cards {"card-1" {:name "Bad Dash Ref"
+ :entity_id "card-1"
+ :dashboard_id "card-2"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}
+ "card-2" {:name "Other Card"
+ :entity_id "card-2"
+ :dataset_query {:database "DB"
+ :type "query"
+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["card-1"])
+ result (get results "card-1")]
+ (is (seq (:unresolved result)) "dashboard_id pointing to a card should be flagged")
+ (is (some #(and (= :dashboard (:type %))
+ (re-find #"card" (or (:message %) "")))
+ (:unresolved result))
+ "Error should mention it points to a card"))))
+
+(deftest dashboard-bad-collection-id-test
+ (testing "Dashboard with invalid collection_id is flagged"
+ (let [entities {:databases {} :tables {} :fields {} :cards {}
+ :dashboards {"dash-1" {:name "My Dashboard"
+ :entity_id "dash-1"
+ :collection_id "nonexistent"}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index)]
+ (is (contains? results "dash-1") "Dashboard should be in results")
+ (let [result (get results "dash-1")]
+ (is (seq (:unresolved result)) "Dashboard with bad collection_id should be flagged")
+ (is (some #(= :collection (:type %)) (:unresolved result)))))))
+
+(deftest dashboard-valid-collection-id-not-flagged-test
+ (testing "Dashboard with valid collection_id passes"
+ (let [entities {:databases {} :tables {} :fields {} :cards {}
+ :dashboards {"dash-1" {:name "Good Dashboard"
+ :entity_id "dash-1"
+ :collection_id "coll-1"}}
+ :collections {"coll-1" {:name "My Collection"
+ :entity_id "coll-1"}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index)]
+ (is (= :ok (checker/result-status (get results "dash-1")))
+ "Dashboard with valid collection_id should be OK"))))
+
+;;; ===========================================================================
+;;; Dashboard semantic validation tests
+;;; ===========================================================================
+
+(defn- with-temp-dashboard
+ "Run checks with a dashboard entity provided via in-memory source.
+ `dashboard-data` is a map representing the dashboard.
+ `extra-index` is merged into the index (e.g. to add cards).
+ Calls `(f results)` with the check results."
+ [dashboard-data extra-index f]
+ (let [entity-id (:entity_id dashboard-data)
+ entities {:databases {}
+ :tables {}
+ :fields {}
+ :cards {}
+ :dashboards {entity-id dashboard-data}}
+ [schema assets base-index] (helpers/make-sources-and-index entities)
+ index (merge base-index
+ {:collection {}
+ :card {}}
+ extra-index)
+ results (checker/check-entities schema assets index)]
+ (f results)))
+
+(deftest dashboard-card-ref-valid-test
+ (testing "Dashboard with valid card_id refs passes"
+ (with-temp-dashboard
+ {:name "Good Dashboard"
+ :entity_id "dash-1"
+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]
+ :dashcards [{:entity_id "dc-1"
+ :card_id "card-1"
+ :dashboard_tab_id ["dash-1" "tab-1"]
+ :row 0 :col 0 :size_x 12 :size_y 6}]}
+ {:card {"card-1" :memory}}
+ (fn [results]
+ (is (= :ok (checker/result-status (get results "dash-1")))
+ "Dashboard with valid refs should be OK")))))
+
+(deftest dashboard-card-ref-missing-test
+ (testing "Dashboard with card_id pointing to unknown card is flagged"
+ (with-temp-dashboard
+ {:name "Bad Card Ref"
+ :entity_id "dash-1"
+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]
+ :dashcards [{:entity_id "dc-1"
+ :card_id "nonexistent-card"
+ :dashboard_tab_id ["dash-1" "tab-1"]
+ :row 0 :col 0 :size_x 12 :size_y 6}]}
+ {}
+ (fn [results]
+ (is (contains? results "dash-1"))
+ (let [result (get results "dash-1")]
+ (is (some #(= :dashcard-card-ref (:type %)) (:unresolved result))
+ "Should flag missing card ref"))))))
+
+(deftest dashboard-virtual-card-null-card-id-test
+ (testing "Dashboard with null card_id (heading/text) is fine"
+ (with-temp-dashboard
+ {:name "Headings Dashboard"
+ :entity_id "dash-1"
+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]
+ :dashcards [{:entity_id "dc-1"
+ :card_id nil
+ :dashboard_tab_id ["dash-1" "tab-1"]
+ :row 0 :col 0 :size_x 24 :size_y 1}]}
+ {}
+ (fn [results]
+ (is (= :ok (checker/result-status (get results "dash-1")))
+ "Virtual card with null card_id should be fine")))))
+
+(deftest dashboard-tab-ref-invalid-test
+ (testing "Dashboard with dashcard referencing nonexistent tab is flagged"
+ (with-temp-dashboard
+ {:name "Bad Tab Ref"
+ :entity_id "dash-1"
+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]
+ :dashcards [{:entity_id "dc-1"
+ :card_id nil
+ :dashboard_tab_id ["dash-1" "wrong-tab"]
+ :row 0 :col 0 :size_x 12 :size_y 6}]}
+ {}
+ (fn [results]
+ (is (contains? results "dash-1"))
+ (let [result (get results "dash-1")]
+ (is (some #(= :dashcard-tab-ref (:type %)) (:unresolved result))
+ "Should flag bad tab ref"))))))
+
+(deftest dashboard-grid-overflow-test
+ (testing "Dashboard with dashcard extending beyond grid width is flagged"
+ (with-temp-dashboard
+ {:name "Grid Overflow"
+ :entity_id "dash-1"
+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]
+ :dashcards [{:entity_id "dc-1"
+ :card_id nil
+ :dashboard_tab_id ["dash-1" "tab-1"]
+ :row 0 :col 20 :size_x 12 :size_y 6}]}
+ {}
+ (fn [results]
+ (is (contains? results "dash-1"))
+ (let [result (get results "dash-1")]
+ (is (some #(= :dashcard-grid (:type %)) (:unresolved result))
+ "Should flag grid overflow"))))))
+
+(deftest dashboard-grid-negative-col-test
+ (testing "Dashboard with negative col is flagged"
+ (with-temp-dashboard
+ {:name "Negative Col"
+ :entity_id "dash-1"
+ :tabs []
+ :dashcards [{:entity_id "dc-1"
+ :card_id nil
+ :row 0 :col -1 :size_x 12 :size_y 6}]}
+ {}
+ (fn [results]
+ (is (contains? results "dash-1"))
+ (let [result (get results "dash-1")]
+ (is (some #(= :dashcard-grid (:type %)) (:unresolved result))
+ "Should flag negative col"))))))
+
+(deftest dashboard-valid-full-width-test
+ (testing "Dashboard with full-width card (col=0, size_x=24) passes"
+ (with-temp-dashboard
+ {:name "Full Width"
+ :entity_id "dash-1"
+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]
+ :dashcards [{:entity_id "dc-1"
+ :card_id nil
+ :dashboard_tab_id ["dash-1" "tab-1"]
+ :row 0 :col 0 :size_x 24 :size_y 1}]}
+ {}
+ (fn [results]
+ (is (= :ok (checker/result-status (get results "dash-1")))
+ "Full-width card should be valid")))))
+
+;;; ===========================================================================
+;;; Transform query validation tests
+;;; ===========================================================================
+
+(defn- with-temp-transform
+ "Run checks with a transform entity provided via in-memory source.
+ `transform-data` is a map representing the transform.
+ `extra-index` is merged into the index (e.g. to add databases).
+ Calls `(f results)` with the check results."
+ [transform-data extra-index f]
+ (let [entity-id (:entity_id transform-data)
+ entities {:databases (get extra-index :_databases {})
+ :tables (get extra-index :_tables {})
+ :fields (get extra-index :_fields {})
+ :cards {}
+ :transforms {entity-id transform-data}}
+ [schema assets base-index] (helpers/make-sources-and-index entities)
+ index (merge base-index
+ {:collection {}
+ :card {}}
+ (dissoc extra-index :_databases :_tables :_fields))
+ results (checker/check-entities schema assets index)]
+ (f results)))
+
+(deftest transform-bad-native-sql-test
+ (testing "Transform with invalid native SQL is flagged"
+ (with-temp-transform
+ {:name "Bad SQL Transform"
+ :entity_id "tx-1"
+ :source_database_id "Test DB"
+ :source {:type "query"
+ :query {"lib/type" "mbql/query"
+ :database "Test DB"
+ :stages [{"lib/type" "mbql.stage/native"
+ :native "SELET * FROM ORDERS"}]}}
+ :target {:type "table" :schema "TRANSFORMS" :name "bad_transform"}
+ :serdes/meta [{:id "tx-1" :model "Transform"}]}
+ {:_databases {"Test DB" {:name "Test DB" :engine "h2"}}
+ :_tables {["Test DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :_fields {}
+ :database {"Test DB" :memory}}
+ (fn [results]
+ (is (contains? results "tx-1") "Transform with bad SQL should be in results")
+ (let [result (get results "tx-1")]
+ (is (or (seq (:native-errors result))
+ (:error result)
+ (seq (:unresolved result)))
+ "Should have native errors, error, or unresolved refs"))))))
+
+(deftest transform-valid-native-sql-test
+ (testing "Transform with valid native SQL passes"
+ (with-temp-transform
+ {:name "Good SQL Transform"
+ :entity_id "tx-1"
+ :source_database_id "Test DB"
+ :source {:type "query"
+ :query {"lib/type" "mbql/query"
+ :database "Test DB"
+ :stages [{"lib/type" "mbql.stage/native"
+ :native "SELECT ID, TOTAL FROM ORDERS"}]}}
+ :target {:type "table" :schema "TRANSFORMS" :name "good_transform"}
+ :serdes/meta [{:id "tx-1" :model "Transform"}]}
+ {:_databases {"Test DB" {:name "Test DB" :engine "h2"}}
+ :_tables {["Test DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}
+ :_fields {["Test DB" "PUBLIC" "ORDERS" "ID"]
+ {:name "ID" :base_type "type/BigInteger" :database_type "BIGINT"
+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}
+ ["Test DB" "PUBLIC" "ORDERS" "TOTAL"]
+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"
+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}
+ :database {"Test DB" :memory}}
+ (fn [results]
+ (is (= :ok (checker/result-status (get results "tx-1")))
+ "Transform with valid SQL should be OK")))))
+
+(deftest transform-mbql-unresolved-table-test
+ (testing "Transform with MBQL referencing nonexistent table is flagged"
+ (with-temp-transform
+ {:name "Bad MBQL Transform"
+ :entity_id "tx-1"
+ :source_database_id "Test DB"
+ :source {:type "query"
+ :query {"lib/type" "mbql/query"
+ :database "Test DB"
+ :stages [{"lib/type" "mbql.stage/mbql"
+ :source-table ["Test DB" "PUBLIC" "NONEXISTENT"]}]}}
+ :target {:type "table" :schema "TRANSFORMS" :name "bad_mbql_transform"}
+ :serdes/meta [{:id "tx-1" :model "Transform"}]}
+ {:_databases {"Test DB" {:name "Test DB" :engine "h2"}}
+ :_tables {}
+ :_fields {}
+ :database {"Test DB" :memory}}
+ (fn [results]
+ (is (contains? results "tx-1") "Transform with unresolved table should be in results")
+ (let [result (get results "tx-1")]
+ (is (seq (:unresolved result)) "Should have unresolved refs"))))))
+
+(deftest transform-missing-database-test
+ (testing "Transform referencing nonexistent database is flagged"
+ (with-temp-transform
+ {:name "Bad DB Transform"
+ :entity_id "tx-1"
+ :source_database_id "Nonexistent DB"
+ :source {:type "query"
+ :query {"lib/type" "mbql/query"
+ :database "Nonexistent DB"
+ :stages [{"lib/type" "mbql.stage/native"
+ :native "SELECT 1"}]}}
+ :target {:type "table" :schema "TRANSFORMS" :name "bad_db"}
+ :serdes/meta [{:id "tx-1" :model "Transform"}]}
+ {}
+ (fn [results]
+ (is (contains? results "tx-1") "Transform with missing DB should be in results")
+ (let [result (get results "tx-1")]
+ (is (some #(= :database (:type %)) (:unresolved result))
+ "Should flag missing database"))))))
+
+;;; ===========================================================================
+;;; Multi-database tests
+;;;
+;;; The provider's `(database)` method must return the correct database for the
+;;; entity being checked, not just the first database in the store.
+;;; ===========================================================================
+
+(deftest multi-database-cards-test
+ (testing "Cards from different databases each get the correct database metadata"
+ (let [entities {:databases {"H2 DB" {:name "H2 DB" :engine "h2"}
+ "Postgres DB" {:name "Postgres DB" :engine "postgres"}}
+ :tables {["H2 DB" "PUBLIC" "ORDERS"]
+ {:name "ORDERS" :schema "PUBLIC"}
+ ["Postgres DB" "public" "customers"]
+ {:name "customers" :schema "public"}}
+ :fields {["H2 DB" "PUBLIC" "ORDERS" "ID"]
+ {:name "ID" :base_type "type/BigInteger"
+ :database_type "BIGINT"
+ :table_id ["H2 DB" "PUBLIC" "ORDERS"]}
+ ["Postgres DB" "public" "customers" "id"]
+ {:name "id" :base_type "type/BigInteger"
+ :database_type "bigint"
+ :table_id ["Postgres DB" "public" "customers"]}}
+ :cards {"h2-card" {:name "H2 Card"
+ :entity_id "h2-card"
+ :type "question"
+ :dataset_query {:database "H2 DB"
+ :type "query"
+ :query {:source-table ["H2 DB" "PUBLIC" "ORDERS"]}}}
+ "postgres-card" {:name "Postgres Card"
+ :entity_id "postgres-card"
+ :type "question"
+ :dataset_query {:database "Postgres DB"
+ :type "query"
+ :query {:source-table ["Postgres DB" "public" "customers"]}}}}}
+ [schema assets index] (helpers/make-sources-and-index entities)
+ results (checker/check-entities schema assets index ["h2-card" "postgres-card"])
+ h2-result (get results "h2-card")
+ pg-result (get results "postgres-card")]
+ (is (some? h2-result) "H2 card should be checked")
+ (is (some? pg-result) "Postgres card should be checked")
+ (is (nil? (:error h2-result)) (str "H2 card should not error: " (:error h2-result)))
+ (is (nil? (:error pg-result)) (str "Postgres card should not error: " (:error pg-result)))
+ (is (empty? (:bad-refs h2-result)) "H2 card should have no bad refs")
+ (is (empty? (:bad-refs pg-result)) "Postgres card should have no bad refs"))))
+
+;;; ===========================================================================
+;;; REPL Helpers
+;;; ===========================================================================
+
+(comment
+ ;; Run all tests
+ (clojure.test/run-tests 'metabase-enterprise.checker.semantic-test)
+
+ ;; Check fixtures path
+ (fixtures-path)
+
+ ;; Manually check all cards in fixtures
+ (def source (make-test-source))
+ (check-all source))
diff --git a/enterprise/backend/test/metabase_enterprise/checker/store_test.clj b/enterprise/backend/test/metabase_enterprise/checker/store_test.clj
new file mode 100644
index 00000000000..708cdef8ab2
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/checker/store_test.clj
@@ -0,0 +1,211 @@
+(ns metabase-enterprise.checker.store-test
+ "Tests for the checker store — ID registry, index queries, entity loading/caching."
+ (:require
+ [clojure.test :refer [deftest is testing]]
+ [metabase-enterprise.checker.store :as store]
+ [metabase-enterprise.checker.test-helpers :as helpers]))
+
+(set! *warn-on-reflection* true)
+
+;;; ===========================================================================
+;;; Test data
+;;; ===========================================================================
+
+(def ^:private test-entities
+ {:databases {"DB" {:name "DB" :engine "h2"}}
+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}
+ ["DB" "PUBLIC" "PRODUCTS"] {:name "PRODUCTS" :schema "PUBLIC"}}
+ :fields {["DB" "PUBLIC" "ORDERS" "ID"] {:name "ID" :base_type "type/BigInteger"}
+ ["DB" "PUBLIC" "ORDERS" "TOTAL"] {:name "TOTAL" :base_type "type/Float"}
+ ["DB" "PUBLIC" "PRODUCTS" "ID"] {:name "ID" :base_type "type/BigInteger"}
+ ["DB" "PUBLIC" "PRODUCTS" "CATEGORY"] {:name "CATEGORY" :base_type "type/Text"}}
+ :cards {"card-1" {:name "Card 1" :entity_id "card-1"}}})
+
+(defn- make-test-store []
+ (let [[schema assets index] (helpers/make-sources-and-index test-entities)]
+ (store/make-store schema assets index)))
+
+;;; ===========================================================================
+;;; Index query tests
+;;; ===========================================================================
+
+(deftest all-refs-test
+ (let [store (make-test-store)]
+ (testing "all-refs returns refs for each kind"
+ (is (= ["DB"] (store/all-database-names store)))
+ (is (= #{"card-1"} (set (store/all-card-ids store))))
+ (is (= 2 (count (store/all-table-paths store))))
+ (is (= 4 (count (store/all-field-paths store)))))))
+
+(deftest exists?-test
+ (let [store (make-test-store)]
+ (testing "known refs return true"
+ (is (store/exists? store :database "DB"))
+ (is (store/exists? store :table ["DB" "PUBLIC" "ORDERS"]))
+ (is (store/exists? store :card "card-1")))
+ (testing "unknown refs return false"
+ (is (not (store/exists? store :database "Nonexistent")))
+ (is (not (store/exists? store :table ["DB" "PUBLIC" "MISSING"])))
+ (is (not (store/exists? store :card "no-such-card"))))))
+
+(deftest index-kind-of-test
+ (let [store (make-test-store)]
+ (testing "returns the kind for known entity-ids"
+ (is (= :card (store/index-kind-of store "card-1"))))
+ (testing "returns nil for unknown entity-ids"
+ (is (nil? (store/index-kind-of store "unknown"))))))
+
+(deftest index-kind-of-multi-kind-test
+ (testing "index-kind-of distinguishes between kinds"
+ (let [empty-entities {:databases {} :tables {} :fields {} :cards {}}
+ [schema assets _] (helpers/make-sources-and-index empty-entities)
+ index {:card {"eid-1" :memory}
+ :dashboard {"eid-2" :memory}
+ :collection {"eid-3" :memory}
+ :document {"eid-4" :memory}}
+ store (store/make-store schema assets index)]
+ (is (= :card (store/index-kind-of store "eid-1")))
+ (is (= :dashboard (store/index-kind-of store "eid-2")))
+ (is (= :collection (store/index-kind-of store "eid-3")))
+ (is (= :document (store/index-kind-of store "eid-4"))))))
+
+(deftest index-file-test
+ (let [store (make-test-store)]
+ (testing "returns the file path for known refs"
+ (is (= :memory (store/index-file store :database "DB")))
+ (is (= :memory (store/index-file store :card "card-1"))))
+ (testing "returns nil for unknown refs"
+ (is (nil? (store/index-file store :database "Nonexistent"))))))
+
+(deftest fields-for-table-test
+ (let [store (make-test-store)]
+ (testing "returns field paths for a known table"
+ (is (= #{["DB" "PUBLIC" "ORDERS" "ID"]
+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]}
+ (store/fields-for-table store ["DB" "PUBLIC" "ORDERS"]))))
+ (testing "returns field paths for another table"
+ (is (= #{["DB" "PUBLIC" "PRODUCTS" "ID"]
+ ["DB" "PUBLIC" "PRODUCTS" "CATEGORY"]}
+ (store/fields-for-table store ["DB" "PUBLIC" "PRODUCTS"]))))
+ (testing "returns nil for unknown table"
+ (is (nil? (store/fields-for-table store ["DB" "PUBLIC" "MISSING"]))))))
+
+;;; ===========================================================================
+;;; ID registry tests
+;;; ===========================================================================
+
+(deftest get-or-assign-returns-stable-ids-test
+ (let [store (make-test-store)]
+ (testing "first call assigns an ID"
+ (let [id (store/get-or-assign! store :database "DB")]
+ (is (pos-int? id))))
+ (testing "second call returns the same ID"
+ (let [id1 (store/get-or-assign! store :database "DB")
+ id2 (store/get-or-assign! store :database "DB")]
+ (is (= id1 id2))))))
+
+(deftest different-refs-get-different-ids-test
+ (let [store (make-test-store)]
+ (testing "different refs get different IDs"
+ (let [id1 (store/get-or-assign! store :table ["DB" "PUBLIC" "ORDERS"])
+ id2 (store/get-or-assign! store :table ["DB" "PUBLIC" "PRODUCTS"])]
+ (is (not= id1 id2))))))
+
+(deftest id-ref-roundtrip-test
+ (let [store (make-test-store)]
+ (testing "id->ref and ref->id are inverses"
+ (let [ref ["DB" "PUBLIC" "ORDERS"]
+ id (store/get-or-assign! store :table ref)]
+ (is (= ref (store/id->ref store :table id)))
+ (is (= id (store/ref->id store :table ref)))))))
+
+(deftest ref->id-returns-nil-before-assignment-test
+ (let [store (make-test-store)]
+ (testing "ref->id returns nil for unassigned refs"
+ (is (nil? (store/ref->id store :table ["DB" "PUBLIC" "ORDERS"]))))
+ (testing "id->ref returns nil for unassigned IDs"
+ (is (nil? (store/id->ref store :table 999))))))
+
+(deftest ids-are-unique-across-kinds-test
+ (let [store (make-test-store)]
+ (testing "IDs don't collide across different kinds"
+ (let [db-id (store/get-or-assign! store :database "DB")
+ table-id (store/get-or-assign! store :table ["DB" "PUBLIC" "ORDERS"])
+ card-id (store/get-or-assign! store :card "card-1")]
+ (is (= 3 (count (distinct [db-id table-id card-id]))))))))
+
+;;; ===========================================================================
+;;; Entity loading and caching tests
+;;; ===========================================================================
+
+(deftest load-database-test
+ (let [store (make-test-store)]
+ (testing "load-database! returns data with assigned :id"
+ (let [data (store/load-database! store "DB")]
+ (is (some? data))
+ (is (= "DB" (:name data)))
+ (is (= "h2" (:engine data)))
+ (is (pos-int? (:id data)))))
+ (testing "load-database! returns nil for unknown database"
+ (is (nil? (store/load-database! store "Nonexistent"))))))
+
+(deftest load-table-test
+ (let [store (make-test-store)]
+ (testing "load-table! returns data with :id and :db_id"
+ (let [data (store/load-table! store ["DB" "PUBLIC" "ORDERS"])]
+ (is (some? data))
+ (is (= "ORDERS" (:name data)))
+ (is (pos-int? (:id data)))
+ (is (pos-int? (:db_id data)))))
+ (testing "load-table! returns nil for unknown table"
+ (is (nil? (store/load-table! store ["DB" "PUBLIC" "MISSING"]))))))
+
+(deftest load-field-test
+ (let [store (make-test-store)]
+ (testing "load-field! returns data with :id and :table_id"
+ (let [data (store/load-field! store ["DB" "PUBLIC" "ORDERS" "ID"])]
+ (is (some? data))
+ (is (= "ID" (:name data)))
+ (is (pos-int? (:id data)))
+ (is (pos-int? (:table_id data)))))
+ (testing "load-field! returns nil for unknown field"
+ (is (nil? (store/load-field! store ["DB" "PUBLIC" "ORDERS" "MISSING"]))))))
+
+(deftest load-card-test
+ (let [store (make-test-store)]
+ (testing "load-card! returns data with assigned :id"
+ (let [data (store/load-card! store "card-1")]
+ (is (some? data))
+ (is (= "Card 1" (:name data)))
+ (is (pos-int? (:id data)))))
+ (testing "load-card! returns nil for unknown card"
+ (is (nil? (store/load-card! store "nonexistent"))))))
+
+(deftest caching-returns-same-object-test
+ (let [store (make-test-store)]
+ (testing "loading the same entity twice returns the cached version"
+ (let [first-load (store/load-database! store "DB")
+ second-load (store/load-database! store "DB")]
+ (is (identical? first-load second-load))))))
+
+(deftest cached-entity-test
+ (let [store (make-test-store)]
+ (testing "cached-entity returns nil before loading"
+ (is (nil? (store/cached-entity store :database "DB"))))
+ (testing "cached-entity returns data after loading"
+ (store/load-database! store "DB")
+ (is (some? (store/cached-entity store :database "DB"))))))
+
+(deftest load-table-assigns-database-id-test
+ (let [store (make-test-store)]
+ (testing "loading a table also assigns an ID to its database"
+ (store/load-table! store ["DB" "PUBLIC" "ORDERS"])
+ (is (some? (store/ref->id store :database "DB"))
+ "Database should have an assigned ID after loading a table"))))
+
+(deftest load-field-assigns-table-id-test
+ (let [store (make-test-store)]
+ (testing "loading a field also assigns an ID to its table"
+ (store/load-field! store ["DB" "PUBLIC" "ORDERS" "ID"])
+ (is (some? (store/ref->id store :table ["DB" "PUBLIC" "ORDERS"]))
+ "Table should have an assigned ID after loading a field"))))
diff --git a/enterprise/backend/test/metabase_enterprise/checker/test_helpers.clj b/enterprise/backend/test/metabase_enterprise/checker/test_helpers.clj
new file mode 100644
index 00000000000..dcc3c8ca4cc
--- /dev/null
+++ b/enterprise/backend/test/metabase_enterprise/checker/test_helpers.clj
@@ -0,0 +1,68 @@
+(ns metabase-enterprise.checker.test-helpers
+ "Shared test helpers for the checker module."
+ (:require
+ [metabase-enterprise.checker.source :as source]))
+
+(defn make-schema-source
+ "Create an in-memory SchemaSource from maps of databases, tables, and fields."
+ [{:keys [databases tables fields]}]
+ (let [fields-by-table (reduce-kv (fn [m field-path _]
+ (let [table-path (subvec field-path 0 3)]
+ (update m table-path (fnil conj #{}) field-path)))
+ {}
+ fields)]
+ (reify
+ source/SchemaSource
+ (resolve-database [_ db-name] (get databases db-name))
+ (resolve-table [_ table-path] (get tables table-path))
+ (resolve-field [_ field-path] (get fields field-path))
+ (fields-for-table [_ table-path] (get fields-by-table table-path))
+ (all-field-paths [_] (keys fields))
+ (all-database-names [_] (keys databases))
+ (all-table-paths [_] (keys tables))
+ (tables-for-database [_ db-name] (filterv #(= (first %) db-name) (keys tables))))))
+
+(defn make-assets-source
+ "Create an in-memory AssetsSource from maps of entities.
+ Supports :cards, :snippets, :transforms, :segments, :dashboards,
+ :collections, :documents, :measures."
+ [{:keys [cards snippets transforms segments dashboards collections documents measures]}]
+ (reify
+ source/AssetsSource
+ (resolve-card [_ entity-id] (get cards entity-id))
+ (resolve-snippet [_ entity-id] (get snippets entity-id))
+ (resolve-transform [_ entity-id] (get transforms entity-id))
+ (resolve-segment [_ entity-id] (get segments entity-id))
+ (resolve-dashboard [_ entity-id] (get dashboards entity-id))
+ (resolve-collection [_ entity-id] (get collections entity-id))
+ (resolve-document [_ entity-id] (get documents entity-id))
+ (resolve-measure [_ entity-id] (get measures entity-id))))
+
+(defn make-schema-index
+ "Create a file index for an in-memory schema source.
+ Values are :memory since resolution goes through the source, not files."
+ [{:keys [databases tables fields]}]
+ {:database (zipmap (keys databases) (repeat :memory))
+ :table (zipmap (keys tables) (repeat :memory))
+ :field (zipmap (keys fields) (repeat :memory))})
+
+(defn make-assets-index
+ "Create a file index for an in-memory assets source.
+ Values are :memory since resolution goes through the source, not files."
+ [{:keys [cards snippets transforms segments dashboards collections documents measures]}]
+ (cond-> {:card (zipmap (keys cards) (repeat :memory))}
+ snippets (assoc :snippet (zipmap (keys snippets) (repeat :memory)))
+ transforms (assoc :transform (zipmap (keys transforms) (repeat :memory)))
+ segments (assoc :segment (zipmap (keys segments) (repeat :memory)))
+ dashboards (assoc :dashboard (zipmap (keys dashboards) (repeat :memory)))
+ collections (assoc :collection (zipmap (keys collections) (repeat :memory)))
+ documents (assoc :document (zipmap (keys documents) (repeat :memory)))
+ measures (assoc :measure (zipmap (keys measures) (repeat :memory)))))
+
+(defn make-sources-and-index
+ "Convenience: split an entities map into schema-source, assets-source, and merged index.
+ Returns [schema-source assets-source index]."
+ [entities]
+ [(make-schema-source entities)
+ (make-assets-source entities)
+ (merge (make-schema-index entities) (make-assets-index entities))])
diff --git a/src/metabase/core/bootstrap.clj b/src/metabase/core/bootstrap.clj
index e0eeac050bd..eb79d3ed4cd 100644
--- a/src/metabase/core/bootstrap.clj
+++ b/src/metabase/core/bootstrap.clj
@@ -1,7 +1,8 @@
(ns metabase.core.bootstrap
(:gen-class)
(:require
- [clojure.java.io :as io]))
+ [clojure.java.io :as io]
+ [clojure.tools.cli :as cli]))
(set! *warn-on-reflection* true)
@@ -26,11 +27,46 @@
;; ensure the [[clojure.tools.logging]] logger factory is the log4j2 version (slf4j is far slower and identified first)
(System/setProperty "clojure.tools.logging.factory" "clojure.tools.logging.impl/log4j2-factory")
+;;; ===========================================================================
+;;; Standalone modes
+;;;
+;;; These modes don't need the full Metabase infrastructure (app-db, events,
+;;; etc.) and can run with minimal namespace loading for fast startup.
+;;;
+;;; Usage: java -jar metabase.jar --mode checker [checker-specific args...]
+;;; ===========================================================================
+
+#_{:clj-kondo/ignore [:discouraged-var]}
+(def output!
+ "Alias for println so can suppress warning in one place"
+ println)
+
+(defn- run-standalone-mode
+ "Errors if --mode is specified but not recognized.
+ The mode's -main function is responsible for calling System/exit."
+ [mode args]
+ (let [startup (case mode
+ ;; schema checker was moved out to js. Perhaps this will get a long running server mode checker? Or
+ ;; perhaps just this one.
+ "checker" 'metabase-enterprise.checker.cli/entrypoint
+ nil)]
+ (if startup
+ ((requiring-resolve startup) args)
+ (do (binding [*out* *err*]
+ (output! (str "Unknown mode: " mode))
+ (output! "Available modes: checker"))
+ (System/exit 1)))))
+
(defn -main
"Main entrypoint. Invokes [[metabase.core.core/entrypoint]]"
[& args]
;; We need to install the classloader here before other namespaces are loaded since they could launch threads on load.
- ;; If a thread is spun uo and put back into a pool before this happens that pool will have a poisoned thread unable to
+ ;; If a thread is spun up and put back into a pool before this happens that pool will have a poisoned thread unable to
;; see classes in our classloader.
((requiring-resolve 'metabase.classloader.core/the-classloader))
- (apply (requiring-resolve 'metabase.core.core/entrypoint) args))
+ ;; Check for standalone modes first - these skip loading metabase.core.core
+ (let [{:keys [options]} (cli/parse-opts args [[nil "--mode MODE"]])
+ mode (:mode options)]
+ (if mode
+ (run-standalone-mode mode args)
+ (apply (requiring-resolve 'metabase.core.core/entrypoint) args))))
diff --git a/src/metabase/models/serialization.clj b/src/metabase/models/serialization.clj
index 75b6f6cc2e4..1252723ff11 100644
--- a/src/metabase/models/serialization.clj
+++ b/src/metabase/models/serialization.clj
@@ -70,6 +70,7 @@
[metabase.lib.schema.parameter :as lib.schema.parameter]
[metabase.lib.util.match :as lib.util.match]
[metabase.models.interface :as mi]
+ [metabase.models.serialization.resolve :as resolve]
[metabase.models.visualization-settings :as mb.viz]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
@@ -242,7 +243,7 @@
(defn identity-hash?
"Returns true if s is a valid identity hash string."
[s]
- (boolean (re-matches #"^[0-9a-fA-F]{8}$" s)))
+ (resolve/identity-hash? s))
;; ## Memoizing `hydrated-hash`
;;
@@ -450,7 +451,7 @@
(defmethod make-spec :default [_ _] nil)
-(defn ^:dynamic ^::cache *make-spec*
+(defn ^:dynamic *make-spec*
"Cachable wrapper around [[make-spec]] that is memoized inside [[with-cache]]."
[model-name opts]
(make-spec model-name opts))
@@ -849,9 +850,7 @@
(defn entity-id?
"Checks if the given string is a 21-character NanoID. Useful for telling entity IDs apart from identity hashes."
[id-str]
- (boolean (and id-str
- (string? id-str)
- (re-matches #"^[A-Za-z0-9_-]{21}$" id-str))))
+ (resolve/entity-id? id-str))
;; TODO: Clean up this [[identity-hash]] infrastructure once the `seed_entity_ids` issue is fixed. See above on the
;; details of the two hashing schemes.
@@ -929,32 +928,33 @@
:unique-name-fns (atom {})}))
;;; # Utilities for implementing serdes
-;;; Note that many of these use `^::cache` to cache their lookups during deserialization. This greatly reduces the
-;;; number of database lookups, since many entities might belong to eg. a single collection.
+;;; These wrapper functions delegate to the current resolver (set by [[with-cache]]).
+;;; When no resolver is bound, they fall back to the database-backed resolver.
+
+;; TODO: `requiring-resolve` is needed here because resolve.db requires this ns
+;; (for `generate-path`, `field-hierarchy`, `lookup-by-id`, `recursively-find-field-q`).
+;; Moving those into resolve.db (or a shared utils ns) would break the cycle and
+;; let us require resolve.db directly.
+(defn- export-resolver []
+ (or resolve/*export-resolver*
+ @(requiring-resolve 'metabase.models.serialization.resolve.db/default-export-resolver)))
+
+(defn- import-resolver []
+ (or resolve/*import-resolver*
+ @(requiring-resolve 'metabase.models.serialization.resolve.db/default-import-resolver)))
;;; ## General foreign keys
-(mu/defn ^:dynamic ^::cache *export-fk*
+(mu/defn ^:dynamic *export-fk*
"Given a numeric foreign key and its model (symbol, name or IModel), looks up the entity by ID and gets its entity ID
or identity hash.
- Unusual parameter order means this can be used as `(update x :some_id export-fk 'SomeModel)`.
+ Unusual parameter order means this can be used as `(update x :some_id *export-fk* 'SomeModel)`.
NOTE: This works for both top-level and nested entities. Top-level entities like `Card` are returned as just a
portable ID string.. Nested entities are returned as a vector of such ID strings."
[id :- [:maybe pos-int?]
model :- ::model-keyword-or-symbol]
- (when id
- (let [model-name (name model)
- entity (t2/select-one model (first (t2/primary-keys model)) id)
- path (when entity
- (mapv :id (generate-path model-name entity)))]
- (cond
- (nil? entity) (throw (ex-info "FK target not found" {:model model
- :id id
- :skip true
- ::type :target-not-found}))
- (= (count path) 1) (first path)
- :else path))))
+ (resolve/export-fk (export-resolver) id model))
(defmacro ^:private fk-elide
"If a call to `*export-fk*` inside of this fails, do not export the whole data structure"
@@ -967,7 +967,7 @@
(throw e#))
nil)))
-(mu/defn ^:dynamic ^::cache *import-fk*
+(mu/defn ^:dynamic *import-fk*
"Given an identifier, and the model it represents (symbol, name or IModel), looks up the corresponding
entity and gets its primary key.
@@ -980,67 +980,56 @@
Unusual parameter order means this can be used as `(update x :some_id import-fk 'SomeModel)`."
[eid
model :- ::model-keyword-or-symbol]
- (when eid
- (let [eid (if (vector? eid)
- (last eid)
- eid)
- entity (lookup-by-id model eid)]
- (if entity
- (get entity (first (t2/primary-keys model)))
- (throw (ex-info "Could not find foreign key target - bad serdes dependencies or other serialization error"
- {:entity_id eid :model (name model)}))))))
-
-(mu/defn ^:dynamic ^::cache *export-fk-keyed*
+ (resolve/import-fk (import-resolver) eid model))
+
+(mu/defn ^:dynamic *export-fk-keyed*
"Given a numeric ID, look up a different identifying field for that entity, and return it as a portable ID.
Eg. `Database.name`.
- [[import-fk-keyed]] is the inverse.
- Unusual parameter order lets this be called as, for example, `(update x :db_id export-fk-keyed :model/Database :name)`.
+ [[*import-fk-keyed*]] is the inverse.
+ Unusual parameter order lets this be called as, for example, `(update x :db_id *export-fk-keyed* :model/Database :name)`.
Note: This assumes the primary key is called `:id`."
[id
model :- ::model-keyword-or-symbol
field]
- (t2/select-one-fn field model :id id))
+ (resolve/export-fk-keyed (export-resolver) id model field))
-(defn ^:dynamic ^::cache *import-fk-keyed*
+(defn ^:dynamic *import-fk-keyed*
"Given a single, portable, identifying field and the model it refers to, this resolves the entity and returns its
numeric `:id`.
Eg. `Database.name`.
Unusual parameter order lets this be called as, for example,
- `(update x :creator_id import-fk-keyed :model/Database :name)`."
+ `(update x :creator_id *import-fk-keyed* :model/Database :name)`."
[portable model field]
- (t2/select-one-pk model field portable))
+ (resolve/import-fk-keyed (import-resolver) portable model field))
;;; ## Users
-(mu/defn ^:dynamic ^::cache *export-user*
+(mu/defn ^:dynamic *export-user*
"Exports a user as the email address.
- This just calls [[export-fk-keyed]], but the counterpart [[import-user]] is more involved. This is a unique function
+ This just calls [[*export-fk-keyed*]], but the counterpart [[*import-user*]] is more involved. This is a unique function
so they form a pair."
[id :- [:maybe ::lib.schema.id/user]]
- (when id (*export-fk-keyed* id 'User :email)))
+ (resolve/export-user (export-resolver) id))
-(mu/defn ^:dynamic ^::cache *import-user*
+(mu/defn ^:dynamic *import-user*
"Imports a user by their email address.
If a user with that email address exists, returns its primary key.
If no such user exists, creates a dummy inactive one with the default settings, blank name, and randomized password.
Does not send any invite emails."
[email :- [:maybe string?]]
- (when email
- (or (*import-fk-keyed* email 'User :email)
- ;; Need to break a circular dependency here.
- (:id ((resolve 'metabase.users.models.user/serdes-synthesize-user!) {:email email :is_active false})))))
+ (resolve/import-user (import-resolver) email))
;;; ## Databases
-(defn ^:dynamic ^::cache *export-database-fk*
+(defn ^:dynamic *export-database-fk*
"Given a numeric database ID, return its name as a portable reference.
[[*import-database-fk*]] is the inverse."
[id]
(when id
(t2/select-one-fn :name [:model/Database :id :name] :id id)))
-(defn ^:dynamic ^::cache *import-database-fk*
+(defn ^:dynamic *import-database-fk*
"Given a portable database name, resolve it back to a numeric ID.
[[*export-database-fk*]] is the inverse."
[db-name]
@@ -1048,29 +1037,22 @@
;;; ## Tables
-(mu/defn ^:dynamic ^::cache *export-table-fk*
+(mu/defn ^:dynamic *export-table-fk*
"Given a numeric `table_id`, return a portable table reference.
If the `table_id` is `nil`, return `nil`. This is legal for a native question.
That has the form `[db-name schema table-name]`, where the `schema` might be nil.
- [[import-table-fk]] is the inverse."
+ [[*import-table-fk*]] is the inverse."
[table-id :- [:maybe ::lib.schema.id/table]]
(when table-id
(let [{:keys [db_id name schema]} (t2/select-one [:model/Table :id :db_id :name :schema] :id table-id)
db-name (*export-database-fk* db_id)]
[db-name schema name])))
-(mu/defn ^:dynamic ^::cache *import-table-fk*
- "Given a `table_id` as exported by [[export-table-fk]], resolve it back into a numeric `table_id`.
+(mu/defn ^:dynamic *import-table-fk*
+ "Given a `table_id` as exported by [[*export-table-fk*]], resolve it back into a numeric `table_id`.
The input might be nil, in which case so is the output. This is legal for a native question."
- [[db-name schema table-name :as table-id] :- [:maybe [:tuple string? [:maybe string?] string?]]]
- (when table-id
- (if-let [db-id (t2/select-one-fn :id :model/Database :name db-name)]
- (or (t2/select-one-fn :id :model/Table :name table-name :schema schema :db_id db-id)
- (throw (ex-info (format "table id present, but no table found: %s" table-id)
- {:table-id table-id})))
- (throw (ex-info (format "table id present, but database not found: %s" table-id)
- {:table-id table-id
- :database-names (sort (t2/select-fn-vec :name :model/Table))})))))
+ [[_db-name _schema _table-name :as table-id] :- [:maybe [:tuple string? [:maybe string?] string?]]]
+ (resolve/import-table-fk (import-resolver) table-id))
(defn table->path
"Given a `table_id` as exported by [[export-table-fk]], turn it into a `[{:model ...}]` path for the Table.
@@ -1102,7 +1084,9 @@
;;; ## Fields
-(defn- field-hierarchy [id]
+(defn field-hierarchy
+ "Returns the field hierarchy (field + parents) for a field ID. Used by resolvers."
+ [id]
(reverse
(t2/select :model/Field
{:with-recursive [[[:parents {:columns [:id :name :parent_id :table_id]}]
@@ -1128,26 +1112,20 @@
[:= :name field]
[:= :parent_id (recursively-find-field-q table-id rest)]]}))
-(mu/defn ^:dynamic ^::cache *export-field-fk*
+(mu/defn ^:dynamic *export-field-fk*
"Given a numeric `field_id`, return a portable field reference.
That has the form `[db-name schema table-name field-name]`, where the `schema` might be nil.
[[*import-field-fk*]] is the inverse."
[field-id :- [:maybe ::lib.schema.id/field]]
(when field-id
- (let [{field-name :name :keys [parent_id table_id]} (t2/select-one [:model/Field :name :parent_id :table_id] :id field-id)]
- (if (nil? parent_id)
- (conj (*export-table-fk* table_id) field-name)
- (let [fields (field-hierarchy field-id)
- [db-name schema table-name] (*export-table-fk* table_id)]
- (into [db-name schema table-name] (map :name fields)))))))
-
-(mu/defn ^:dynamic ^::cache *import-field-fk* :- [:maybe pos-int?]
+ (let [fields (field-hierarchy field-id)
+ [db-name schema table-name] (*export-table-fk* (:table_id (first fields)))]
+ (into [db-name schema table-name] (map :name fields)))))
+
+(mu/defn ^:dynamic *import-field-fk*
"Given a `field_id` as exported by [[*export-field-fk*]], resolve it back into a numeric `field_id`."
- [[db-name schema table-name & fields :as field-id] :- [:maybe [:cat string? [:maybe string?] string? #_fields [:+ string?]]]]
- (when field-id
- (let [table-id (*import-table-fk* [db-name schema table-name])
- field-q (recursively-find-field-q table-id (reverse fields))]
- (t2/select-one-pk :model/Field field-q))))
+ [[_db-name _schema _table-name & _fields :as field-id] :- [:maybe [:cat string? [:maybe string?] string? #_fields [:+ string?]]]]
+ (resolve/import-field-fk (import-resolver) field-id))
(defn field->path
"Given a `field_id` as exported by [[export-field-fk]], turn it into a `[{:model ...}]` path for the Field.
@@ -1936,12 +1914,8 @@
;;; ## Memoizing appdb lookups
(defmacro with-cache
- "Runs body with all functions marked with ::cache re-bound to memoized versions for performance."
+ "Runs body with resolvers bound to cached (memoized) versions for performance."
[& body]
- (let [ns* 'metabase.models.serialization]
- `(binding ~(reduce into []
- (for [[var-sym var] (ns-interns ns*)
- :when (::cache (meta var))
- :let [fq-sym (symbol (name ns*) (name var-sym))]]
- [fq-sym `(memoize ~fq-sym)]))
- ~@body)))
+ `(binding [resolve/*export-resolver* ((requiring-resolve 'metabase.models.serialization.resolve.db/cached-export-resolver))
+ resolve/*import-resolver* ((requiring-resolve 'metabase.models.serialization.resolve.db/cached-import-resolver))]
+ ~@body))
diff --git a/src/metabase/models/serialization/resolve.clj b/src/metabase/models/serialization/resolve.clj
new file mode 100644
index 00000000000..84f7b8786b3
--- /dev/null
+++ b/src/metabase/models/serialization/resolve.clj
@@ -0,0 +1,252 @@
+(ns metabase.models.serialization.resolve
+ "Protocols, dynamic vars, and portable-ID utilities for serdes FK resolution.
+
+ This namespace has NO toucan2 dependency — it can be used by lightweight consumers
+ (like the checker) that don't need the database."
+ (:require
+ ^{:clj-kondo/ignore [:discouraged-namespace]} [metabase.legacy-mbql.schema :as mbql.s] ;; legacy usages -- do not use in new code
+ [metabase.lib.core :as lib]
+ [metabase.lib.schema.id :as lib.schema.id]
+ [metabase.lib.schema.parameter :as lib.schema.parameter]
+ [metabase.lib.util.match :as lib.util.match]
+ [metabase.models.visualization-settings :as mb.viz]
+ [metabase.util.malli :as mu]))
+
+(set! *warn-on-reflection* true)
+
+;;; ============================================================
+;;; Protocols
+;;; ============================================================
+
+(defprotocol SerdesExportResolver
+ "Resolve database IDs to portable references during export."
+ (export-fk [this id model] "Given a numeric FK and model, return a portable ID (string or vector).")
+ (export-fk-keyed [this id model field] "Given a numeric ID, look up a different identifying field.")
+ (export-user [this id] "Export a user as their email address.")
+ (export-table-fk [this table-id] "Given a numeric table_id, return [db-name schema table-name].")
+ (export-field-fk [this field-id] "Given a numeric field_id, return [db-name schema table-name field-name]."))
+
+(defprotocol SerdesImportResolver
+ "Resolve portable references back to database IDs during import."
+ (import-fk [this eid model] "Given a portable ID and model, return the numeric PK.")
+ (import-fk-keyed [this portable model field] "Given a portable identifying field value, return the numeric :id.")
+ (import-user [this email] "Import a user by email, creating if needed. Returns PK.")
+ (import-table-fk [this path] "Given [db-name schema table-name], return numeric table_id.")
+ (import-field-fk [this path] "Given [db-name schema table-name field-name], return numeric field_id."))
+
+;;; ============================================================
+;;; Dynamic vars — bound to resolver instances
+;;; ============================================================
+
+(def ^:dynamic *export-resolver*
+ "The current `SerdesExportResolver` instance. Bound during export."
+ nil)
+
+(def ^:dynamic *import-resolver*
+ "The current `SerdesImportResolver` instance. Bound during import."
+ nil)
+
+;;; ============================================================
+;;; Pure predicates
+;;; ============================================================
+
+(defn entity-id?
+ "Checks if the given string is a 21-character NanoID."
+ [id-str]
+ (boolean (and id-str
+ (string? id-str)
+ (re-matches #"^[A-Za-z0-9_-]{21}$" id-str))))
+
+(defn identity-hash?
+ "Returns true if s is a valid identity hash string."
+ [s]
+ (boolean (re-matches #"^[0-9a-fA-F]{8}$" s)))
+
+(defn- portable-id?
+ "True if the provided string is either an Entity ID or identity-hash string."
+ [s]
+ (and (string? s)
+ (or (entity-id? s)
+ (identity-hash? s))))
+
+;;; ============================================================
+;;; import-mbql — depends only on protocols, lib.util.match, lib.schema.id
+;;; ============================================================
+
+(defn- mbql-fully-qualified-names->ids*
+ [resolver entity]
+ (lib.util.match/replace-lite entity
+ [#{:field "field"} (opts :guard map?) (fully-qualified-name :guard vector?)]
+ [:field (mbql-fully-qualified-names->ids* resolver opts)
+ (import-field-fk resolver fully-qualified-name)]
+
+ ;; legacy field refs, still used in parameters and result metadata `field_ref`
+ [#{:field "field"} (fully-qualified-name :guard vector?) (opts :guard (some-fn map? nil))]
+ [:field (import-field-fk resolver fully-qualified-name) (some->> opts (mbql-fully-qualified-names->ids* resolver))]
+
+ ;; MBQL 3 `:field-id` can (allegedly) still show up sometimes? Support it just in case.
+ [(tag :guard #{:field :field-id "field" "field-id"}) (id :guard vector?)]
+ [:field (import-field-fk resolver id) nil]
+
+ ;; source-field is also used within parameter mapping dimensions
+ ;; example relevant clause - [:field 2 {:source-field 1}]
+ {:source-field (fully-qualified-name :guard vector?)}
+ (assoc &match :source-field (import-field-fk resolver fully-qualified-name))
+
+ {:database (fully-qualified-name :guard string?)}
+ (-> &match
+ (assoc :database (if (= fully-qualified-name "database/__virtual")
+ lib.schema.id/saved-questions-virtual-database-id
+ (import-fk-keyed resolver fully-qualified-name :model/Database :name)))
+ (->> (mbql-fully-qualified-names->ids* resolver)))
+
+ {:card-id (entity-id :guard portable-id?)}
+ (-> &match
+ (assoc :card-id (import-fk resolver entity-id 'Card))
+ (->> (mbql-fully-qualified-names->ids* resolver)))
+
+ [#{:metric "metric"} opts (entity-id :guard portable-id?)]
+ [:metric (mbql-fully-qualified-names->ids* resolver opts)
+ (import-fk resolver entity-id 'Card)]
+
+ [#{:segment "segment"} opts (entity-id :guard portable-id?)]
+ [:segment (mbql-fully-qualified-names->ids* resolver opts)
+ (import-fk resolver entity-id 'Segment)]
+
+ [#{:measure "measure"} opts (entity-id :guard portable-id?)]
+ [:measure (mbql-fully-qualified-names->ids* resolver opts)
+ (import-fk resolver entity-id 'Measure)]
+
+ ;; support legacy MBQL 4 refs for things like the serialized Audit v2 queries
+ [#{:metric "metric"} (entity-id :guard portable-id?)]
+ [:metric (import-fk resolver entity-id 'Card)]
+
+ [#{:segment "segment"} (entity-id :guard portable-id?)]
+ [:segment (import-fk resolver entity-id 'Segment)]
+
+ [#{:measure "measure"} (entity-id :guard portable-id?)]
+ [:measure (import-fk resolver entity-id 'Measure)]
+
+ {:source-table (_ :guard vector?)}
+ (-> &match
+ (update :source-table (partial import-table-fk resolver))
+ (->> (mbql-fully-qualified-names->ids* resolver)))
+
+ {:source_table (_ :guard vector?)}
+ (-> &match
+ (update :source_table (partial import-table-fk resolver))
+ (->> (mbql-fully-qualified-names->ids* resolver)))
+
+ ;; support legacy MBQL 4 for the Audit v2 queries
+ {:source-table (id :guard portable-id?)}
+ (-> &match
+ (assoc :source-table (str "card__" (import-fk resolver id 'Card)))
+ (->> (mbql-fully-qualified-names->ids* resolver)))
+
+ {:source-card (id :guard portable-id?)}
+ (-> &match
+ (assoc :source-card (import-fk resolver id 'Card))
+ (->> (mbql-fully-qualified-names->ids* resolver)))
+
+ {:snippet-id (id :guard portable-id?)}
+ (-> &match
+ (assoc :snippet-id (import-fk resolver id 'NativeQuerySnippet))
+ (->> (mbql-fully-qualified-names->ids* resolver)))))
+
+(defn import-mbql
+ "Given an MBQL expression with portable IDs, convert back to numeric IDs.
+ Uses `*import-resolver*` if bound, otherwise requires a resolver argument."
+ ([exported]
+ (import-mbql *import-resolver* exported))
+ ([resolver exported]
+ (mbql-fully-qualified-names->ids* resolver exported)))
+
+(mu/defn- mbql-clause-tag :- [:maybe [:enum :field :dimension :metric :segment :measure]]
+ "Is given form an MBQL entity reference?"
+ [form]
+ (when (and (vector? form)
+ (#{:field :dimension :metric :segment :measure} (keyword (first form))))
+ (keyword (first form))))
+
+(defn- normalize [mbql]
+ (let [tag (mbql-clause-tag mbql)
+ schema (case tag
+ :field [:multi
+ {:dispatch #(and (vector? %)
+ (map? (second %)))}
+ [true :mbql.clause/field]
+ [false ::mbql.s/field]] ; legacy MBQL clause
+ :dimension ::lib.schema.parameter/dimension
+ :metric :mbql.clause/metric
+ :segment :mbql.clause/segment
+ :measure :mbql.clause/measure
+ #_else nil)]
+ (cond->> mbql
+ schema (lib/normalize schema mbql))))
+
+(defn- mbql-id->fully-qualified-name
+ [resolver mbql]
+ (lib.util.match/replace-lite (normalize mbql)
+ ;; `pos-int?` guard is here to make the operation idempotent
+ [:field (opts :guard map?) (id :guard pos-int?)]
+ [:field (mbql-id->fully-qualified-name resolver opts) (export-field-fk resolver id)]
+
+ ;; legacy (MBQL 4) field refs are still supported in parameter targets and in result metadata `field_ref`...
+ [:field (id :guard pos-int?) (opts :guard (some-fn map? nil?))]
+ [:field (export-field-fk resolver id) (mbql-id->fully-qualified-name resolver opts)]
+
+ ;; MBQL 3 `:field-id` can (allegedly) still show up sometimes? Support it just in case.
+ [(tag :guard #{:field :field-id}) (id :guard pos-int?)]
+ [tag (export-field-fk resolver id)]
+
+ {:source-table (id :guard pos-int?)}
+ (assoc &match :source-table (export-table-fk resolver id))
+
+ ;; source-field is also used within parameter mapping dimensions
+ ;; example relevant clause - [:field 2 {:source-field 1}]
+ {:source-field (id :guard pos-int?)}
+ (assoc &match :source-field (export-field-fk resolver id))
+
+ [:dimension (dim :guard vector?)]
+ [:dimension (mbql-id->fully-qualified-name resolver dim)]
+
+ [:metric opts (id :guard pos-int?)]
+ [:metric (mbql-id->fully-qualified-name resolver opts) (export-fk resolver id 'Card)]
+
+ [:segment opts (id :guard pos-int?)]
+ [:segment (mbql-id->fully-qualified-name resolver opts) (export-fk resolver id 'Segment)]
+
+ [:measure opts (id :guard pos-int?)]
+ [:measure (mbql-id->fully-qualified-name resolver opts) (export-fk resolver id 'Measure)]))
+
+(defn export-mbql
+ "Given an MBQL expression, convert it to an EDN structure and turn the non-portable Database, Table and Field IDs
+ inside it into portable references."
+ ([entity] (export-mbql *export-resolver* entity))
+ ([resolver entity]
+ (lib.util.match/replace-lite entity
+ (_ :guard mbql-clause-tag)
+ (mbql-id->fully-qualified-name resolver &match)
+
+ (_ :guard sequential?)
+ (mapv export-mbql &match)
+
+ (_ :guard map?)
+ (reduce-kv
+ (fn [entity k _v]
+ (let [f (case k
+ :database (fn [db-id]
+ (if (= db-id lib.schema.id/saved-questions-virtual-database-id)
+ "database/__virtual"
+ (export-fk-keyed resolver db-id :model/Database :name)))
+ (:card_id :card-id) #(export-fk resolver % :model/Card) ; attributes that refer to db fields use `_`; template-tags use `-`
+ (:source_table :source-table) #(export-table-fk resolver %)
+ (:source_card :source-card) #(export-fk resolver % :model/Card)
+ ::mb.viz/param-mapping-source #(export-field-fk resolver %)
+ :segment #(export-fk resolver % :model/Segment)
+ :snippet-id #(export-fk resolver % :model/NativeQuerySnippet)
+ :lib/metadata (constantly nil)
+ #_else (partial export-mbql resolver))]
+ (update entity k f)))
+ &match
+ &match))))
diff --git a/src/metabase/models/serialization/resolve/db.clj b/src/metabase/models/serialization/resolve/db.clj
new file mode 100644
index 00000000000..89f52e575fd
--- /dev/null
+++ b/src/metabase/models/serialization/resolve/db.clj
@@ -0,0 +1,157 @@
+(ns metabase.models.serialization.resolve.db
+ "Database-backed implementations of the serdes resolver protocols.
+
+ These are the default resolvers used during normal serdes export/import
+ against the application database."
+ (:require
+ [metabase.models.serialization :as serdes]
+ [metabase.models.serialization.resolve :as resolve]
+ [toucan2.core :as t2]))
+
+(set! *warn-on-reflection* true)
+
+;;; ============================================================
+;;; Export functions
+;;; ============================================================
+
+(defn export-fk
+ "Given a numeric foreign key and its model, looks up the entity by ID and gets its entity ID or identity hash."
+ [id model]
+ (when id
+ (let [model-name (name model)
+ entity (t2/select-one model (first (t2/primary-keys model)) id)
+ path (when entity
+ (mapv :id (serdes/generate-path model-name entity)))]
+ (cond
+ (nil? entity) (throw (ex-info "FK target not found" {:model model
+ :id id
+ :skip true
+ :metabase.models.serialization/type :target-not-found}))
+ (= (count path) 1) (first path)
+ :else path))))
+
+(defn export-fk-keyed
+ "Given a numeric ID, look up a different identifying field for that entity."
+ [id model field]
+ (t2/select-one-fn field model :id id))
+
+(defn export-user
+ "Export a user as their email address."
+ [resolver id]
+ (when id (resolve/export-fk-keyed resolver id 'User :email)))
+
+(defn export-table-fk
+ "Given a numeric table_id, return a portable table reference [db-name schema table-name]."
+ [table-id]
+ (when table-id
+ (let [{:keys [db_id name schema]} (t2/select-one :model/Table :id table-id)
+ db-name (t2/select-one-fn :name :model/Database :id db_id)]
+ [db-name schema name])))
+
+(defn export-field-fk
+ "Given a numeric field_id, return a portable field reference [db-name schema table-name field-name]."
+ [resolver field-id]
+ (when field-id
+ (let [fields (serdes/field-hierarchy field-id)
+ [db-name schema field-name] (resolve/export-table-fk resolver (:table_id (first fields)))]
+ (into [db-name schema field-name] (map :name fields)))))
+
+;;; ============================================================
+;;; Import functions
+;;; ============================================================
+
+(defn import-fk
+ "Given a portable ID and model, return the numeric PK."
+ [eid model]
+ (when eid
+ (let [eid (if (vector? eid) (last eid) eid)
+ entity (serdes/lookup-by-id model eid)]
+ (if entity
+ (get entity (first (t2/primary-keys model)))
+ (throw (ex-info "Could not find foreign key target - bad serdes dependencies or other serialization error"
+ {:entity_id eid :model (name model)}))))))
+
+(defn import-fk-keyed
+ "Given a portable identifying field value, return the numeric :id."
+ [portable model field]
+ (t2/select-one-pk model field portable))
+
+(defn import-user
+ "Import a user by email, creating if needed. Returns PK."
+ [resolver email]
+ (when email
+ (or (resolve/import-fk-keyed resolver email 'User :email)
+ ;; Need to break a circular dependency here.
+ (:id ((clojure.core/resolve 'metabase.users.models.user/serdes-synthesize-user!) {:email email :is_active false})))))
+
+(defn import-table-fk
+ "Given [db-name schema table-name], return numeric table_id."
+ [[db-name schema table-name :as table-id]]
+ (when table-id
+ (if-let [db-id (t2/select-one-fn :id :model/Database :name db-name)]
+ (or (t2/select-one-fn :id :model/Table :name table-name :schema schema :db_id db-id)
+ (throw (ex-info (format "table id present, but no table found: %s" table-id)
+ {:table-id table-id})))
+ (throw (ex-info (format "table id present, but database not found: %s" table-id)
+ {:table-id table-id
+ :database-names (sort (t2/select-fn-vec :name :model/Table))})))))
+
+(defn import-field-fk
+ "Given [db-name schema table-name field-name ...], return numeric field_id."
+ [resolver [db-name schema table-name & fields :as field-id]]
+ (when field-id
+ (let [table-id (resolve/import-table-fk resolver [db-name schema table-name])
+ field-q (serdes/recursively-find-field-q table-id (reverse fields))]
+ (t2/select-one-pk :model/Field field-q))))
+
+;;; ============================================================
+;;; Resolver constructors
+;;; ============================================================
+
+(def default-export-resolver
+ "A stateless database-backed export resolver."
+ (reify resolve/SerdesExportResolver
+ (export-fk [_ id model] (export-fk id model))
+ (export-fk-keyed [_ id model field] (export-fk-keyed id model field))
+ (export-user [this id] (export-user this id))
+ (export-table-fk [_ table-id] (export-table-fk table-id))
+ (export-field-fk [this field-id] (export-field-fk this field-id))))
+
+(def default-import-resolver
+ "A stateless database-backed import resolver."
+ (reify resolve/SerdesImportResolver
+ (import-fk [_ eid model] (import-fk eid model))
+ (import-fk-keyed [_ portable model field] (import-fk-keyed portable model field))
+ (import-user [this email] (import-user this email))
+ (import-table-fk [_ path] (import-table-fk path))
+ (import-field-fk [this path] (import-field-fk this path))))
+
+(defn cached-export-resolver
+ "Returns a database-backed export resolver with memoized lookups."
+ []
+ (let [export-fk* (memoize export-fk)
+ export-fk-keyed* (memoize export-fk-keyed)
+ export-user* (memoize export-user)
+ export-table-fk* (memoize export-table-fk)
+ export-field-fk* (memoize export-field-fk)]
+ (reify resolve/SerdesExportResolver
+ (export-fk [_ id model] (export-fk* id model))
+ (export-fk-keyed [_ id model field] (export-fk-keyed* id model field))
+ (export-user [this id] (export-user* this id))
+ (export-table-fk [_ table-id] (export-table-fk* table-id))
+ (export-field-fk [this field-id] (export-field-fk* this field-id)))))
+
+(defn cached-import-resolver
+ "Returns a database-backed import resolver with memoized lookups."
+ []
+ (let [import-fk* (memoize import-fk)
+ import-fk-keyed* (memoize import-fk-keyed)
+ import-user* (memoize import-user)
+ import-table-fk* (memoize import-table-fk)
+ import-field-fk* (memoize import-field-fk)]
+ (reify resolve/SerdesImportResolver
+ (import-fk [_ eid model] (import-fk* eid model))
+ (import-fk-keyed [_ portable model field] (import-fk-keyed* portable model field))
+ (import-user [this email] (import-user* this email))
+ (import-table-fk [_ path] (import-table-fk* path))
+ (import-field-fk [this path] (import-field-fk* this path)))))
diff --git a/test_resources/yaml_checks/collections/cards/native-orders_native_orders.yaml b/test_resources/yaml_checks/collections/cards/native-orders_native_orders.yaml
new file mode 100644
index 00000000000..bc67d6d9905
--- /dev/null
+++ b/test_resources/yaml_checks/collections/cards/native-orders_native_orders.yaml
@@ -0,0 +1,81 @@
+name: Native Orders
+description: A native SQL query
+entity_id: native-orders
+created_at: '2026-01-01T00:00:00.000000Z'
+creator_id: [email protected]
+display: table
+archived: false
+collection_id: null
+collection_preview: true
+collection_position: null
+query_type: native
+database_id: Test Database
+table_id: null
+enable_embedding: false
+embedding_params: null
+made_public_by_id: null
+public_uuid: null
+parameters: []
+parameter_mappings: []
+dataset_query:
+ database: Test Database
+ native:
+ query: select * from orders
+ type: native
+result_metadata:
+- base_type: type/BigInteger
+ database_type: BIGINT
+ display_name: ID
+ effective_type: type/BigInteger
+ field_ref:
+ - field
+ - id
+ - base-type: type/BigInteger
+ name: id
+ semantic_type: type/PK
+ source: native
+- base_type: type/Float
+ database_type: DOUBLE PRECISION
+ display_name: total
+ effective_type: type/Float
+ field_ref:
+ - field
+ - total
+ - base-type: type/Float
+ name: total
+ source: native
+- base_type: type/Integer
+ database_type: INTEGER
+ display_name: product_id
+ effective_type: type/Integer
+ field_ref:
+ - field
+ - product_id
+ - base-type: type/Integer
+ name: product_id
+ source: native
+- base_type: type/DateTime
+ database_type: TIMESTAMP
+ display_name: created_at
+ effective_type: type/DateTime
+ field_ref:
+ - field
+ - created_at
+ - base-type: type/DateTime
+ name: created_at
+ semantic_type: type/CreationTimestamp
+ source: native
+visualization_settings:
+ column_settings: null
+card_schema: 23
+serdes/meta:
+- id: native-orders
+ label: native_orders
+ model: Card
+archived_directly: false
+dashboard_id: null
+document_id: null
+embedding_type: null
+metabase_version: v0.55.1-SNAPSHOT
+source_card_id: null
+type: question
diff --git a/test_resources/yaml_checks/collections/cards/orders-with-fk_orders_with_fk.yaml b/test_resources/yaml_checks/collections/cards/orders-with-fk_orders_with_fk.yaml
new file mode 100644
index 00000000000..4cd56b57269
--- /dev/null
+++ b/test_resources/yaml_checks/collections/cards/orders-with-fk_orders_with_fk.yaml
@@ -0,0 +1,66 @@
+name: Orders With FK
+description: Query with FK in result_metadata
+entity_id: orders-with-fk
+created_at: '2026-01-01T00:00:00.000000Z'
+creator_id: [email protected]
+display: table
+archived: false
+query_type: query
+database_id: SQLite DB
+table_id:
+- SQLite DB
+- null
+- orders
+dataset_query:
+ database: SQLite DB
+ query:
+ source-table:
+ - SQLite DB
+ - null
+ - orders
+ type: query
+result_metadata:
+- base_type: type/Integer
+ display_name: ID
+ effective_type: type/Integer
+ field_ref:
+ - field
+ - - SQLite DB
+ - null
+ - orders
+ - id
+ - null
+ id:
+ - SQLite DB
+ - null
+ - orders
+ - id
+ name: id
+ semantic_type: type/PK
+- base_type: type/Integer
+ display_name: User ID
+ effective_type: type/Integer
+ field_ref:
+ - field
+ - - SQLite DB
+ - null
+ - orders
+ - user_id
+ - null
+ fk_target_field_id:
+ - SQLite DB
+ - null
+ - users
+ - id
+ id:
+ - SQLite DB
+ - null
+ - orders
+ - user_id
+ name: user_id
+ semantic_type: type/FK
+serdes/meta:
+- id: orders-with-fk
+ label: orders_with_fk
+ model: Card
+type: question
diff --git a/test_resources/yaml_checks/collections/cards/orders-with-products_orders_with_products.yaml b/test_resources/yaml_checks/collections/cards/orders-with-products_orders_with_products.yaml
new file mode 100644
index 00000000000..4f03f128b5f
--- /dev/null
+++ b/test_resources/yaml_checks/collections/cards/orders-with-products_orders_with_products.yaml
@@ -0,0 +1,197 @@
+name: Orders With Products
+description: Orders joined with products table
+entity_id: orders-with-products
+created_at: '2026-01-01T00:00:00.000000Z'
+creator_id: [email protected]
+display: table
+archived: false
+collection_id: null
+collection_preview: true
+collection_position: null
+query_type: query
+database_id: Test Database
+table_id:
+- Test Database
+- public
+- orders
+enable_embedding: false
+embedding_params: null
+made_public_by_id: null
+public_uuid: null
+parameters: []
+parameter_mappings: []
+dataset_query:
+ database: Test Database
+ query:
+ fields:
+ - - field
+ - - Test Database
+ - public
+ - orders
+ - id
+ - base-type: type/BigInteger
+ - - field
+ - - Test Database
+ - public
+ - orders
+ - total
+ - base-type: type/Float
+ - - field
+ - - Test Database
+ - public
+ - products
+ - title
+ - base-type: type/Text
+ join-alias: Products
+ - - field
+ - - Test Database
+ - public
+ - products
+ - price
+ - base-type: type/Float
+ join-alias: Products
+ joins:
+ - alias: Products
+ condition:
+ - =
+ - - field
+ - - Test Database
+ - public
+ - orders
+ - product_id
+ - base-type: type/Integer
+ - - field
+ - - Test Database
+ - public
+ - products
+ - id
+ - base-type: type/BigInteger
+ join-alias: Products
+ fields: all
+ source-table:
+ - Test Database
+ - public
+ - products
+ strategy: left-join
+ source-table:
+ - Test Database
+ - public
+ - orders
+ type: query
+result_metadata:
+- active: true
+ base_type: type/BigInteger
+ database_type: BIGINT
+ display_name: ID
+ effective_type: type/BigInteger
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - orders
+ - id
+ - null
+ id:
+ - Test Database
+ - public
+ - orders
+ - id
+ name: id
+ position: 0
+ semantic_type: type/PK
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - orders
+ visibility_type: normal
+- active: true
+ base_type: type/Float
+ database_type: DOUBLE PRECISION
+ display_name: Total
+ effective_type: type/Float
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - orders
+ - total
+ - null
+ id:
+ - Test Database
+ - public
+ - orders
+ - total
+ name: total
+ position: 1
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - orders
+ visibility_type: normal
+- active: true
+ base_type: type/Text
+ database_type: VARCHAR
+ display_name: Products -> Title
+ effective_type: type/Text
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - products
+ - title
+ - join-alias: Products
+ id:
+ - Test Database
+ - public
+ - products
+ - title
+ name: title
+ position: 0
+ semantic_type: type/Title
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - products
+ visibility_type: normal
+- active: true
+ base_type: type/Float
+ database_type: DOUBLE PRECISION
+ display_name: Products -> Price
+ effective_type: type/Float
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - products
+ - price
+ - join-alias: Products
+ id:
+ - Test Database
+ - public
+ - products
+ - price
+ name: price
+ position: 1
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - products
+ visibility_type: normal
+visualization_settings:
+ column_settings: null
+card_schema: 23
+serdes/meta:
+- id: orders-with-products
+ label: orders_with_products
+ model: Card
+archived_directly: false
+dashboard_id: null
+document_id: null
+embedding_type: null
+metabase_version: v0.55.1-SNAPSHOT
+source_card_id: null
+type: question
diff --git a/test_resources/yaml_checks/collections/cards/schemaless-query_schemaless_query.yaml b/test_resources/yaml_checks/collections/cards/schemaless-query_schemaless_query.yaml
new file mode 100644
index 00000000000..18555cd1930
--- /dev/null
+++ b/test_resources/yaml_checks/collections/cards/schemaless-query_schemaless_query.yaml
@@ -0,0 +1,61 @@
+name: Schemaless Query
+description: Query on schema-less SQLite database
+entity_id: schemaless-query
+created_at: '2026-01-01T00:00:00.000000Z'
+creator_id: [email protected]
+display: table
+archived: false
+query_type: query
+database_id: SQLite DB
+table_id:
+- SQLite DB
+- null
+- users
+dataset_query:
+ database: SQLite DB
+ query:
+ source-table:
+ - SQLite DB
+ - null
+ - users
+ type: query
+result_metadata:
+- base_type: type/Integer
+ display_name: ID
+ effective_type: type/Integer
+ field_ref:
+ - field
+ - - SQLite DB
+ - null
+ - users
+ - id
+ - null
+ id:
+ - SQLite DB
+ - null
+ - users
+ - id
+ name: id
+ semantic_type: type/PK
+- base_type: type/Text
+ display_name: Name
+ effective_type: type/Text
+ field_ref:
+ - field
+ - - SQLite DB
+ - null
+ - users
+ - name
+ - null
+ id:
+ - SQLite DB
+ - null
+ - users
+ - name
+ name: name
+ semantic_type: type/Name
+serdes/meta:
+- id: schemaless-query
+ label: schemaless_query
+ model: Card
+type: question
diff --git a/test_resources/yaml_checks/collections/cards/simple-orders_simple_orders.yaml b/test_resources/yaml_checks/collections/cards/simple-orders_simple_orders.yaml
new file mode 100644
index 00000000000..cc209b395f4
--- /dev/null
+++ b/test_resources/yaml_checks/collections/cards/simple-orders_simple_orders.yaml
@@ -0,0 +1,153 @@
+name: Simple Orders
+description: A simple query on orders table
+entity_id: simple-orders
+created_at: '2026-01-01T00:00:00.000000Z'
+creator_id: [email protected]
+display: table
+archived: false
+collection_id: null
+collection_preview: true
+collection_position: null
+query_type: query
+database_id: Test Database
+table_id:
+- Test Database
+- public
+- orders
+enable_embedding: false
+embedding_params: null
+made_public_by_id: null
+public_uuid: null
+parameters: []
+parameter_mappings: []
+dataset_query:
+ database: Test Database
+ query:
+ source-table:
+ - Test Database
+ - public
+ - orders
+ type: query
+result_metadata:
+- active: true
+ base_type: type/BigInteger
+ database_type: BIGINT
+ display_name: ID
+ effective_type: type/BigInteger
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - orders
+ - id
+ - null
+ id:
+ - Test Database
+ - public
+ - orders
+ - id
+ name: id
+ position: 0
+ semantic_type: type/PK
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - orders
+ visibility_type: normal
+- active: true
+ base_type: type/Float
+ database_type: DOUBLE PRECISION
+ display_name: Total
+ effective_type: type/Float
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - orders
+ - total
+ - null
+ id:
+ - Test Database
+ - public
+ - orders
+ - total
+ name: total
+ position: 1
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - orders
+ visibility_type: normal
+- active: true
+ base_type: type/Integer
+ database_type: INTEGER
+ display_name: Product ID
+ effective_type: type/Integer
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - orders
+ - product_id
+ - null
+ fk_target_field_id:
+ - Test Database
+ - public
+ - products
+ - id
+ id:
+ - Test Database
+ - public
+ - orders
+ - product_id
+ name: product_id
+ position: 2
+ semantic_type: type/FK
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - orders
+ visibility_type: normal
+- active: true
+ base_type: type/DateTime
+ database_type: TIMESTAMP
+ display_name: Created At
+ effective_type: type/DateTime
+ field_ref:
+ - field
+ - - Test Database
+ - public
+ - orders
+ - created_at
+ - null
+ id:
+ - Test Database
+ - public
+ - orders
+ - created_at
+ name: created_at
+ position: 3
+ semantic_type: type/CreationTimestamp
+ source: fields
+ table_id:
+ - Test Database
+ - public
+ - orders
+ visibility_type: normal
+visualization_settings:
+ column_settings: null
+card_schema: 23
+serdes/meta:
+- id: simple-orders
+ label: simple_orders
+ model: Card
+archived_directly: false
+dashboard_id: null
+document_id: null
+embedding_type: null
+metabase_version: v0.55.1-SNAPSHOT
+source_card_id: null
+type: question
diff --git a/test_resources/yaml_checks/databases/SQLite DB/SQLite DB.yaml b/test_resources/yaml_checks/databases/SQLite DB/SQLite DB.yaml
new file mode 100644
index 00000000000..bd75c443074
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/SQLite DB.yaml
@@ -0,0 +1,7 @@
+name: SQLite DB
+engine: sqlite
+created_at: '2026-01-01T00:00:00.000000Z'
+settings: null
+serdes/meta:
+- id: SQLite DB
+ model: Database
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/id.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/id.yaml
new file mode 100644
index 00000000000..34fcc5de514
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/id.yaml
@@ -0,0 +1,43 @@
+name: id
+display_name: ID
+description: Primary key
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- SQLite DB
+- null
+- orders
+database_type: INTEGER
+base_type: type/Integer
+effective_type: null
+semantic_type: type/PK
+database_is_auto_increment: true
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 0
+custom_position: 0
+database_position: 0
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: SQLite DB
+ model: Database
+- id: orders
+ model: Table
+- id: id
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: false
+database_is_pk: true
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/user_id.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/user_id.yaml
new file mode 100644
index 00000000000..53ab777bb7e
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/user_id.yaml
@@ -0,0 +1,47 @@
+name: user_id
+display_name: User ID
+description: Foreign key to users table
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- SQLite DB
+- null
+- orders
+database_type: INTEGER
+base_type: type/Integer
+effective_type: null
+semantic_type: type/FK
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id:
+- SQLite DB
+- null
+- users
+- id
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 1
+custom_position: 0
+database_position: 1
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: SQLite DB
+ model: Database
+- id: orders
+ model: Table
+- id: user_id
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/orders.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/orders.yaml
new file mode 100644
index 00000000000..9e6bd0b675e
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/orders.yaml
@@ -0,0 +1,13 @@
+name: orders
+display_name: Orders
+description: Orders table
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: null
+schema: null
+db_id: SQLite DB
+serdes/meta:
+- id: SQLite DB
+ model: Database
+- id: orders
+ model: Table
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/segments/recent_orders.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/segments/recent_orders.yaml
new file mode 100644
index 00000000000..4e44ec6479a
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/segments/recent_orders.yaml
@@ -0,0 +1,21 @@
+name: Recent Orders
+entity_id: recent-orders-segment
+definition:
+ database: SQLite DB
+ stages:
+ - filters:
+ - - ">"
+ - {}
+ - - field
+ - {}
+ - created_at
+ - 2025-01-01
+ source-table:
+ - SQLite DB
+ - null
+ - orders
+ lib/type: mbql.stage/mbql
+ lib/type: mbql/query
+serdes/meta:
+- id: recent-orders-segment
+ model: Segment
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/id.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/id.yaml
new file mode 100644
index 00000000000..6a2e07a3e2d
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/id.yaml
@@ -0,0 +1,43 @@
+name: id
+display_name: ID
+description: Primary key
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- SQLite DB
+- null
+- users
+database_type: INTEGER
+base_type: type/Integer
+effective_type: null
+semantic_type: type/PK
+database_is_auto_increment: true
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 0
+custom_position: 0
+database_position: 0
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: SQLite DB
+ model: Database
+- id: users
+ model: Table
+- id: id
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: false
+database_is_pk: true
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/name.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/name.yaml
new file mode 100644
index 00000000000..a48af8a14f8
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/name.yaml
@@ -0,0 +1,43 @@
+name: name
+display_name: Name
+description: User name
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- SQLite DB
+- null
+- users
+database_type: TEXT
+base_type: type/Text
+effective_type: null
+semantic_type: type/Name
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 1
+custom_position: 0
+database_position: 1
+has_field_values: auto-list
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: SQLite DB
+ model: Database
+- id: users
+ model: Table
+- id: name
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/users/users.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/users/users.yaml
new file mode 100644
index 00000000000..090f0d17259
--- /dev/null
+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/users/users.yaml
@@ -0,0 +1,13 @@
+name: users
+display_name: Users
+description: User accounts table
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: null
+schema: null
+db_id: SQLite DB
+serdes/meta:
+- id: SQLite DB
+ model: Database
+- id: users
+ model: Table
diff --git a/test_resources/yaml_checks/databases/Test Database/Test Database.yaml b/test_resources/yaml_checks/databases/Test Database/Test Database.yaml
new file mode 100644
index 00000000000..fce6f68cae9
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/Test Database.yaml
@@ -0,0 +1,33 @@
+name: Test Database
+description: A test database for CI checker tests
+engine: h2
+dbms_version:
+ flavor: H2
+ version: 2.1.214 (2022-06-13)
+ semantic-version:
+ - 2
+ - 1
+created_at: '2026-01-01T00:00:00.000000Z'
+creator_id: [email protected]
+timezone: UTC
+auto_run_queries: true
+refingerprint: null
+is_full_sync: true
+is_on_demand: false
+is_sample: false
+is_audit: false
+is_attached_dwh: false
+metadata_sync_schedule: 0 0 * * * ? *
+cache_field_values_schedule: 0 0 0 * * ? *
+settings: null
+caveats: null
+points_of_interest: null
+initial_sync_status: complete
+serdes/meta:
+- id: Test Database
+ model: Database
+provider_name: null
+router_database_id: null
+uploads_enabled: false
+uploads_schema_name: null
+uploads_table_prefix: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/created_at.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/created_at.yaml
new file mode 100644
index 00000000000..29f6ed79e2e
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/created_at.yaml
@@ -0,0 +1,45 @@
+name: created_at
+display_name: Created At
+description: When the order was created
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- orders
+database_type: TIMESTAMP
+base_type: type/DateTime
+effective_type: null
+semantic_type: type/CreationTimestamp
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 3
+custom_position: 0
+database_position: 3
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: orders
+ model: Table
+- id: created_at
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/id.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/id.yaml
new file mode 100644
index 00000000000..523095c67a9
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/id.yaml
@@ -0,0 +1,45 @@
+name: id
+display_name: ID
+description: Primary key for orders
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- orders
+database_type: BIGINT
+base_type: type/BigInteger
+effective_type: null
+semantic_type: type/PK
+database_is_auto_increment: true
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 0
+custom_position: 0
+database_position: 0
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: orders
+ model: Table
+- id: id
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: false
+database_is_pk: true
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/product_id.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/product_id.yaml
new file mode 100644
index 00000000000..dad77cd1f6b
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/product_id.yaml
@@ -0,0 +1,49 @@
+name: product_id
+display_name: Product ID
+description: Foreign key to products table
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- orders
+database_type: INTEGER
+base_type: type/Integer
+effective_type: null
+semantic_type: type/FK
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id:
+- Test Database
+- public
+- products
+- id
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 2
+custom_position: 0
+database_position: 2
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: orders
+ model: Table
+- id: product_id
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/total.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/total.yaml
new file mode 100644
index 00000000000..7dc2eb020e0
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/total.yaml
@@ -0,0 +1,45 @@
+name: total
+display_name: Total
+description: Order total amount
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- orders
+database_type: DOUBLE PRECISION
+base_type: type/Float
+effective_type: null
+semantic_type: null
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 1
+custom_position: 0
+database_position: 1
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: orders
+ model: Table
+- id: total
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/measures/total_revenue.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/measures/total_revenue.yaml
new file mode 100644
index 00000000000..09908d42fa7
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/measures/total_revenue.yaml
@@ -0,0 +1,20 @@
+name: Total Revenue
+entity_id: total-revenue-measure
+definition:
+ database: Test Database
+ stages:
+ - aggregation:
+ - - sum
+ - {}
+ - - field
+ - {}
+ - total
+ source-table:
+ - Test Database
+ - public
+ - orders
+ lib/type: mbql.stage/mbql
+ lib/type: mbql/query
+serdes/meta:
+- id: total-revenue-measure
+ model: Measure
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/orders.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/orders.yaml
new file mode 100644
index 00000000000..6fb5c39a56f
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/orders.yaml
@@ -0,0 +1,34 @@
+name: orders
+display_name: Orders
+description: Order transactions table
+created_at: '2026-01-01T00:00:00.000000Z'
+db_id: Test Database
+schema: public
+entity_type: entity/TransactionTable
+active: true
+is_upload: false
+field_order: database
+visibility_type: null
+show_in_getting_started: false
+initial_sync_status: complete
+points_of_interest: null
+caveats: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: orders
+ model: Table
+archived_at: null
+collection_id: null
+data_authority: unconfigured
+data_layer: internal
+data_source: null
+database_require_filter: null
+deactivated_at: null
+is_published: false
+is_writable: true
+owner_email: null
+owner_user_id: null
+transform_id: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/segments/big_orders.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/segments/big_orders.yaml
new file mode 100644
index 00000000000..bfec8397f15
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/segments/big_orders.yaml
@@ -0,0 +1,21 @@
+name: Big Orders
+entity_id: big-orders-segment
+definition:
+ database: Test Database
+ stages:
+ - filters:
+ - - ">"
+ - {}
+ - - field
+ - {}
+ - total
+ - 100
+ source-table:
+ - Test Database
+ - public
+ - orders
+ lib/type: mbql.stage/mbql
+ lib/type: mbql/query
+serdes/meta:
+- id: big-orders-segment
+ model: Segment
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/id.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/id.yaml
new file mode 100644
index 00000000000..cd6fb4cb53c
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/id.yaml
@@ -0,0 +1,45 @@
+name: id
+display_name: ID
+description: Primary key for products
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- products
+database_type: BIGINT
+base_type: type/BigInteger
+effective_type: null
+semantic_type: type/PK
+database_is_auto_increment: true
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 0
+custom_position: 0
+database_position: 0
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: products
+ model: Table
+- id: id
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: false
+database_is_pk: true
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/price.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/price.yaml
new file mode 100644
index 00000000000..18c888b5383
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/price.yaml
@@ -0,0 +1,45 @@
+name: price
+display_name: Price
+description: Product price
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- products
+database_type: DOUBLE PRECISION
+base_type: type/Float
+effective_type: null
+semantic_type: null
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 2
+custom_position: 0
+database_position: 2
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: products
+ model: Table
+- id: price
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/title.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/title.yaml
new file mode 100644
index 00000000000..87e204b4c19
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/title.yaml
@@ -0,0 +1,45 @@
+name: title
+display_name: Title
+description: Product title
+created_at: '2026-01-01T00:00:00.000000Z'
+active: true
+visibility_type: normal
+table_id:
+- Test Database
+- public
+- products
+database_type: VARCHAR
+base_type: type/Text
+effective_type: null
+semantic_type: type/Title
+database_is_auto_increment: false
+database_required: false
+fk_target_field_id: null
+dimensions: []
+json_unfolding: false
+parent_id: null
+coercion_strategy: null
+preview_display: true
+position: 1
+custom_position: 0
+database_position: 1
+has_field_values: null
+settings: null
+caveats: null
+points_of_interest: null
+nfc_path: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: products
+ model: Table
+- id: title
+ model: Field
+database_default: null
+database_indexed: null
+database_is_generated: false
+database_is_nullable: true
+database_is_pk: false
+database_partitioned: null
diff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/products.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/products.yaml
new file mode 100644
index 00000000000..1c541117070
--- /dev/null
+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/products.yaml
@@ -0,0 +1,34 @@
+name: products
+display_name: Products
+description: Product catalog
+created_at: '2026-01-01T00:00:00.000000Z'
+db_id: Test Database
+schema: public
+entity_type: entity/ProductTable
+active: true
+is_upload: false
+field_order: database
+visibility_type: null
+show_in_getting_started: false
+initial_sync_status: complete
+points_of_interest: null
+caveats: null
+serdes/meta:
+- id: Test Database
+ model: Database
+- id: public
+ model: Schema
+- id: products
+ model: Table
+archived_at: null
+collection_id: null
+data_authority: unconfigured
+data_layer: internal
+data_source: null
+database_require_filter: null
+deactivated_at: null
+is_published: false
+is_writable: true
+owner_email: null
+owner_user_id: null
+transform_id: null