The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Checker module (#70866)
dpsuttond4a861b3c928
diff --git a/.claude/skills/serdes-workflow/SKILL.md b/.claude/skills/serdes-workflow/SKILL.mdnew file mode 100644index 00000000000..dcc82bee984--- /dev/null+++ b/.claude/skills/serdes-workflow/SKILL.md@@ -0,0 +1,76 @@+---+name: serdes-workflow+description: Export content from a running Metabase instance, validate with checkers, edit YAML, and import back. Use when the user wants to export, import, or run the full serdes round-trip workflow.+---++# Serdes Workflow Skill++End-to-end workflow for exporting Metabase content as YAML, validating it, editing it, and importing it back.++## Prerequisites++All commands require the `:ee` alias since export, import, and checkers live in enterprise code.++You need `MB_DB_CONNECTION_URI` set to point at the target Metabase Postgres database when importing. The running dev instance's connection string is in the worktree environment (check the nREPL or startup output).++## Export++```bash+clojure -M:run:ee export /path/to/export-dir+```++Flags:+- `-c 123,456` - export only specific collection IDs+- `-f` - include field values+- `-e` - continue on error++This writes YAML files in the serdes directory layout:+```+databases/<db-name>/<db-name>.yaml+databases/<db-name>/schemas/<schema>/tables/<table>/<table>.yaml+databases/<db-name>/schemas/<schema>/tables/<table>/fields/<field>.yaml+collections/<eid>_<slug>/cards/<eid>_<slug>.yaml+```++## Validate++Always run both checkers after any modification. They skip full Metabase startup (~8s each).++### Structural checker (fast, schema-based)+```bash+clojure -M:run:ee --mode checker --checker structural --export /path/to/export-dir+```+Validates YAML shapes against Malli schemas. Catches typos, missing required fields, wrong types.++### Cards checker (query validation)+```bash+clojure -M:run:ee --mode checker --checker cards --export /path/to/export-dir+```+Validates that card queries resolve correctly against exported metadata (tables, fields, FKs). Uses MLv2 under the hood.++Additional cards checker flags:+- `--lenient` - fabricate metadata on demand (when no database schema files exist)+- `--manifest /path/to/manifest.yaml` - write manifest of all referenced entities (lenient mode only)+- `--output /path/to/results.edn` - write detailed results to file++## Edit YAML++See the **serdes-yaml-edit** skill for guidance on editing YAML files. The critical rule: **run both checkers after every edit before proceeding**.++## Import++```bash+MB_DB_CONNECTION_URI="postgres://user:pass@localhost:5432/metabase_dbname" \+ clojure -M:run:ee import /path/to/export-dir -e+```++The `-e` flag continues on error. The `MB_DB_CONNECTION_URI` must point at the same Postgres database the running Metabase instance uses - without it, import defaults to H2 which is wrong.++## Full Demo Sequence++1. Export: `clojure -M:run:ee export /tmp/metabase-export`+2. Validate baseline: run both checkers+3. Edit YAML files (see serdes-yaml-edit skill)+4. Re-validate: run both checkers again+5. Import: `MB_DB_CONNECTION_URI=... clojure -M:run:ee import /tmp/metabase-export -e`+6. Verify in browserdiff --git a/.claude/skills/serdes-yaml-edit/SKILL.md b/.claude/skills/serdes-yaml-edit/SKILL.mdnew file mode 100644index 00000000000..1d6413ecddd--- /dev/null+++ b/.claude/skills/serdes-yaml-edit/SKILL.md@@ -0,0 +1,243 @@+---+name: serdes-yaml-edit+description: Edit Metabase serdes YAML files (cards, dashboards, databases) with correct portable references and structural conventions. Use when modifying exported YAML content.+---++# Serdes YAML Edit Skill++## Golden Rule++**Run both checkers after every edit.** No exceptions. Do not batch multiple edits before validating.++```bash+clojure -M:run:ee --mode checker --checker structural --export /path/to/export-dir+clojure -M:run:ee --mode checker --checker cards --export /path/to/export-dir+```++If either checker fails, fix the issue before making further edits.++## Portable References++Serdes YAML uses portable references instead of integer IDs. This is the most important concept for editing.++### Database references+String name: `"Sample Database"`++### Table references+Array of `[database, schema, table]`:+```yaml+table_id:+- Sample Database+- PUBLIC+- ACCOUNTS+```++### Field references+Array of `[database, schema, table, field]`:+```yaml+id:+- Sample Database+- PUBLIC+- ACCOUNTS+```++### Field refs in queries+```yaml+field_ref:+- field+- - Sample Database+ - PUBLIC+ - ACCOUNTS+- null+```+The outer array is `[field, <field-path>, <options>]`. The options are usually `null`.++### Card references+Entity ID string (21 characters): `"Qk5TgsNx4ubXIUtsQmT8G"`++### Source table in queries+```yaml+dataset_query:+ database: Sample Database+ query:+ source-table:+ - Sample Database+ - PUBLIC+ - ACCOUNTS+ type: query+```++## Safe Edits (low risk)++These rarely break validation:+- `name` - card/dashboard display name+- `description` - card/dashboard description+- `display` - visualization type (table, bar, line, pie, etc.)+- `visualization_settings` - chart configuration+- `archived` - true/false+- `collection_id` - move to a different collection (use entity_id of target collection, or null for root)++## Structural Edits (must match schema)++These must use valid portable refs and will be caught by the cards checker if wrong:+- `dataset_query` - the query definition+- `result_metadata` - column metadata (must match the query's output columns)+- `table_id` - must reference a table that exists in the export+- `database_id` - must reference a database that exists in the export++## Editing result_metadata++Each entry in `result_metadata` describes an output column. When changing a query's source table or fields, you must update result_metadata to match. Each field entry needs at minimum:+- `name` - column name (e.g., `EMAIL`)+- `base_type` - Metabase type (e.g., `type/Text`, `type/Integer`, `type/DateTime`)+- `display_name` - human-readable name+- `field_ref` - portable field reference+- `id` - portable field path+- `table_id` - portable table path+- `source` - usually `fields`++## Common Operations++### Rename a card+Change `name:` at the top level. Safe, no ref changes needed.++### Change a card's source table+Update all of these consistently:+1. `table_id` - top-level+2. `dataset_query.query.source-table` - in the query+3. `result_metadata` - every field entry's `id`, `field_ref`, and `table_id`++### Add a filter to a structured query+```yaml+dataset_query:+ database: Sample Database+ query:+ source-table:+ - Sample Database+ - PUBLIC+ - ORDERS+ filter:+ - ">"+ - - field+ - - Sample Database+ - PUBLIC+ - ORDERS+ - TOTAL+ - null+ - 100+ type: query+```++### Change visualization type+```yaml+display: bar # was: table+```+Valid types: `table`, `bar`, `line`, `pie`, `scalar`, `row`, `area`, `combo`, `scatter`, `funnel`, `map`, `pivot`, `progress`, `gauge`, `waterfall`++## Looking Up Valid References++The export directory IS the reference catalog. When you need to find the correct name for a database, table, or field, look it up directly from the export.++### List valid databases+```bash+ls databases/+```+Each entry is a database name (directory for serdes format, .yaml for concise format).++### List valid tables for a database+```bash+ls databases/<db-name>/schemas/<schema>/tables/+```+Example: `ls databases/Sample\ Database/schemas/PUBLIC/tables/` shows `ACCOUNTS`, `ORDERS`, `PRODUCTS`, etc.++### List valid fields for a table+```bash+ls databases/<db-name>/schemas/<schema>/tables/<table>/fields/+```+Example: `ls databases/Sample\ Database/schemas/PUBLIC/tables/PRODUCTS/fields/` shows `CATEGORY.yaml`, `TITLE.yaml`, `PRICE.yaml`, etc. The filename (minus `.yaml`) is the field name.++### Find card entity IDs+Card filenames encode the entity ID: `<entity-id>_<slug>.yaml`. The entity ID is the part before the first underscore (21 characters).++## Self-Healing Loop++When a checker reports errors, follow this loop:++1. **Read the error message** - it tells you what's wrong and often suggests the fix+2. **Look up the correct value** from the export directory (see "Looking Up Valid References" above)+3. **Fix the YAML**+4. **Re-run both checkers**+5. **Repeat until clean**++Do not guess at fixes. Always look up the correct value from the export.++## Understanding Checker Errors++### Structural checker errors++The structural checker validates YAML shape against Malli schemas. Common errors:++**Missing required key with typo suggestion:**+```+Missing required key 'name' - found 'nameee' which may be a typo+```+Fix: rename the typo'd key back to the correct name. The checker tells you what it expected and what it found.++**Wrong type:**+```+'archived' should be a boolean, got: "yes"+```+Fix: use `true`/`false`, not strings.++**Unknown key:**+```+Unknown key 'foobar' in card+```+Fix: remove the key, or check if it's a typo of a known key.++### Cards checker errors++The cards checker validates that queries resolve against exported metadata. Error types:++**UNRESOLVED REFERENCES:**+```+UNRESOLVED REFERENCES:+ - field: Sample Database.PUBLIC.PRODUCTS.CATEGORYYY+```+The dotted path tells you exactly which reference failed. Look up the correct value:+- Last segment is the field name - check `databases/.../tables/<table>/fields/`+- Third segment is the table name - check `databases/.../schemas/<schema>/tables/`+- First segment is the database name - check `databases/`++**ERROR (query construction failed):**+```+ERROR: Error creating query from legacy query: Invalid output: ...+```+Usually means the query structure is malformed. This often follows unresolved references - fix the refs first, then re-check.++**ERROR (nil name):**+```+ERROR: Invalid output: {:name ["should be a string, got: nil"]}+```+The card is missing its `name` field (likely a structural issue that also affects the cards checker).++### Exit codes++- Exit 0: all checks passed+- Exit 1: one or more failures++## File Naming Convention++Card files: `collections/<collection-path>/cards/<entity-id>_<slug>.yaml`++The filename slug should match the card name (lowercase, underscored). If you rename a card, consider renaming the file to match, though import works on entity_id not filename.++## What NOT to Edit++- `entity_id` - this is the identity of the object, changing it creates a new object on import+- `serdes/meta` - internal serdes metadata, leave it alone+- `created_at` - timestamp, no reason to change+- `creator_id` - email reference, leave it+- `metabase_version` - informational, leave itdiff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.ednindex 50d076bf85c..abca4ff91ff 100644--- a/.clj-kondo/config/modules/config.edn+++ b/.clj-kondo/config/modules/config.edn@@ -1290,6 +1290,7 @@metabase.models.interfacemetabase.models.resolutionmetabase.models.serialization+ metabase.models.serialization.resolvemetabase.models.util.spec-updatemetabase.models.visualization-settings}:uses #{api@@ -2741,6 +2742,18 @@:model/QueryCache:model/QueryExecution}}+ enterprise/checker+ {:team "Graphy"+ ;; todo: add ability to prevent usage (transitive even) of app db+ :api #{}+ :uses #{lib+ models+ query-processor+ sql-tools+ util+ enterprise/dependencies}+ :model-imports #{:model/Database}}+enterprise/cloud-add-ons{:team "Cloud":api #{metabase-enterprise.cloud-add-ons.api}@@ -2836,7 +2849,8 @@enterprise/dependencies{:team "Graphy"- :api #{metabase-enterprise.dependencies.api+ :api #{metabase-enterprise.dependencies.analysis+ metabase-enterprise.dependencies.apimetabase-enterprise.dependencies.coremetabase-enterprise.dependencies.initmetabase-enterprise.dependencies.models.dependency}diff --git a/CLAUDE.md b/CLAUDE.mdindex 1558c62ae77..480b696abc0 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -26,6 +26,11 @@ For detailed guidance on writing and reviewing code and documentation, see the s- **[docs-write](.claude/skills/docs-write/SKILL.md)** - Documentation writing with Metabase style guide- **[docs-review](.claude/skills/docs-review/SKILL.md)** - Documentation review checklist+## Serialization++- **[serdes-workflow](.claude/skills/serdes-workflow/SKILL.md)** - Export, validate, and import Metabase content via serdes+- **[serdes-yaml-edit](.claude/skills/serdes-yaml-edit/SKILL.md)** - Edit exported YAML files with correct portable references+## Frontend- **[analytics-events](.claude/skills/analytics-events/SKILL.md)** - Add product analytics events to track user interactionsdiff --git a/enterprise/backend/src/metabase_enterprise/checker/README.md b/enterprise/backend/src/metabase_enterprise/checker/README.mdnew file mode 100644index 00000000000..7d2fd200201--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/README.md@@ -0,0 +1,350 @@+# Checker Module++Validates serdes YAML exports without a running Metabase instance or database connection.++Given two directories — database schemas and exported entities — the checker+resolves all references, runs query validation, and reports what's broken.++## Quick Start++```bash+# Dev mode+clj -M:dev:drivers:ee -m metabase.core.bootstrap \+ --mode checker \+ --export /path/to/export \+ --schema-dir /path/to/databases++# Built jar+java -jar metabase.jar --mode checker \+ --export /path/to/export --schema-dir /path/to/databases+```++Exit code 0 when all checks pass, 1 when any fail.++Example call against representations with edits intentionall introducing errors:++```+❯ clj -M:dev:drivers:ee -m metabase.core.bootstrap --mode checker --export /Users/dan/projects/work/representations/examples/v1 --schema-dir /Users/dan/projects/work/yaml-checked-files-v1/exports/sqlite-based/databases+[lots of logging removed]+Semantic Check Results+=====================+Total entities: 79+ OK: 76+ Errors: 0+ Unresolved refs: 1+ Native SQL errors: 2+ Issues: 0++Failures:+---------++=== Daily Order Summary [7XqqzlsonSq4KFyOvtrAm] ===+ Kind: transform+ UNRESOLVED REFERENCES:+ NATIVE SQL ERRORS:+ - {:type :syntax-error}+ SQL: SELECT+ DATE_TRUNC('day', CREATED_AT) AS order_date+ COUT(*) AS order_count,+ SU(TOTAL) AS total_revenue+FROM ORDERS+GROUP BY 1+ Status: NATIVE SQL ERRORS++=== Native Query - Card and Snippet References [HiBFSt0BNx5s5MxVDLLKB] ===+ Card ID: 94+ Tables: Sample Database.PUBLIC.ORDERS+ Fields: Sample Database.PUBLIC.ORDERS.CREATED_AT+ Source Cards: Basic Aggregations+ NATIVE SQL ERRORS:+ - {:type :missing-column, :name "created_attt", :source-entity-type :card, :source-entity-id "h5F2EjHsRd73Dqqh8sAtd"}+ - {:type :missing-column, :name "cooount", :source-entity-type :card, :source-entity-id "h5F2EjHsRd73Dqqh8sAtd"}+ Status: NATIVE SQL ERRORS++=== Product Analysis Report [dOc1PrOdAnAlYsIsRpTx2] ===+ Kind: document+ UNRESOLVED REFERENCES:+ - document-link: Q_jD-f-9clKLFZ2TfUG2g+ Status: MISSING REFS++❯ echo $?+1+```++## What It Validates++### Query validation (via `deps.analysis/check-entity`)++- **MBQL**: bad field/table/card references, broken joins+- **Native SQL**: missing columns, syntax errors, bad table references+- **Transforms**: query validation plus duplicate output column detection+- Source attribution on errors (which table or card caused the problem)++### Structural checks++- `collection_id`, `dashboard_id`, `document_id`, `parent_id` point to entities of the correct kind+- Dashboard layout: card refs exist, tab refs match, grid positions in bounds+- Document content: embedded card refs and entity links resolve+- Transform database refs exist in the schema+- Duplicate `entity_id`s across files++### Dependency tracking++Each card result includes a `:refs` map showing what it touches:++- `:tables` — database tables referenced+- `:fields` — specific fields used+- `:source-cards` — other cards used as data sources+- `:snippets` — native query snippets used+- `:measures`, `:metrics`, `:segments` — referenced by entity-id++These propagate transitively through source-card chains.++## CLI Options++```+--export PATH Path to serdes export directory (collections/)+--schema-dir PATH Path to database schema directory (databases/)+--output PATH Write raw results to a file+--errors-only Output only errors to stdout (concise format)+--help Show help+```++`--errors-only` outputs one block per failing entity:++```+card: Number of Orders (entity_id: 8EdazRgPwfxdiltp7NCjS)+ unresolved field: Sample Database.PUBLIC.ORDERS.STARTED_AT+transform: Bad Transform (entity_id: 6Fdv3rO4bB5xyXusrVEGS)+ unresolved field: zomato..zomato.deliveryyyy+```++## Running Tests++```bash+clj -X:dev:test:ee:ee-dev :module enterprise/checker+```++Tests include unit tests (in-memory sources), integration tests (YAML fixtures),+and e2e tests against the baseline export at `test_resources/serialization_baseline`.++## REPL Usage++```clojure+(require '[metabase-enterprise.checker.semantic :as checker])++;; Check the baseline export+(def results+ (:results (checker/check "test_resources/serialization_baseline"+ "test_resources/serialization_baseline/databases")))++(checker/summarize-results results)+;; => {:total 80, :ok 80, :errors 0, :unresolved 0, :native-errors 0, :issues 0}++;; Inspect specific entities+(select-keys results ["wpW9JO5MsHQd0-7WrR7gN"+ "kBjQ5VXJ5z3vYSW72J6qa"+ "dOc1PrOdAnAlYsIsRpTx2"])+;; =>+;; {"wpW9JO5MsHQd0-7WrR7gN"+;; {:card-id 67,+;; :name "Joins - Strategies and Compound Conditions",+;; :entity-id "wpW9JO5MsHQd0-7WrR7gN",+;; :refs {:tables ["Sample Database.PUBLIC.PRODUCTS"+;; "Sample Database.PUBLIC.REVIEWS"+;; "Sample Database.PUBLIC.ORDERS"+;; "Sample Database.PUBLIC.PEOPLE"],+;; :fields ["Sample Database.PUBLIC.ORDERS.ID"+;; "Sample Database.PUBLIC.ORDERS.USER_ID"+;; ...19 fields across 4 tables...]}},+;;+;; "kBjQ5VXJ5z3vYSW72J6qa"+;; {:card-id 6,+;; :name "Metric, Measure, and Segment References",+;; :entity-id "kBjQ5VXJ5z3vYSW72J6qa",+;; :refs {:tables ["Sample Database.PUBLIC.ORDERS"],+;; :source-cards ["Total Revenue"],+;; :measures ["DBap2523KuN4Lt-cYrUjF"],+;; :metrics ["IW8kbqZVaMxdGtCM2F4U6"],+;; :segments ["OaFXAvEzPvu9ZXW5PUMRa"]}},+;;+;; "dOc1PrOdAnAlYsIsRpTx2"+;; {:name "Product Analysis Report",+;; :entity-id "dOc1PrOdAnAlYsIsRpTx2",+;; :kind :document}}++;; Interactive iteration on a single card+(def ctx (checker/setup "test_resources/serialization_baseline"+ "test_resources/serialization_baseline/databases"))+(checker/check-one ctx "HiBFSt0BNx5s5MxVDLLKB" :verbose true)+```++## Architecture++```+ SchemaSource ──┐ ┌── query validation (deps.analysis)+ (databases, │ │+ tables, ├─→ Store ─→ Provider ─→ MetadataProvider+ fields) │ (IDs, (lib │+ │ cache) adapter) └── structural checks+ AssetsSource ──┘ (collection refs, dashboard+ (cards, layout, document links)+ snippets,+ transforms,+ segments)+```++1. **Sources** provide entity data. `SchemaSource` resolves databases/tables/fields+ from `--schema-dir`. `AssetsSource` resolves cards/snippets/transforms/segments+ from `--export`. Both load YAML lazily.+2. **Store** combines both sources with a file index. Knows what entities exist,+ loads them on demand, assigns synthetic integer IDs (lib requires ints), caches+ everything.+3. **Provider** adapts the store into a `MetadataProvider` that `lib/query`+ understands. Converts raw YAML to lib metadata format, resolves portable refs+ to integer IDs, normalizes MBQL. Owns a mutable `current-db` atom set per+ entity check.+4. **Validation** runs query checks via `deps.analysis/check-entity` and+ structural checks on entity relationships.++### On-demand indexing++The schema directory can contain 500k+ field files. To avoid parsing them all+at startup:++- **Databases** (~40): indexed at startup by reading only the `name:` field+ from each database YAML via fast regex (no full parse).+- **Tables** (~22k): enumerated on demand by listing directories. Assigned+ lightweight IDs from directory paths via `ensure-table-id!` (no YAML parse).+ Full YAML is parsed only when table data is actually needed.+- **Fields** (~500k): resolved on demand from `<table>/fields/` directories.+ Never pre-indexed.+- **Segments/measures**: indexed from the export dir only (walks table+ directories for `segments/` and `measures/` subdirs). Not indexed in the+ schema dir.++Directory names are slugified (e.g. `analytics_data_warehouse` for+"Analytics Data Warehouse"). A `db-name->dir` mapping built from database+YAMLs handles the translation. Schema and table directory names match their+real names.++This brings setup time from ~64s to ~100ms.++### Running without an app DB++The checker runs the full QP compilation pipeline (template tag expansion,+snippet inlining, card ref resolution) but without a running Metabase instance.+Two things make this possible:++- **`qp.i/*skip-middleware-because-app-db-access*`** is bound to `true`, which+ skips middleware that queries the app DB for access control. Currently skipped:+ - **Impersonation** — connection role swapping based on user permissions+ - **Database routing** — routing queries to alternative destination databases++ These are security middleware, not semantic middleware. They control *who runs+ where*, not the shape of the query. Skipping them is safe for validation.++- **`mu.fn/*enforce*`** is bound to `false`, disabling Malli output schema+ enforcement so `lib/query` accepts valid MBQL clauses that the schemas don't+ fully enumerate.++This means native SQL validation resolves through `{{#card-ref}}` and+`{{snippet: Name}}` template tags using real data from the store, catching+missing columns with source attribution back to the originating card or snippet.++### Namespaces++- **`checker.source`** — `SchemaSource` and `AssetsSource` protocols+- **`checker.format.serdes`** — serdes directory walking, YAML extraction, file index building, `SerdesSource` (implements both protocols)+- **`checker.store`** — combines sources + index, ID registry, entity caches, lazy loading+- **`checker.provider`** — adapts store to lib `MetadataProvider`, data conversion, MBQL normalization, reference resolution+- **`checker.semantic`** — entity validation, result formatting, top-level API (`check`, `setup`, `check-one`)+- **`checker.cli`** — CLI entrypoint and argument parsing++## Performance++- **Setup**: ~100ms (reads ~40 database YAMLs)+- **Per entity**: ~1s average, cached after first access per database+- **First entity per database**: ~5-8s (table directory listing + ID assignment)+- **Complex native SQL**: 5-20s per entity (SQLGlot parsing bottleneck)+- **Full run** (177 entities, 40 databases, 22k tables): ~90s++### Parallelism++The architecture is ready for multi-threaded entity checking:+- The store (atom) is thread-safe+- Sources are stateless/immutable+- Each thread needs its own `provider` (owns mutable `current-db`)+- Dynamic bindings (`mu.fn/*enforce*`, `qp.i/*skip-middleware*`) are per-thread++To parallelize: create a provider per thread, replace the `for` in+`check-entities` with `pmap` or a thread pool.++## Roadmap++### Dependency graph++The checker computes forward edges per entity (what it depends on). The next+step is building a bidirectional dependency graph using+`metabase.graph.core/InMemoryGraph`.++The `dependencies` module already has the right pieces:++- **`calculation.clj`** — `upstream-deps:card`, `upstream-deps:transform`, etc.+ extract forward edges from resolved lib queries. These just call+ `lib/all-source-table-ids` etc., which work fine against our provider.+- **`analysis.clj`** — `check-entity` validates individual entities. The checker+ already uses this.+- **`metabase.graph.core`** — generic `Graph` protocol with `InMemoryGraph`,+ `transitive`, `find-cycle`, etc.++The plan:++1. Call `calculation.clj`'s `upstream-deps:*` functions against the checker's+ resolved entities (or align our `extract-refs-from-query` output to match)+2. Collect all edges into an adjacency map, wrap in `graph/in-memory`+3. Invert the map for the reverse graph+4. Return both from `checker/check`++This enables "who uses this snippet?", "what breaks if I rename this table?",+cycle detection via `graph/find-cycle`, and full impact analysis — all offline.++### Decouple dependencies module from app DB++`deps.analysis` and `deps.calculation` are clean — they operate on+MetadataProviders and lib queries with no toucan2 dependency. But+`deps.core` and `deps.models.dependency` are tightly coupled to the app DB+(toucan2, `:model/Dependency` table, etc.).++Currently the checker imports `deps.analysis` directly, avoiding the toucan2+dependency chain. For the graph work, we'll also need `deps.calculation`+(also clean) and `metabase.graph.core` (also clean).++The goal is that the "pure analysis" side of the dependencies module+(`analysis.clj`, `calculation.clj`) stays usable without an app DB connection,+while the "storage" side (`models/dependency.clj`, `events.clj`) stays in leaf+namespaces that the checker never touches. This mirrors the pattern in+`models.serialization` where we separated the resolve protocols from the+DB-backed implementations.++### Merge native errors into bad-refs++`:native-errors` (from SQL parsing) and `:bad-refs` (from MBQL analysis) are+both query issues found through different code paths. Consider merging them+into a single `:query-issues` key. The other result keys (`:error` for fatal+failures, `:unresolved` for missing schema refs) are genuinely different+categories and should stay separate.++### Table-type template tag validation++Native queries with `{{table_one}}`-style template tags (type: table) currently+get placeholder substitution — column validation is skipped because the actual+table is unknown. The template tag contains a `table-id` that could be resolved+via the provider to substitute the real table name, enabling full column+validation against the schema.++### Schema mismatch false positives++`sql-tools/validate-query` defaults unqualified tables to `default-schema`,+which returns `"public"` for schema-less databases where tables have `nil`+schema. Root cause is in `sql-tools`, not the checker.diff --git a/enterprise/backend/src/metabase_enterprise/checker/cli.clj b/enterprise/backend/src/metabase_enterprise/checker/cli.cljnew file mode 100644index 00000000000..e1fc45d62e5--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/cli.clj@@ -0,0 +1,126 @@+(ns metabase-enterprise.checker.cli+ "CLI entrypoint for the checker module.++ Invoked from metabase.core.bootstrap:+ java -jar metabase.jar --mode checker --export /path/to/export --schema-dir /path/to/schemas"+ (:require+ [clojure.java.io :as io]+ [clojure.string :as str]+ [clojure.tools.cli :as cli]+ [metabase-enterprise.checker.semantic :as checker]))++(set! *warn-on-reflection* true)++#_{:clj-kondo/ignore [:discouraged-var]}+(def output!+ "Alias for println so can suppress warning in one place"+ println)++(defn- fail!+ "Print error message to stderr and exit with code 1."+ [& messages]+ (binding [*out* *err*]+ (doseq [msg messages]+ (output! msg)))+ (flush)+ (System/exit 1))++(defn- validate-directory!+ "Validate that path exists and is a directory. Exits on failure."+ [path]+ (let [f (io/file path)]+ (cond+ (not (.exists f))+ (fail! (str "Directory does not exist: " path))++ (not (.isDirectory f))+ (fail! (str "Not a directory: " path)))))++(defn- run-checker+ "Run the semantic checker."+ [export-dir {:keys [output errors-only schema-dir]}]+ (when-not schema-dir+ (fail! "Missing --schema-dir option"))+ (validate-directory! schema-dir)+ (let [{:keys [results]} (try+ (checker/check export-dir schema-dir)+ (catch clojure.lang.ExceptionInfo e+ (fail! (.getMessage e))))+ summary (checker/summarize-results results)+ failures (filter #(not= :ok (checker/result-status (second %))) results)]+ (if errors-only+ ;; Errors-only mode: just errors to stdout, nothing else+ (doseq [entry (sort-by (comp :name second) failures)+ :let [error-str (checker/format-error entry)]+ :when error-str]+ #_{:clj-kondo/ignore [:discouraged-var]}+ (output! error-str))+ ;; Normal mode: full output+ (do+ ;; Write to output file if specified+ (when output+ (spit output (pr-str results))+ (output! "Results written to:" output))+ ;; Print summary+ (output! "Semantic Check Results")+ (output! "=====================")+ (output! "Total entities:" (:total summary))+ (output! " OK:" (:ok summary))+ (output! " Errors:" (:errors summary))+ (output! " Unresolved refs:" (:unresolved summary))+ (output! " Native SQL errors:" (:native-errors summary))+ (output! " Issues:" (:issues summary))+ ;; Print failures+ (when (seq failures)+ (output! "\nFailures:")+ (output! "---------")+ (doseq [entry (sort-by (comp :name second) failures)]+ (output!)+ (output! (checker/format-result entry))))))+ ;; Exit with appropriate code+ (flush)+ (System/exit (if (zero? (+ (:errors summary) (:unresolved summary)+ (:native-errors summary) (:issues summary)))+ 0+ 1))))++(def ^:private cli-spec+ "CLI options for checker mode."+ [[nil "--mode MODE" "Mode (handled by bootstrap, included here for completeness)"]+ [nil "--export PATH" "Path to serdes export directory"]+ [nil "--schema-dir PATH" "Path to database schema directory"]+ [nil "--output PATH" "Path to output file for results"]+ [nil "--errors-only" "Output only errors to stdout (concise format for LLM consumption)"]+ ["-h" "--help" "Show this help"]])++(defn- usage [summary]+ (str "Usage: java -jar metabase.jar --mode checker [options]\n\n"+ "Options:\n"+ summary))++(defn entrypoint+ "Main entrypoint for checker mode. Receives raw args. Intended to enter from+ metabase.core.bootstrap/run-standalone-mode so that it can skip loading everything that+ metabase.core.core/entrypoint loads. This entrypoint owns the process and will call System/exit."+ [args]+ (let [{:keys [options errors summary]} (cli/parse-opts args cli-spec)+ {:keys [export help]} options]+ (cond+ help+ (do (output! (usage summary))+ (System/exit 0))++ errors+ (fail! (str "Error parsing arguments:\n" (str/join "\n" errors))+ ""+ (usage summary))++ (not export)+ (fail! "Missing --export option"+ ""+ (usage summary))++ :else+ (do+ (validate-directory! export)+ (run-checker export options)))))diff --git a/enterprise/backend/src/metabase_enterprise/checker/format/serdes.clj b/enterprise/backend/src/metabase_enterprise/checker/format/serdes.cljnew file mode 100644index 00000000000..accb84b0378--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/format/serdes.clj@@ -0,0 +1,564 @@+(ns metabase-enterprise.checker.format.serdes+ "Serdes export format — directory structure, path parsing, entity loading.++ Directory structure (on disk, names may be slugified):+ databases/<db-dir>/<db-dir>.yaml+ databases/<db-dir>/schemas/<schema-dir>/tables/<table-dir>/<table-dir>.yaml+ databases/<db-dir>/schemas/<schema-dir>/tables/<table-dir>/fields/<field>.yaml+ databases/<db-dir>/tables/<table-dir>/<table-dir>.yaml # schema-less+ databases/<db-dir>/tables/<table-dir>/fields/<field>.yaml # schema-less+ collections/<entity-id>_<slug>/<entity-id>_<slug>.yaml+ collections/.../cards/<entity-id>_<slug>.yaml+ collections/.../dashboards/<entity-id>_<slug>.yaml++ Directory names may differ from entity names (e.g. `analytics_data_warehouse`+ for \"Analytics Data Warehouse\", or `orders` for \"ORDERS\" in H2).++ ## Lazy resolution cache++ The `SerdesSource` maintains a single lazily-populated cache atom that maps+ real entity names (from YAML) to file locations:++ {db-name ; real name, e.g. \"Analytics Data Warehouse\"+ {schema ; real name or nil for schema-less+ ::not-indexed ; schema exists but tables not yet walked+ | {table-name ; real name from YAML+ {:table-file \"path/to/table.yaml\" ; path string to the table YAML+ :fields ::not-indexed ; fields not yet walked for this table+ | {field-name \"path\" ...}}}}} ; real name → field YAML path++ Two levels of laziness:+ 1. Schema → tables: realized when any table in that schema is resolved+ (via `resolve-table`, `resolve-field`, `fields-for-table`) or when+ `tables-for-database` is called. Walks the schema's `tables/` directory+ and reads each table's `name:` from YAML via fast regex.+ 2. Table → fields: realized when `resolve-field` or `fields-for-table` is+ called for that specific table. Walks the table's `fields/` directory+ and reads each field's `name:` via fast regex. Other tables in the same+ schema are unaffected.++ The `db-name->dir` map (built at startup from ~40 database YAMLs) handles the+ database name → directory name translation. Schema and table name → directory+ name translation is handled by case-insensitive directory matching during+ schema indexing."+ (:require+ [clojure.java.io :as io]+ [clojure.string :as str]+ [metabase-enterprise.checker.source :as source]+ [metabase.util.yaml :as yaml])+ (:import+ (java.io File)+ (java.util Locale)))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; YAML Loading+;;; ===========================================================================++(defn load-yaml+ "Load and parse a YAML file."+ [path]+ (yaml/parse-string (slurp path)))++(defn quick-extract+ "Extract a field from YAML using regex (fast) with full parse fallback."+ [file-path field-name pattern]+ (try+ (let [content (slurp file-path)]+ (if-let [[_ value] (re-find pattern content)]+ (str/trim value)+ (get (yaml/parse-string content) (keyword field-name))))+ (catch Exception _ nil)))++(defn extract-entity-id+ "Extract the entity id from a yaml file without parsing it. Looking for a top level entity_id at start of line."+ [file-path]+ (quick-extract file-path "entity_id" #"(?m)^entity_id:\s*(\S+)"))++(defn extract-model+ "Extract the serdes model from a YAML file (e.g., \"Collection\", \"Card\", \"Dashboard\").+ Looks for the model field inside serdes/meta to avoid matching embedded refs."+ [file-path]+ (try+ (let [content (slurp file-path)]+ (when-let [[_ model] (re-find #"(?m)^serdes/meta:\s*\n\s*- id:.*\n\s+(?:label:.*\n\s+)?model:\s*(\S+)" content)]+ (str/trim model)))+ (catch Exception _ nil)))++(defn- read-yaml-name+ "Read the `name:` field from a YAML file using regex (fast).+ Falls back to `fallback` if the file doesn't exist or can't be read."+ [^File yaml-file ^String fallback]+ (if (.exists yaml-file)+ (or (quick-extract (.getPath yaml-file) "name" #"(?m)^name:\s*(.+)")+ fallback)+ fallback))++(defn- db-dir+ "Resolve a real database name to its directory on disk."+ ^File [^File databases-dir db-name->dir db-name]+ (let [dir-name (get db-name->dir db-name db-name)]+ (io/file databases-dir dir-name)))++(defn- list-table-dirs+ "Return a seq of table directory Files for a database.+ Walks schemas/<schema>/tables/<table>/ and tables/<table>/ directories."+ [^File databases-dir db-name->dir-map db-name]+ (let [^File db (db-dir databases-dir db-name->dir-map db-name)+ result (volatile! [])]+ (let [^File schemas-dir (io/file db "schemas")]+ (when (.isDirectory schemas-dir)+ (doseq [^File schema-d (.listFiles schemas-dir)+ :when (.isDirectory schema-d)]+ (let [^File tables-dir (io/file schema-d "tables")]+ (when (.isDirectory tables-dir)+ (doseq [^File td (.listFiles tables-dir)+ :when (.isDirectory td)]+ (vswap! result conj td)))))))+ (let [^File tables-dir (io/file db "tables")]+ (when (.isDirectory tables-dir)+ (doseq [^File td (.listFiles tables-dir)+ :when (.isDirectory td)]+ (vswap! result conj td))))+ @result))++(defn- index-segments-and-measures+ "Walk a table directory's segments/ and measures/ subdirectories.+ Reads entity_id from each YAML (only a handful exist).+ Returns a seq of {:kind :ref :file} entries."+ [^File table-dir]+ (into []+ (mapcat+ (fn [[subdir kind]]+ (let [^File d (io/file table-dir subdir)]+ (when (.isDirectory d)+ (keep (fn [^File f]+ (when (and (.isFile f) (str/ends-with? (.getName f) ".yaml"))+ (when-let [eid (extract-entity-id (.getPath f))]+ {:kind kind :ref eid :file (.getPath f)})))+ (.listFiles d))))))+ [["segments" :segment] ["measures" :measure]]))++(defn- index-schema-dir+ "Walk a databases directory and index databases, segments, and measures.+ Reads ~40 database YAMLs for real names, plus a handful of segment/measure YAMLs+ for entity_ids. Tables and fields are resolved on demand from the filesystem.++ When `include-segments-measures?` is true, also walks table directories+ to find segment and measure YAMLs (for export dirs). Set to false for+ schema-only dirs to avoid walking thousands of table directories.++ Returns {:entries [...]+ :db-name->dir {real-db-name dir-name}}."+ [schema-dir & {:keys [include-segments-measures?]}]+ (let [schema-dir (io/file schema-dir)+ db-name->dir (volatile! {})+ entries (volatile! [])]+ (doseq [^File db-dir (.listFiles schema-dir)+ :when (.isDirectory db-dir)]+ (let [dir-name (.getName db-dir)+ db-yaml (io/file db-dir (str dir-name ".yaml"))+ db-name (read-yaml-name db-yaml dir-name)]+ (vswap! db-name->dir assoc db-name dir-name)+ (vswap! entries conj {:kind :database :ref db-name+ :file (if (.exists db-yaml) (.getPath db-yaml) (.getPath db-dir))})+ ;; Walk table directories for segments and measures (export dirs only)+ (when include-segments-measures?+ (doseq [^File table-dir (list-table-dirs schema-dir @db-name->dir db-name)]+ (vswap! entries into (index-segments-and-measures table-dir))))))+ {:entries @entries+ :db-name->dir @db-name->dir}))++;;; ===========================================================================+;;; File Walking - Build index of all entities+;;; ===========================================================================++(def ^:private model->kind+ "Map serdes model names to index kinds."+ {"Card" :card+ "Dashboard" :dashboard+ "Collection" :collection+ "Document" :document+ "Measure" :measure+ "Segment" :segment+ "NativeQuerySnippet" :snippet+ "Transform" :transform})++(defn- index-collections-tree+ "Walk the collections/ directory and index all entities by their serdes model.+ Returns entries for cards, dashboards, and collections."+ [export-dir]+ (let [collections-dir (io/file export-dir "collections")]+ (when (.isDirectory collections-dir)+ (for [^File file (file-seq collections-dir)+ :when (.isFile file)+ :let [path (.getPath file)]+ :when (str/ends-with? (.getName file) ".yaml")+ :let [model (extract-model path)+ kind (get model->kind model)]+ :when kind+ :let [entity-id (extract-entity-id path)]+ :when entity-id]+ {:kind kind :ref entity-id :file path}))))++(defn build-file-index+ "Build index of all entity files in an export directory.+ Returns `{:index {kind {ref file-path}} :db-name->dir {real-name dir-name}}`+ where index also has `:duplicates` for refs appearing in multiple files."+ [export-dir]+ (let [databases-dir (io/file export-dir "databases")+ schema-result (when (.isDirectory databases-dir)+ (index-schema-dir databases-dir :include-segments-measures? true))+ entries (concat+ (:entries schema-result)+ (index-collections-tree export-dir))+ ;; Group by [kind ref] to find duplicates+ by-key (reduce (fn [m {:keys [kind ref file]}]+ (update m [kind ref] (fnil conj []) file))+ {}+ entries)+ ;; Build the index (first file wins) and collect duplicates+ index (reduce-kv (fn [idx [kind ref] files]+ (assoc-in idx [kind ref] (first files)))+ {}+ by-key)+ dupes (into []+ (keep (fn [[[kind ref] files]]+ (when (> (count files) 1)+ {:kind kind :ref ref :files files})))+ by-key)]+ {:index (cond-> index+ (seq dupes) (assoc :duplicates dupes))+ :db-name->dir (:db-name->dir schema-result)}))++(defn build-database-dir-index+ "Build index of databases from a schema directory.+ Tables and fields are resolved on demand.++ Returns `{:index {:database {name file-path}} :db-name->dir {real-name dir-name}}`."+ [databases-dir]+ (let [{:keys [entries db-name->dir]} (index-schema-dir databases-dir)]+ {:index (reduce (fn [idx {:keys [kind ref file]}]+ (assoc-in idx [kind ref] file))+ {}+ entries)+ :db-name->dir db-name->dir}))++(defn index-stats+ "Get statistics about a file index.+ Note: tables and fields are resolved on demand and won't appear in the index+ for schema-dir sources. They will be 0 for `build-database-dir-index` results."+ [index]+ {:databases (count (:database index))+ :tables (count (:table index))+ :fields (count (:field index))+ :measures (count (:measure index))+ :segments (count (:segment index))+ :cards (count (:card index))+ :dashboards (count (:dashboard index))+ :collections (count (:collection index))+ :database-names (vec (keys (:database index)))})++;;; ===========================================================================+;;; MetadataSource Implementation+;;;+;;; Returns raw YAML data - the checker handles conversion to lib format.+;;; This keeps format knowledge here, lib knowledge in checker.+;;; ===========================================================================++(defn- table-real-name+ "Read the real table name from its YAML file via regex. Falls back to dir name."+ [^File table-dir]+ (let [dir-name (.getName table-dir)+ ^File yaml (io/file table-dir (str dir-name ".yaml"))]+ (read-yaml-name yaml dir-name)))++(defn- field-real-name+ "Read the real field name from its YAML file via regex. Falls back to filename without extension."+ [^File field-yaml]+ (let [fname (.getName field-yaml)+ fallback (str/replace fname #"\.yaml$" "")]+ (read-yaml-name field-yaml fallback)))++(defn- read-yaml-schema+ "Read the `schema:` field from a YAML file using regex (fast).+ Returns nil if not found or if the value is `null`."+ [^File yaml-file]+ (when (.exists yaml-file)+ (let [v (quick-extract (.getPath yaml-file) "schema" #"(?m)^schema:\s*(\S+)")]+ (when (and v (not= v "null"))+ v))))++(defn- index-schema-tables+ "Walk a single schema's tables directory and return:+ {:real-schema-name string-or-nil ; the real schema name from YAML (may differ from dir name)+ :tables {real-table-name {:table-file \"path\", :fields ::not-indexed} ...}}+ Reads real table names from YAMLs via fast regex.+ Fields are NOT indexed here — they are indexed lazily per-table."+ [^File tables-dir]+ (when (.isDirectory tables-dir)+ (let [real-schema (volatile! nil)+ tables (into {}+ (keep (fn [^File td]+ (when (.isDirectory td)+ (let [table-name (table-real-name td)+ dir-name (.getName td)+ ^File yaml (io/file td (str dir-name ".yaml"))]+ (when (.exists yaml)+ (when-not @real-schema+ (vreset! real-schema (read-yaml-schema yaml)))+ [table-name {:table-file (.getPath yaml) :fields ::not-indexed}])))))+ (.listFiles tables-dir))]+ {:real-schema-name @real-schema+ :tables tables})))++(defn- index-table-fields+ "Walk a table's fields/ directory and return a map of:+ {real-field-name File ...}+ Reads real field names from YAMLs via fast regex.+ Skips FieldValues and FieldUserSettings files."+ [table-file-path]+ (let [^File table-dir (.getParentFile (io/file table-file-path))+ ^File fields-dir (io/file table-dir "fields")]+ (if (.isDirectory fields-dir)+ (into {}+ (keep (fn [^File f]+ (let [fname (.getName f)]+ (when (and (.isFile f)+ (str/ends-with? fname ".yaml")+ (not (str/includes? fname "___")))+ [(field-real-name f) (.getPath f)]))))+ (.listFiles fields-dir))+ {})))++(defn- resolve-schema-name+ "Resolve a real schema name to the key used in the model.++ Schema names are the one entity type with no canonical YAML file — the directory+ name is the filesystem key, and the real name comes from the `schema:` field in+ table YAMLs. These may differ (e.g. dir=`public`, real=`PUBLIC` for H2).++ Strategy:+ 1. Exact match in the model's :schemas+ 2. Case-insensitive match (handles H2-style uppercasing)+ 3. nil for schema-less databases++ To change schema resolution in the future, modify this function."+ [model db-name schema]+ (let [schemas (:schemas (get model db-name))+ lower (fn [s] (.toLowerCase (str s) Locale/US))]+ (cond+ (contains? schemas schema) schema+ schema (let [lowered (lower schema)]+ (some (fn [k] (when (and (string? k) (= (lower k) lowered)) k))+ (keys schemas)))+ :else (when (contains? schemas nil) nil))))++(defn- ensure-schema-indexed!+ "Ensure a schema's tables are indexed in the model. No-op if already done.++ Resolves the requested schema name to a model key (possibly case-insensitive),+ walks the table directories, reads real names from YAMLs, and stores the results.+ If the real schema name (from the table YAML `schema:` field) differs from the+ directory name, the model entry is re-keyed under the real name."+ [^File databases-dir db-name->dir db-name schema model]+ (let [model-key (resolve-schema-name @model db-name schema)]+ (when (= ::not-indexed (get-in @model [db-name :schemas model-key]))+ (let [^File db (db-dir databases-dir db-name->dir db-name)+ ^File tables-dir (if model-key+ (io/file db "schemas" model-key "tables")+ (io/file db "tables"))+ {:keys [real-schema-name tables]} (index-schema-tables tables-dir)+ tables (or tables {})+ ;; The real schema name (from YAML) is the canonical key.+ ;; Falls back to model-key (dir name) or the requested name.+ final-key (or real-schema-name model-key schema)]+ (swap! model (fn [m]+ (cond-> m+ ;; Remove old key if re-keying+ (and model-key (not= model-key final-key))+ (update-in [db-name :schemas] dissoc model-key)+ ;; Store under canonical name+ true+ (assoc-in [db-name :schemas final-key] tables))))))))++(defn- ensure-fields-indexed!+ "Ensure a table's fields are indexed in the model. No-op if already done."+ [^File databases-dir db-name->dir db-name schema table-name model]+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema model)+ (when (= ::not-indexed (get-in @model [db-name :schemas schema table-name :fields]))+ (when-let [table-file (get-in @model [db-name :schemas schema table-name :table-file])]+ (let [field-map (index-table-fields table-file)]+ (swap! model assoc-in [db-name :schemas schema table-name :fields] field-map)))))++(defn- list-schemas+ "List schema directory names for a database. Returns seq of strings, or [nil] for schema-less."+ [^File databases-dir db-name->dir db-name]+ (let [^File db (db-dir databases-dir db-name->dir db-name)+ ^File schemas-dir (io/file db "schemas")+ ^File tables-dir (io/file db "tables")]+ (cond-> []+ (.isDirectory schemas-dir)+ (into (map #(.getName ^File %))+ (filter #(.isDirectory ^File %) (.listFiles schemas-dir)))+ (.isDirectory tables-dir)+ (conj nil))))++(defn- init-schema-model+ "Build the initial schema model with db-file paths and ::not-indexed for each schema.++ Schema model shape:+ {db-name {:db-file \"path/to/db.yaml\"+ :schemas {schema ::not-indexed+ | {table {:table-file \"path\" :fields ::not-indexed | {field \"path\"}}}}}}++ Two levels of laziness:+ 1. Schema → tables: populated when the schema is first accessed+ 2. Table → fields: populated when that table's fields are first accessed"+ [^File databases-dir db-name->dir db-file-index]+ (atom+ (into {}+ (for [[db-name _dir-name] db-name->dir]+ [db-name {:db-file (get db-file-index db-name)+ :schemas (into {}+ (for [schema (list-schemas databases-dir db-name->dir db-name)]+ [schema ::not-indexed]))}]))))++(deftype SerdesSource [databases-dir assets-index db-name->dir+ ;; See init-schema-model for shape documentation+ schema-model]+ source/SchemaSource+ (resolve-database [_ db-name]+ (when-let [file (get-in @schema-model [db-name :db-file])]+ (load-yaml file)))++ (resolve-table [_ [db schema table-name :as _table-path]]+ (ensure-schema-indexed! databases-dir db-name->dir db schema schema-model)+ (when-let [path (get-in @schema-model [db :schemas schema table-name :table-file])]+ (load-yaml path)))++ (resolve-field [_ [db schema table-name field :as _field-path]]+ (ensure-fields-indexed! databases-dir db-name->dir db schema table-name schema-model)+ (when-let [path (get-in @schema-model [db :schemas schema table-name :fields field])]+ (load-yaml path)))++ (fields-for-table [_ [db schema table-name :as _table-path]]+ (ensure-fields-indexed! databases-dir db-name->dir db schema table-name schema-model)+ (when-let [entry (get-in @schema-model [db :schemas schema table-name])]+ (let [fields (:fields entry)]+ (when (map? fields)+ (into #{} (map (fn [field-name] [db schema table-name field-name])) (keys fields))))))++ (all-field-paths [_]+ (doseq [[db-name {:keys [schemas]}] @schema-model+ [schema tables] schemas+ :when (= tables ::not-indexed)]+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema schema-model))+ (doseq [[db-name {:keys [schemas]}] @schema-model+ [schema tables] schemas+ :when (map? tables)+ [table-name entry] tables+ :when (= (:fields entry) ::not-indexed)]+ (ensure-fields-indexed! databases-dir db-name->dir db-name schema table-name schema-model))+ (into #{}+ (for [[db {:keys [schemas]}] @schema-model+ [schema tables] schemas+ :when (map? tables)+ [table-name {:keys [fields]}] tables+ :when (map? fields)+ field-name (keys fields)]+ [db schema table-name field-name])))++ (all-database-names [_]+ (keys @schema-model))++ (all-table-paths [_]+ (doseq [[db-name {:keys [schemas]}] @schema-model+ [schema _] schemas]+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema schema-model))+ (into []+ (for [[db {:keys [schemas]}] @schema-model+ [schema tables] schemas+ :when (map? tables)+ table-name (keys tables)]+ [db schema table-name])))++ (tables-for-database [_ db-name]+ (doseq [[schema _] (:schemas (get @schema-model db-name))]+ (ensure-schema-indexed! databases-dir db-name->dir db-name schema schema-model))+ (into []+ (for [[schema tables] (:schemas (get @schema-model db-name))+ :when (map? tables)+ table-name (keys tables)]+ [db-name schema table-name])))++ source/AssetsSource+ (resolve-card [_ entity-id]+ (when-let [file (get-in assets-index [:card entity-id])]+ (load-yaml file)))++ (resolve-snippet [_ entity-id]+ (when-let [file (get-in assets-index [:snippet entity-id])]+ (load-yaml file)))++ (resolve-transform [_ entity-id]+ (when-let [file (get-in assets-index [:transform entity-id])]+ (load-yaml file)))++ (resolve-segment [_ entity-id]+ (when-let [file (get-in assets-index [:segment entity-id])]+ (load-yaml file)))++ (resolve-dashboard [_ entity-id]+ (when-let [file (get-in assets-index [:dashboard entity-id])]+ (load-yaml file)))++ (resolve-collection [_ entity-id]+ (when-let [file (get-in assets-index [:collection entity-id])]+ (load-yaml file)))++ (resolve-document [_ entity-id]+ (when-let [file (get-in assets-index [:document entity-id])]+ (load-yaml file)))++ (resolve-measure [_ entity-id]+ (when-let [file (get-in assets-index [:measure entity-id])]+ (load-yaml file))))++(defn make-source+ "Create a MetadataSource for a serdes export directory.+ The databases-dir for field resolution is export-dir/databases."+ [export-dir]+ (let [databases-dir (io/file export-dir "databases")+ {:keys [index db-name->dir]} (build-file-index export-dir)+ db-files (:database index)+ assets-index (dissoc index :database)]+ (->SerdesSource databases-dir assets-index db-name->dir+ (init-schema-model databases-dir db-name->dir db-files))))++(defn make-database-source+ "Create a MetadataSource for a directory that IS the databases directory.+ The directory should contain database subdirectories directly (e.g., `Sample Database/`).+ This source only resolves databases, tables, and fields — not cards."+ [databases-dir]+ (let [{:keys [index db-name->dir]} (build-database-dir-index databases-dir)+ databases-dir (io/file databases-dir)+ db-files (:database index)]+ (->SerdesSource databases-dir {} db-name->dir+ (init-schema-model databases-dir db-name->dir db-files))))++(defn source-index+ "Get the assets index from a SerdesSource. Returns a flat map of+ {[kind ref] file-path} for cards, dashboards, segments, etc."+ [^SerdesSource source]+ (.-assets-index source))++;;; ===========================================================================+;;; Enumeration+;;; ===========================================================================++(defn all-card-ids+ "Get all card entity-ids from source."+ [^SerdesSource source]+ (keys (:card (.-assets-index source))))diff --git a/enterprise/backend/src/metabase_enterprise/checker/provider.clj b/enterprise/backend/src/metabase_enterprise/checker/provider.cljnew file mode 100644index 00000000000..c28a94b6a50--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/provider.clj@@ -0,0 +1,465 @@+(ns metabase-enterprise.checker.provider+ "Adapts a checker store into a lib MetadataProvider.++ The store (checker.store) knows what entities exist and can load them.+ This namespace converts that raw data into the lib metadata format that+ `lib/query` and `deps.analysis/check-entity` expect.++ Responsibilities:+ - Convert raw YAML entity maps → lib metadata maps (keywordize types,+ resolve FK targets, normalize MBQL operators, resolve portable refs)+ - Implement `MetadataProvider` by delegating to the store+ - Assign sentinel IDs for unresolved refs so queries can still be+ constructed and `lib/find-bad-refs` reports issues with context++ Public API:+ - `(make-provider store)` — wrap a store in a MetadataProvider+ - `(card-metadata store data)` — convert a card, collecting resolution failures+ - `(transform-metadata store data)` — convert a transform, collecting resolution failures"+ (:require+ [metabase-enterprise.checker.store :as store]+ [metabase.lib.core :as lib]+ [metabase.lib.metadata.protocols :as lib.metadata.protocols]+ [metabase.lib.schema.metadata :as lib.schema.metadata]+ [metabase.models.serialization.resolve :as resolve]+ [potemkin.types :as p.types]))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; Settable database protocol+;;; ===========================================================================++(p.types/defprotocol+ ISettableDatabase+ "Protocol for providers that can switch which database they report."+ (set-database! [this db-name]+ "Set the database name (string) this provider should return from `(database)`.")+ (clear-database! [this]+ "Clear the current database. `(database)` will return nil until `set-database!` is called again."))++;;; ===========================================================================+;;; Reference resolution+;;;+;;; Resolution functions take the store and a `!failures` atom (or nil).+;;; When a ref can't be resolved, a failure map is conj'd onto the atom.+;;; ===========================================================================++(defn- resolve-ref+ "Try to resolve `ref` of `kind`. Returns integer ID or nil.+ On failure, appends failure-info to `!failures`.+ When `sentinel?` is true, assigns a sentinel ID for unresolved refs."+ [store kind ref !failures failure-info & {:keys [sentinel?]}]+ (when ref+ (or (store/ref->id store kind ref)+ (let [load-fn (case kind+ :database store/load-database!+ :table store/load-table!+ :field store/load-field!+ :card store/load-card!+ :snippet store/load-snippet!+ :transform store/load-transform!+ :segment store/load-segment!+ :dashboard store/load-dashboard!+ :collection store/load-collection!+ :document store/load-document!+ :measure store/load-measure!)+ found? (some? (load-fn store ref))]+ (if found?+ (store/get-or-assign! store kind ref)+ (do (when !failures (swap! !failures conj failure-info))+ (when sentinel?+ (store/get-or-assign! store kind ref))))))))++(defn- resolve-field-path+ ([store !failures field-path]+ (resolve-ref store :field field-path !failures {:type :field :path field-path}))+ ([store !failures field-path sentinel?]+ (resolve-ref store :field field-path !failures {:type :field :path field-path} :sentinel? sentinel?)))++(defn- resolve-table-path+ ([store !failures table-path]+ (resolve-ref store :table table-path !failures {:type :table :path table-path}))+ ([store !failures table-path sentinel?]+ (resolve-ref store :table table-path !failures {:type :table :path table-path} :sentinel? sentinel?)))++(defn- resolve-db-name [store !failures db-name]+ (resolve-ref store :database db-name !failures {:type :database :name db-name}))++(defn- resolve-card-entity-id [store !failures entity-id]+ (resolve-ref store :card entity-id !failures {:type :card :entity-id entity-id}))++;;; ---------------------------------------------------------------------------+;;; SerdesImportResolver — bridges store-based resolution into import-mbql+;;; ---------------------------------------------------------------------------++(defn- make-import-resolver+ [store !failures & {:keys [sentinel?]}]+ (reify resolve/SerdesImportResolver+ (import-fk [_ eid model]+ #_{:clj-kondo/ignore [:case-symbol-test]}+ (case model+ Card (resolve-card-entity-id store !failures eid)+ Segment (resolve-ref store :segment eid !failures {:type :segment :entity-id eid})+ Measure (resolve-ref store :measure eid !failures {:type :measure :entity-id eid})+ NativeQuerySnippet (resolve-ref store :snippet eid !failures {:type :snippet :entity-id eid})+ (do (when !failures+ (swap! !failures conj {:type :unknown :model model :entity-id eid}))+ nil)))+ (import-fk-keyed [_ portable model field]+ (case [model field]+ [:model/Database :name] (resolve-db-name store !failures portable)+ (do (when !failures+ (swap! !failures conj {:type :keyed-lookup :model model :field field :value portable}))+ nil)))+ (import-user [_ _email] nil)+ (import-table-fk [_ path] (resolve-table-path store !failures path sentinel?))+ (import-field-fk [_ path] (resolve-field-path store !failures path sentinel?))))++;;; ===========================================================================+;;; Data conversion — raw YAML entity maps → lib metadata maps+;;; ===========================================================================++(defn- ->database-metadata [data]+ {:lib/type :metadata/database+ :id (:id data)+ :name (:name data)+ :engine (keyword (:engine data))+ :dbms-version (:dbms_version data)+ :features #{:foreign-keys :nested-queries :expressions :native-parameters+ :basic-aggregations :standard-deviation-aggregations+ :expression-aggregations :left-join :right-join :inner-join :full-join}+ :settings (:settings data)})++(defn- ->table-metadata [data]+ {:lib/type :metadata/table+ :id (:id data)+ :name (:name data)+ :display-name (:display_name data)+ :schema (:schema data)+ :db-id (:db_id data)+ :active (if (contains? data :active) (:active data) true)+ :visibility-type (some-> (:visibility_type data) keyword)})++(defn- ->field-metadata+ [resolver data]+ (cond-> {:lib/type :metadata/column+ :id (:id data)+ :table-id (:table_id data)+ :name (:name data)+ :display-name (:display_name data)+ :base-type (keyword (:base_type data))+ :effective-type (or (some-> (:effective_type data) keyword)+ (keyword (:base_type data)))+ :semantic-type (some-> (:semantic_type data) keyword)+ :database-type (:database_type data)+ :active (if (contains? data :active) (:active data) true)+ :visibility-type (some-> (:visibility_type data) keyword)+ :position (:position data)}+ (and resolver (vector? (:fk_target_field_id data)))+ (as-> m (if-let [fk-id (resolve/import-field-fk resolver (:fk_target_field_id data))]+ (assoc m :fk-target-field-id fk-id)+ m))))++(defn- convert-result-metadata-column+ "Resolve portable refs in a result_metadata column map.+ Portable refs appear as path vectors in :id, :table_id, :fk_target_field_id,+ and as MBQL field refs in :field_ref."+ [resolver col]+ (let [resolve-or-drop (fn [col k resolve-fn]+ (let [v (get col k)]+ (if-not (vector? v)+ col+ (if-let [id (resolve-fn resolver v)]+ (assoc col k id)+ (dissoc col k)))))]+ (-> col+ (resolve-or-drop :id resolve/import-field-fk)+ (resolve-or-drop :table_id resolve/import-table-fk)+ (resolve-or-drop :fk_target_field_id resolve/import-field-fk)+ (cond-> (:field_ref col)+ (update :field_ref (fn [ref]+ (let [resolved (resolve/import-mbql resolver ref)]+ (if (and (vector? resolved) (= :field (first resolved)) (nil? (second resolved)))+ ::drop+ resolved)))))+ (as-> c (if (= ::drop (:field_ref c)) (dissoc c :field_ref) c)))))++(defn- convert-dataset-query+ "Convert a dataset query from serdes format, resolving paths to IDs.++ Pipeline:+ 1. lib/normalize — convert YAML strings to keywords (operators, types, etc.)+ 2. resolve/import-mbql — resolve portable refs (path vectors) to integer IDs+ Uses a sentinel resolver so unresolved refs get IDs for better error reporting."+ [resolver query]+ (when query+ (try+ (let [query (lib/normalize nil query)+ db-name (:database query)+ db-id (when (string? db-name)+ (resolve/import-fk-keyed resolver db-name :model/Database :name))+ query (cond-> query db-id (assoc :database db-id))+ query (resolve/import-mbql resolver query)]+ query)+ (catch Exception _+ (let [db-name (:database query)+ db-id (when (string? db-name)+ (resolve/import-fk-keyed resolver db-name :model/Database :name))]+ (cond-> query db-id (assoc :database db-id)))))))++;;; ---------------------------------------------------------------------------+;;; High-level entity conversion (public — used by semantic.clj)+;;; ---------------------------------------------------------------------------++(declare compute-result-metadata)++(defn card-metadata+ "Convert a cached card entity to lib metadata. Returns the metadata map with+ `:provider/resolution-failures` and `:provider/missing-database` keys when resolution fails."+ [store data]+ (let [!failures (atom [])+ ;; Sentinel resolver: unresolved refs get fake integer IDs so queries can still be constructed and+ ;; lib/find-bad-refs reports issues with context. We use the same resolver for both the query and metadata+ ;; (table_id, FK targets, result_metadata). If we ever need to distinguish "soft failure" (metadata)+ ;; from "must-build" (query), create a second resolver with :sentinel? false and use it for the metadata+ ;; resolution calls below.+ resolver (make-import-resolver store !failures :sentinel? true)+ table-id (when-let [t (:table_id data)]+ (if (vector? t)+ (resolve/import-table-fk resolver t)+ t))+ db-name (get-in data [:dataset_query :database])+ db-id (when (string? db-name)+ (resolve/import-fk-keyed resolver db-name :model/Database :name))+ dataset-query (convert-dataset-query resolver (:dataset_query data))+ result-metadata (if-let [cols (seq (:result_metadata data))]+ (->> cols+ (map (partial convert-result-metadata-column resolver))+ (lib/normalize [:sequential ::lib.schema.metadata/lib-or-legacy-column]))+ (when (and dataset-query db-id)+ (compute-result-metadata store dataset-query)))]+ (cond-> {:lib/type :metadata/card+ :id (:id data)+ :name (:name data)+ :type (some-> (:type data) keyword)+ :dataset-query dataset-query+ :result-metadata result-metadata+ :archived (:archived data)}+ db-id (assoc :database-id db-id)+ (not db-id) (assoc :provider/missing-database db-name)+ table-id (assoc :table-id table-id)+ (seq @!failures) (assoc :provider/resolution-failures (vec (distinct @!failures))))))++(defn transform-metadata+ "Convert a cached transform entity to lib metadata with its source query resolved."+ [store data]+ (let [!failures (atom [])+ resolver (make-import-resolver store !failures :sentinel? true)+ query (convert-dataset-query resolver (get-in data [:source :query]))]+ (cond-> {:lib/type :metadata/transform+ :id (:id data)+ :name (:name data)+ :source (assoc (:source data) :query query)}+ (seq @!failures) (assoc :provider/resolution-failures (vec (distinct @!failures))))))++(defn- ->segment-metadata+ [store data]+ (let [!failures (atom [])+ resolver (make-import-resolver store !failures :sentinel? true)+ definition (convert-dataset-query resolver (:definition data))]+ (cond-> {:lib/type :metadata/segment+ :id (:id data)+ :name (:name data)+ :definition definition}+ (seq @!failures) (assoc :provider/resolution-failures (vec (distinct @!failures))))))++;;; ===========================================================================+;;; MetadataProvider+;;; ===========================================================================++(deftype SourceMetadataProvider+ ;; The store knows what entities exist, loads raw YAML data, assigns integer IDs, and caches.+ ;; The provider adapts the store into the MetadataProvider interface that lib/query expects,+ ;; converting raw YAML maps to lib metadata format along the way.+ ;;+ ;; `current-db` is an atom holding the database name (string) to return from `(database)`.+ ;; Must be set via `set-database!` before each entity check and unset afterward.+ [store current-db]+ ISettableDatabase+ (set-database! [_this db-name]+ (reset! current-db db-name))+ (clear-database! [_this]+ (reset! current-db nil))++ lib.metadata.protocols/MetadataProvider+ (database [_this]+ (let [db-name @current-db]+ (when (nil? db-name)+ (throw (ex-info "No current database set on provider. Call set-database! before querying." {})))+ (when-let [data (store/load-database! store db-name)]+ (->database-metadata data))))++ (metadatas [_this {:keys [lib/type id table-id name]}]+ (case type+ :metadata/table+ (vec+ (if id+ (for [tid id+ :let [path (store/id->ref store :table tid)]+ :when path+ :let [data (store/load-table! store path)]+ :when data]+ (->table-metadata data))+ ;; Only return tables for the current database — uses lightweight+ ;; path-derived metadata, no YAML parsing+ (let [db-name @current-db]+ (for [path (store/tables-for-database store db-name)+ :let [data (store/ensure-table-id! store path)]+ :when data]+ (->table-metadata data)))))++ :metadata/column+ (vec+ (cond+ id+ (for [fid id+ :let [path (store/id->ref store :field fid)]+ :when path+ :let [data (store/load-field! store path)]+ :when data]+ (->field-metadata (make-import-resolver store nil) data))++ table-id+ (let [table-path (store/id->ref store :table table-id)]+ (when table-path+ (for [fpath (store/fields-for-table store table-path)+ :let [data (store/load-field! store fpath)]+ :when data]+ (->field-metadata (make-import-resolver store nil) data))))++ :else+ (for [path (store/all-field-paths store)+ :let [data (store/load-field! store path)]+ :when data]+ (->field-metadata (make-import-resolver store nil) data))))++ :metadata/card+ (vec+ (if id+ (for [cid id+ :let [eid (store/id->ref store :card cid)]+ :when eid+ :let [data (store/load-card! store eid)]+ :when data]+ (card-metadata store data))+ (for [eid (store/all-card-ids store)+ :let [data (store/load-card! store eid)]+ :when data]+ (card-metadata store data))))++ :metadata/native-query-snippet+ (vec+ (cond+ id+ (for [sid id+ :let [eid (store/id->ref store :snippet sid)]+ :when eid+ :let [data (store/load-snippet! store eid)]+ :when data]+ {:lib/type :metadata/native-query-snippet+ :id (:id data)+ :name (:name data)+ :content (:content data)})++ name+ (for [eid (store/all-refs store :snippet)+ :let [data (store/load-snippet! store eid)]+ :when data+ :when (contains? name (:name data))]+ {:lib/type :metadata/native-query-snippet+ :id (:id data)+ :name (:name data)+ :content (:content data)})++ :else []))++ :metadata/transform+ (vec+ (if id+ (for [tid id+ :let [eid (store/id->ref store :transform tid)]+ :when eid+ :let [data (store/load-transform! store eid)]+ :when data]+ (transform-metadata store data))+ (for [eid (store/all-refs store :transform)+ :let [data (store/load-transform! store eid)]+ :when data]+ (transform-metadata store data))))++ :metadata/segment+ (vec+ (if id+ (for [sid id+ :let [eid (store/id->ref store :segment sid)]+ :when eid+ :let [data (store/load-segment! store eid)]+ :when data]+ (->segment-metadata store data))+ (for [eid (store/all-refs store :segment)+ :let [data (store/load-segment! store eid)]+ :when data]+ (->segment-metadata store data))))++ nil))++ (setting [_this _key] nil)++ lib.metadata.protocols/CachedMetadataProvider+ (cached-metadatas [this type ids]+ (lib.metadata.protocols/metadatas this {:lib/type type :id (set ids)}))+ (store-metadata! [_this _obj] nil)+ (cached-value [_this _k not-found] not-found)+ (cache-value! [_this _k _v] nil)+ (has-cache? [_this] true)+ (clear-cache! [_this] nil))++(declare make-provider)++(defn- compute-result-metadata+ [store dataset-query]+ (try+ (let [provider (make-provider store)+ db-id (:database dataset-query)+ db-name (when (integer? db-id) (store/id->ref store :database db-id))]+ (when db-name+ (set-database! provider db-name))+ (let [query (lib/query provider dataset-query)]+ (mapv #(select-keys % [:name :base-type :effective-type :semantic-type :display-name])+ (lib/returned-columns query))))+ (catch Exception _ nil)))++;;; ===========================================================================+;;; Public API+;;; ===========================================================================++(defn make-provider+ "Create a MetadataProvider backed by a store.+ The store holds sources, index, ID registry, and entity caches —+ see `checker.store/make-store`."+ [store]+ (->SourceMetadataProvider store (atom nil)))++(defn database-name-for-entity+ "Determine the database name for an entity from its raw YAML data.+ Cards have it in dataset_query.database, transforms in source.query.database."+ [store kind data]+ (let [db-ref (case kind+ :card (get-in data [:dataset_query :database])+ :transform (or (get-in data [:source :query :database])+ (:source_database_id data))+ nil)]+ (cond+ (string? db-ref) db-ref+ (integer? db-ref) (store/id->ref store :database db-ref)+ :else nil)))diff --git a/enterprise/backend/src/metabase_enterprise/checker/semantic.clj b/enterprise/backend/src/metabase_enterprise/checker/semantic.cljnew file mode 100644index 00000000000..8a9c656aac9--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/semantic.clj@@ -0,0 +1,780 @@+(ns metabase-enterprise.checker.semantic+ "Entity validation — the checker's main entry point.++ Creates a store (checker.store) from sources, wraps it in a MetadataProvider+ (checker.provider), then validates every entity:+ - Query validation via deps.analysis/check-entity (MBQL + native SQL)+ - Structural checks (collection_id, dashboard layout, document links)++ Public API:+ - `(check export-dir schema-dir)` — check all entities, returns results map+ - `(setup export-dir schema-dir)` + `(check-one ctx entity-id)` — REPL workflow"+ (:require+ [clojure.string :as str]+ [metabase-enterprise.checker.format.serdes :as serdes]+ [metabase-enterprise.checker.provider :as provider]+ [metabase-enterprise.checker.store :as store]+ [metabase-enterprise.dependencies.analysis :as deps.analysis]+ [metabase.lib.core :as lib]+ [metabase.lib.metadata :as lib.metadata]+ [metabase.query-processor.interface :as qp.i]+ ;; sql-tools.init registers multimethod implementations needed by deps.native-validation+ [metabase.sql-tools.init]+ [metabase.util.malli.fn :as mu.fn]))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; Reference resolution — resolve portable refs to IDs, collecting failures+;;;+;;; Resolution functions take the store and a `!failures` atom (or nil).+;;; When a ref can't be resolved, a failure map is conj'd onto the atom+;;; for the caller to inspect. The failure maps look like:+;;; {:type :field :path ["DB" "schema" "table" "field"]}+;;; {:type :table :path ["DB" "schema" "table"]}+;;; {:type :database :name "DB"}+;;; ===========================================================================+;;; Entity loading by kind+;;; ===========================================================================++(def ^:private kind->load-fn+ {:dashboard store/load-dashboard!+ :collection store/load-collection!+ :document store/load-document!+ :transform store/load-transform!+ :segment store/load-segment!+ :measure store/load-measure!+ :snippet store/load-snippet!})++;;; ===========================================================================+;;; Common entity checks — apply to any entity type+;;;+;;; These validations run on every entity (cards, dashboards, collections,+;;; metrics, segments, etc.). Add new cross-entity checks here.+;;; ===========================================================================++(defn- validate-collection-id+ "Validate that `collection_id` in `data` points to a known collection.+ Returns a failure map or nil."+ [store data]+ (when-let [coll-id (:collection_id data)]+ (let [kind (store/index-kind-of store coll-id)]+ (cond+ (nil? kind)+ {:type :collection :entity-id coll-id+ :message (str "collection_id " coll-id " not found")}++ (not= :collection kind)+ {:type :collection :entity-id coll-id+ :message (str "collection_id " coll-id " points to a " (name kind) ", not a collection")}))))++(defn- validate-dashboard-id+ "Validate that `dashboard_id` in `data` points to a known dashboard.+ Returns a failure map or nil."+ [store data]+ (when-let [dash-id (:dashboard_id data)]+ (let [kind (store/index-kind-of store dash-id)]+ (cond+ (nil? kind)+ {:type :dashboard :entity-id dash-id+ :message (str "dashboard_id " dash-id " not found")}++ (not= :dashboard kind)+ {:type :dashboard :entity-id dash-id+ :message (str "dashboard_id " dash-id " points to a " (name kind) ", not a dashboard")}))))++(defn- validate-parent-id+ "Validate that `parent_id` on a collection points to a known collection.+ Returns a failure map or nil."+ [store data]+ (when-let [parent-id (:parent_id data)]+ (let [kind (store/index-kind-of store parent-id)]+ (cond+ (nil? kind)+ {:type :collection :entity-id parent-id+ :message (str "parent_id " parent-id " not found")}++ (not= :collection kind)+ {:type :collection :entity-id parent-id+ :message (str "parent_id " parent-id " points to a " (name kind) ", not a collection")}))))++(defn- validate-document-id+ "Validate that `document_id` in `data` points to a known document.+ Returns a failure map or nil."+ [store data]+ (when-let [doc-id (:document_id data)]+ (let [kind (store/index-kind-of store doc-id)]+ (cond+ (nil? kind)+ {:type :document :entity-id doc-id+ :message (str "document_id " doc-id " not found")}++ (not= :document kind)+ {:type :document :entity-id doc-id+ :message (str "document_id " doc-id " points to a " (name kind) ", not a document")}))))++(defn- validate-container-exclusion+ "Validate that a card doesn't have both dashboard_id and document_id set.+ Returns a failure map or nil."+ [_store data]+ (when (and (:dashboard_id data) (:document_id data))+ {:type :container-conflict+ :message "card has both dashboard_id and document_id set — only one container allowed"}))++(defn- validate-container-collection-match+ "Validate that a card nested under a dashboard or document has the same+ collection_id as its container. Returns a failure map or nil."+ [store data]+ (when-let [container-id (or (:dashboard_id data) (:document_id data))]+ (let [container-kind (if (:dashboard_id data) :dashboard :document)+ load-fn (kind->load-fn container-kind)+ container-data (when load-fn (load-fn store container-id))]+ (when container-data+ (let [container-coll (:collection_id container-data)+ card-coll (:collection_id data)]+ (when (and card-coll container-coll+ (not= card-coll container-coll))+ {:type :container-collection-mismatch+ :message (str "card collection_id " card-coll+ " doesn't match its " (name container-kind)+ "'s collection_id " container-coll)}))))))++(defn- check-common+ "Run checks that apply to any entity (card, dashboard, metric, etc.).+ Returns a vector of failure maps, or empty vector if all checks pass."+ [store data]+ (filterv some?+ [(validate-collection-id store data)+ (validate-dashboard-id store data)+ (validate-document-id store data)+ (validate-parent-id store data)+ (validate-container-exclusion store data)+ (validate-container-collection-match store data)]))++;;; ===========================================================================+;;; Dashboard-specific checks+;;;+;;; Validates dashboard internal consistency:+;;; - dashcard card_id refs point to known cards+;;; - dashcard dashboard_tab_id refs point to tabs in this dashboard+;;; - dashcard grid positions are within bounds (24-column grid)+;;; ===========================================================================++(def ^:private grid-width+ "Dashboard grid width in columns."+ 24)++(defn- validate-dashcard-card-ref+ "Validate that a dashcard's card_id points to a known card.+ Virtual cards (headings, text) have null card_id — that's fine."+ [store dashcard]+ (let [card-id (:card_id dashcard)]+ (when (and card-id (not (store/exists? store :card card-id)))+ {:type :dashcard-card-ref+ :dashcard-entity-id (:entity_id dashcard)+ :entity-id card-id+ :message (str "dashcard " (:entity_id dashcard) " references card " card-id " which is not in the export")})))++(defn- validate-dashcard-tab-ref+ "Validate that a dashcard's dashboard_tab_id references a tab in this dashboard.+ dashboard_tab_id is [dashboard-entity-id, tab-entity-id]."+ [tab-entity-ids dashcard]+ (when-let [tab-ref (:dashboard_tab_id dashcard)]+ (let [tab-eid (if (vector? tab-ref) (second tab-ref) tab-ref)]+ (when (and tab-eid (not (contains? tab-entity-ids tab-eid)))+ {:type :dashcard-tab-ref+ :dashcard-entity-id (:entity_id dashcard)+ :entity-id tab-eid+ :message (str "dashcard " (:entity_id dashcard) " references tab " tab-eid " which is not in this dashboard")}))))++(defn- validate-dashcard-grid+ "Validate that a dashcard's grid position is within bounds."+ [dashcard]+ (let [col (or (:col dashcard) 0)+ row (or (:row dashcard) 0)+ size-x (or (:size_x dashcard) 1)+ size-y (or (:size_y dashcard) 1)+ eid (:entity_id dashcard)]+ (cond+ (neg? col)+ {:type :dashcard-grid :dashcard-entity-id eid+ :message (str "dashcard " eid " has negative col: " col)}++ (> (+ col size-x) grid-width)+ {:type :dashcard-grid :dashcard-entity-id eid+ :message (str "dashcard " eid " extends beyond grid: col=" col " + size_x=" size-x " > " grid-width)}++ (< size-x 1)+ {:type :dashcard-grid :dashcard-entity-id eid+ :message (str "dashcard " eid " has invalid size_x: " size-x)}++ (< size-y 1)+ {:type :dashcard-grid :dashcard-entity-id eid+ :message (str "dashcard " eid " has invalid size_y: " size-y)}++ (neg? row)+ {:type :dashcard-grid :dashcard-entity-id eid+ :message (str "dashcard " eid " has negative row: " row)})))++(defn- check-dashboard+ "Run dashboard-specific semantic checks on loaded dashboard data.+ Returns a vector of failure maps."+ [store data]+ (let [tabs (:tabs data)+ dashcards (:dashcards data)+ tab-entity-ids (into #{} (keep :entity_id) tabs)]+ (into []+ (keep identity)+ (mapcat (fn [dc]+ [(validate-dashcard-card-ref store dc)+ (validate-dashcard-tab-ref tab-entity-ids dc)+ (validate-dashcard-grid dc)])+ dashcards))))++;;; ===========================================================================+;;; Document-specific checks+;;;+;;; Documents contain ProseMirror content with embedded card references.+;;; cardEmbed nodes have attrs.id which is a serdes portable ref:+;;; [{:model "Card" :id "entity-id"}]+;;; ===========================================================================++(defn- extract-card-embeds+ "Walk a ProseMirror content tree and extract card entity-ids from cardEmbed nodes."+ [node]+ (when (map? node)+ (let [embeds (when (= "cardEmbed" (:type node))+ (let [refs (get-in node [:attrs :id])]+ (keep (fn [ref]+ (when (and (map? ref) (= "Card" (:model ref)))+ (:id ref)))+ (if (sequential? refs) refs [refs]))))+ child-embeds (mapcat extract-card-embeds (:content node))]+ (concat embeds child-embeds))))++(defn- extract-entity-links+ "Walk a ProseMirror content tree and extract entity references from link hrefs.+ Links like /dashboard/ABC or /question/ABC reference entities by entity_id."+ [node]+ (when (map? node)+ (let [;; Check marks on text nodes for links+ mark-links (when-let [marks (:marks node)]+ (keep (fn [mark]+ (when (= "link" (:type mark))+ (when-let [href (get-in mark [:attrs :href])]+ (when-let [[_ kind eid] (re-matches #"/(dashboard|question|card|collection)/(.+)" (str href))]+ {:kind (case kind+ "dashboard" :dashboard+ "question" :card+ "card" :card+ "collection" :collection)+ :entity-id eid}))))+ marks))+ child-links (mapcat extract-entity-links (:content node))]+ (concat mark-links child-links))))++(defn- check-document+ "Run document-specific semantic checks. Validates embedded card refs and entity links."+ [store data]+ (let [doc (:document data)+ card-eids (extract-card-embeds doc)+ links (extract-entity-links doc)]+ (into []+ cat+ [;; Check card embeds+ (keep (fn [card-eid]+ (when-not (store/exists? store :card card-eid)+ {:type :document-card-ref+ :entity-id card-eid+ :message (str "document embeds card " card-eid " which is not in the export")}))+ card-eids)+ ;; Check entity links in href attributes+ (keep (fn [{:keys [kind entity-id]}]+ (when-not (store/exists? store kind entity-id)+ {:type :document-link+ :entity-id entity-id+ :message (str "document links to " (name kind) " " entity-id " which is not in the export")}))+ links)])))++;;; ===========================================================================+;;; deps.analysis helpers+;;; ===========================================================================++(defn- partition-errors+ "Split a set of check-entity errors into :bad-refs (MBQL issues) and+ :native-errors (SQL validation issues like :missing-column, :syntax-error)."+ [errors]+ (let [{native true, mbql false} (group-by #(contains? #{:missing-column :syntax-error :missing-table-alias} (:type %))+ errors)]+ {:bad-refs (vec mbql)+ :native-errors (when (seq native) (set native))}))++(defn- humanize-source-entity+ "Replace synthetic :source-entity-id integers with portable refs (entity-ids+ or path strings) so error output is meaningful."+ [store error]+ (if-let [src-id (:source-entity-id error)]+ (let [kind (:source-entity-type error)+ ref (case kind+ :table (some->> (store/id->ref store :table src-id) (str/join "."))+ :card (or (store/id->ref store :card src-id) src-id)+ src-id)]+ (assoc error :source-entity-id ref))+ error))++(defn- humanize-errors+ "Translate all synthetic IDs in a set of errors back to portable refs."+ [store errors]+ (when errors+ (into (empty errors) (map #(humanize-source-entity store %)) errors)))++;;; ===========================================================================+;;; Transform-specific checks+;;;+;;; Transforms have a source query (native or MBQL) and a target table.+;;; We validate the database ref and the source query via deps.analysis.+;;; ===========================================================================++(defn- check-transform+ "Run transform-specific semantic checks via deps.analysis/check-entity.+ Loads the transform from the store (which caches and assigns IDs), then+ delegates query validation to deps.analysis.+ Returns a result map with :unresolved, :native-errors, :bad-refs, :error, :sql keys."+ [store provider entity-id]+ (let [data (store/load-transform! store entity-id)]+ (if (not= "query" (get-in data [:source :type]))+ ;; Non-query transforms (e.g. python) — no query to validate+ {:unresolved []}+ ;; Query-based transforms — validate via deps.analysis+ (let [db-name (or (get-in data [:source :query :database])+ (:source_database_id data))+ unresolved (atom [])+ transform-id (:id data)+ sql (get-in data [:source :query :stages 0 :native])]+ ;; Check database ref+ (when (and db-name (not (store/exists? store :database db-name)))+ (swap! unresolved conj {:type :database :name db-name+ :message (str "transform references database " db-name " which is not in the schema")}))+ ;; Resolution failures from converting transform metadata+ (let [transform-meta (provider/transform-metadata store data)]+ (swap! unresolved into (or (:provider/resolution-failures transform-meta) []))+ ;; Validate via deps.analysis+ (let [errors (try+ (deps.analysis/check-entity provider :transform transform-id)+ (catch Exception e+ #{{:type :validation-exception-error :message (.getMessage e)}}))+ {:keys [bad-refs native-errors]} (partition-errors errors)+ native-errors (humanize-errors store native-errors)+ bad-refs (mapv #(humanize-source-entity store %) bad-refs)]+ (cond-> {:unresolved @unresolved}+ (seq bad-refs) (assoc :bad-refs bad-refs)+ (seq native-errors) (assoc :native-errors native-errors)+ sql (assoc :sql sql))))))))++;;; ===========================================================================+;;; Non-card entity checking — load from files, run common + type-specific checks+;;; ===========================================================================++(defn- id->path+ "Convert an integer ID back to a human-readable path string for error messages."+ [store id id-type]+ (case id-type+ :table (some->> (store/id->ref store :table id) (str/join "."))+ :field (some->> (store/id->ref store :field id) (str/join "."))+ :card (when-let [eid (store/id->ref store :card id)]+ (or (:name (store/cached-entity store :card eid)) eid))+ :database (store/id->ref store :database id)+ nil))++(declare extract-refs-from-card)++(defn- merge-refs+ "Merge multiple ref maps, concatenating and deduplicating each key."+ [& ref-maps]+ (reduce (fn [acc m]+ (reduce-kv (fn [a k v]+ (if (seq v)+ (update a k (fn [existing]+ (vec (distinct (into (or existing []) v)))))+ a))+ acc m))+ {} ref-maps))++(declare extract-entity-refs-from-raw-data)++(defn- extract-refs-from-query+ ([store query] (extract-refs-from-query store query nil #{}))+ ([store query provider visited]+ (let [table-ids (lib/all-source-table-ids query)+ card-ids (lib/all-source-card-ids query)+ cols (try (lib/returned-columns query) (catch Exception _ nil))+ tables (mapv #(id->path store % :table) table-ids)+ fields (when cols (->> cols (keep :id) (mapv #(id->path store % :field))))+ cards (mapv #(id->path store % :card) card-ids)+ transitive (when provider+ (for [cid card-ids :when (not (visited cid))]+ (extract-refs-from-card store provider cid (conj visited cid))))+ direct {:tables (vec (remove nil? tables))+ :fields (vec (remove nil? fields))+ :source-cards (vec (remove nil? cards))}]+ (apply merge-refs direct transitive))))++(defn- extract-refs-from-card [store provider card-id visited]+ (try+ (when-let [card (lib.metadata/card provider card-id)]+ (let [;; Get the raw card data from the store for entity ref extraction+ eid (store/id->ref store :card card-id)+ raw-data (when eid (store/cached-entity store :card eid))+ entity-refs (when raw-data (extract-entity-refs-from-raw-data raw-data))+ query-refs (when-let [dq (:dataset-query card)]+ (extract-refs-from-query store (lib/query provider dq) provider visited))]+ (merge-refs (or query-refs {}) entity-refs)))+ (catch Exception _ {})))++(defn- extract-entity-refs-from-raw-data+ "Extract snippet, measure, metric, and segment references from raw card data.+ These are extracted from the YAML structure before resolution, since the+ resolved lib query normalizes them away."+ [data]+ (let [stages (get-in data [:dataset_query :stages])]+ (reduce+ (fn [acc stage]+ (let [;; Snippets from template-tags+ snippets (->> (vals (:template-tags stage))+ (filter #(= "snippet" (get % :type)))+ (keep :snippet-name))+ ;; Measures and metrics from aggregation clauses+ agg-refs (->> (:aggregation stage)+ (filter vector?)+ (keep (fn [clause]+ (let [tag (first clause)]+ (case (str tag)+ "measure" {:kind :measures :ref (last clause)}+ "metric" {:kind :metrics :ref (last clause)}+ nil)))))+ ;; Segments from filter clauses+ seg-refs (->> (:filters stage)+ (filter vector?)+ (keep (fn [clause]+ (when (= "segment" (str (first clause)))+ {:kind :segments :ref (last clause)}))))]+ (cond-> acc+ (seq snippets) (update :snippets into snippets)+ (seq agg-refs) (as-> a (reduce (fn [m {:keys [kind ref]}]+ (update m kind (fnil conj []) ref))+ a agg-refs))+ (seq seg-refs) (update :segments into (map :ref seg-refs)))))+ {}+ (or stages []))))++;;; ===========================================================================+;;; Type-specific checks+;;; ===========================================================================++(defn- check-card-specific+ "Card-specific checks: query validation via deps.analysis + ref extraction."+ [store provider data]+ (let [card-id (:id data)+ card-meta (provider/card-metadata store data)+ resolution-failures (:provider/resolution-failures card-meta)+ errors (when-not (:provider/missing-database card-meta)+ (try+ (deps.analysis/check-entity provider :card card-id)+ (catch Exception e+ #{{:type :validation-exception-error :message (.getMessage e)}})))+ {:keys [bad-refs native-errors]} (partition-errors errors)+ native-errors (humanize-errors store native-errors)+ bad-refs (mapv #(humanize-source-entity store %) bad-refs)+ query-refs (try+ (let [card (lib.metadata/card provider card-id)+ query (lib/query provider (:dataset-query card))]+ (extract-refs-from-query store query provider #{card-id}))+ (catch Exception _ nil))+ entity-refs (extract-entity-refs-from-raw-data data)+ refs (when query-refs+ (cond-> query-refs+ (seq (:snippets entity-refs)) (assoc :snippets (vec (distinct (:snippets entity-refs))))+ (seq (:measures entity-refs)) (assoc :measures (vec (:measures entity-refs)))+ (seq (:metrics entity-refs)) (assoc :metrics (vec (:metrics entity-refs)))+ (seq (:segments entity-refs)) (assoc :segments (vec (:segments entity-refs)))))]+ {:card-id card-id+ :unresolved (into (or resolution-failures [])+ (when (:provider/missing-database card-meta)+ [{:type :database :name (:provider/missing-database card-meta)}]))+ :refs refs+ :bad-refs bad-refs+ :native-errors native-errors+ :error (when (:provider/missing-database card-meta)+ (str "Unknown database: " (:provider/missing-database card-meta)))}))++;;; ===========================================================================+;;; Unified entity checking+;;; ===========================================================================++(defn- check-entity+ "Check a single entity by kind. Runs common checks on all entities,+ plus type-specific checks (query validation for cards/transforms,+ structural checks for dashboards/documents).+ Returns [entity-id result-map]."+ [store provider kind entity-id]+ (try+ (let [load-fn (or (kind->load-fn kind) store/load-card!)+ data (load-fn store entity-id)+ _ (provider/set-database! provider (provider/database-name-for-entity store kind data))+ common (check-common store data)+ type-specific (case kind+ :card (check-card-specific store provider data)+ :dashboard {:unresolved (check-dashboard store data)}+ :document {:unresolved (check-document store data)}+ :transform (check-transform store provider entity-id)+ {})+ all-unresolved (into common (:unresolved type-specific))+ result (cond-> {:name (or (:name data) entity-id)+ :entity-id entity-id+ :unresolved all-unresolved+ :native-errors (:native-errors type-specific)+ :bad-refs (:bad-refs type-specific)+ :error (:error type-specific)+ :sql (:sql type-specific)}+ (:card-id type-specific) (assoc :card-id (:card-id type-specific))+ (:refs type-specific) (assoc :refs (:refs type-specific))+ (not= kind :card) (assoc :kind kind))]+ [entity-id result])+ (catch Exception e+ [entity-id {:name entity-id+ :entity-id entity-id+ :kind (when (not= kind :card) kind)+ :error (.getMessage e)}])+ (finally+ (provider/clear-database! provider))))++(defn- check-duplicate-entity-ids+ "Check for duplicate entity_ids in the index. Returns a map of+ synthetic key → result for each duplicate group."+ [index]+ (when-let [dupes (:duplicates index)]+ (into {}+ (for [{:keys [kind ref files]} dupes]+ [(str "duplicate:" ref)+ {:name (str "Duplicate " (name kind) " entity_id: " ref)+ :entity-id ref+ :kind kind+ :error (str "entity_id " ref " appears in " (count files) " files: "+ (str/join ", " files))}]))))++(def ^:private checked-kinds+ "Entity kinds that the checker validates, in processing order."+ [:card :dashboard :collection :document :measure :segment :transform])++(defn check-entities+ "Check all entities: cards, dashboards, collections, documents, transforms, etc.+ Returns `{entity-id result-map}`. See the Results processing section for the+ result-map shape.++ `schema-source` — a SchemaSource for resolving databases/tables/fields+ `assets-source` — an AssetsSource for resolving cards/snippets/transforms/segments+ `index` — a file index (see store/make-store)+ `entity-ids` — optional seq of entity-ids to check (defaults to all entities)"+ ([schema-source assets-source index]+ (check-entities schema-source assets-source index nil))+ ([schema-source assets-source index entity-ids]+ (binding [mu.fn/*enforce* false+ qp.i/*skip-middleware-because-app-db-access* true]+ (let [store (store/make-store schema-source assets-source index)+ provider (provider/make-provider store)+ results (if entity-ids+ ;; Check specific entities — look up their kind+ (into {}+ (for [eid entity-ids+ :let [kind (or (store/index-kind-of store eid) :card)]]+ (check-entity store provider kind eid)))+ ;; Check all entities by kind+ (into {}+ (for [kind checked-kinds+ eid (store/all-refs store kind)]+ (check-entity store provider kind eid))))+ dupes (check-duplicate-entity-ids index)]+ (merge results dupes)))))++;;; ===========================================================================+;;; Results processing — pure functions on result data+;;;+;;; Result map shape (per entity):+;;;+;;; :name string — entity name+;;; :entity-id string — portable entity ID+;;; :card-id int? — synthetic integer ID (cards only)+;;; :kind keyword? — :dashboard, :transform, etc. (non-card entities)+;;; :error string? — fatal error message (e.g. unknown database)+;;; :unresolved [map] — unresolved reference failures+;;; each {:type kw, :path vec?, :entity-id str?,+;;; :name str?, :message str?}+;;; :bad-refs [map] — query issues from deps.analysis+;;; (e.g. {:type :validation-exception-error, :message str})+;;; :native-errors #{map} — native SQL errors from deps.analysis+;;; (e.g. {:type :missing-column, :name str,+;;; :source-entity-type kw?, :source-entity-id str?})+;;; :refs map? — extracted references for display (cards only)+;;; {:tables [str], :fields [str], :source-cards [str]}+;;; :sql string? — source SQL (transforms with native queries)+;;;+;;; Summary map shape (from summarize-results):+;;;+;;; :total int — total entities checked+;;; :ok int — entities with no issues+;;; :errors int — entities with fatal errors+;;; :unresolved int — entities with unresolved references+;;; :native-errors int — entities with native SQL errors+;;; :issues int — entities with bad refs / query issues+;;; ===========================================================================++(defn result-status+ "Compute the status of a single entity result.+ Returns :error, :unresolved, :native-errors, :issues, or :ok (checked in priority order)."+ [result]+ (cond+ (:error result) :error+ (seq (:unresolved result)) :unresolved+ (seq (:native-errors result)) :native-errors+ (seq (:bad-refs result)) :issues+ :else :ok))++(defn summarize-results+ "Summarize check results into counts by status."+ [results]+ (let [by-status (group-by (comp result-status second) results)]+ {:total (count results)+ :ok (count (get by-status :ok))+ :errors (count (get by-status :error))+ :unresolved (count (get by-status :unresolved))+ :native-errors (count (get by-status :native-errors))+ :issues (count (get by-status :issues))}))++(defn results-by-status+ "Group results by status. Returns map of status → seq of [entity-id result]."+ [results]+ (group-by (comp result-status second) results))++(defn format-result+ "Format a single entity result as a human-readable string."+ [[entity-id result]]+ (let [lines (atom [(str "=== " (:name result) " [" entity-id "] ===")+ (if-let [kind (:kind result)]+ (str " Kind: " (name kind))+ (str " Card ID: " (:card-id result)))])+ {:keys [tables fields source-cards]} (:refs result)]+ (when (seq tables)+ (swap! lines conj (str " Tables: " (str/join ", " tables))))+ (when (seq fields)+ (swap! lines conj (str " Fields: " (str/join ", " fields))))+ (when (seq source-cards)+ (swap! lines conj (str " Source Cards: " (str/join ", " source-cards))))+ (when-let [unresolved (:unresolved result)]+ (swap! lines conj " UNRESOLVED REFERENCES:")+ (doseq [{:keys [type path entity-id name message]} unresolved]+ (swap! lines conj (str " - " (clojure.core/name type) ": "+ (or (some->> path (str/join ".")) entity-id name message)))))+ (when (seq (:native-errors result))+ (swap! lines conj " NATIVE SQL ERRORS:")+ (doseq [err (:native-errors result)]+ (swap! lines conj (str " - " (pr-str err))))+ (when-let [sql (:sql result)]+ (swap! lines conj (str " SQL: " sql))))+ (swap! lines conj (case (result-status result)+ :error (str " ERROR: " (:error result))+ :unresolved " Status: MISSING REFS"+ :native-errors " Status: NATIVE SQL ERRORS"+ :issues (str " Status: ISSUES FOUND\n"+ (str/join "\n" (map #(str " - " (pr-str %)) (:bad-refs result))))+ :ok " Status: OK"))+ (str/join "\n" @lines)))++(defn format-error+ "Format a single card error concisely for LLM consumption.+ Returns nil for :ok results. Only includes actionable error information."+ [[entity-id result]]+ (let [status (result-status result)]+ (when (not= :ok status)+ (let [kind-label (if-let [kind (:kind result)] (name kind) "card")+ lines (atom [(str kind-label ": " (:name result) " (entity_id: " entity-id ")")])]+ (when-let [unresolved (seq (:unresolved result))]+ (doseq [{:keys [type path entity-id name message]} unresolved]+ (swap! lines conj (str " unresolved " (clojure.core/name type) ": "+ (or (some->> path (str/join ".")) entity-id name message)))))+ (when (seq (:bad-refs result))+ (doseq [ref (:bad-refs result)]+ (swap! lines conj (str " bad ref: " (pr-str ref)))))+ (when (seq (:native-errors result))+ (doseq [err (:native-errors result)]+ (swap! lines conj (str " sql error: " (pr-str (dissoc err :source-entity-type :source-entity-id)))))+ (when-let [sql (:sql result)]+ (swap! lines conj (str " sql: " sql))))+ (when (:error result)+ (swap! lines conj (str " error: " (:error result))))+ (str/join "\n" @lines)))))++(defn- make-sources-and-index+ "Build schema and assets sources from `schema-dir` and `export-dir`,+ and a merged assets index (cards, dashboards, segments, etc.)."+ [export-dir schema-dir]+ (let [schema-source (serdes/make-database-source schema-dir)+ assets-source (serdes/make-source export-dir)+ index (serdes/source-index assets-source)]+ {:schema-source schema-source :assets-source assets-source :index index}))++(defn check+ "Check entities in an export directory against database schemas.++ `export-dir` — directory containing serdes-exported entities (collections/)+ `schema-dir` — directory containing serdes-exported database schemas+ `entity-ids` — optional seq of entity-ids to check (defaults to all entities)++ Returns `{:results results-map}` where results-map is `{entity-id result-map}`.+ See the Results processing section above for the result-map and summary-map shapes."+ ([export-dir schema-dir]+ (let [{:keys [schema-source assets-source index]} (make-sources-and-index export-dir schema-dir)]+ {:results (check-entities schema-source assets-source index)}))+ ([export-dir schema-dir entity-ids]+ (let [{:keys [schema-source assets-source index]} (make-sources-and-index export-dir schema-dir)]+ {:results (check-entities schema-source assets-source index entity-ids)})))++(defn setup+ "Create a store and provider for REPL iteration.+ Returns {:store store :provider provider :index index}.++ Usage:+ (def ctx (setup \"/path/to/export\" \"/path/to/schemas\"))+ (check-one ctx \"entity-id\")+ (check-one ctx \"entity-id\" :verbose true)"+ [export-dir schema-dir]+ (let [{:keys [schema-source assets-source index]} (make-sources-and-index export-dir schema-dir)+ store (binding [mu.fn/*enforce* false]+ (store/make-store schema-source assets-source index))+ provider (provider/make-provider store)]+ {:store store :provider provider :index index}))++(defn check-one+ "Check a single entity by entity-id using a pre-built context from [[setup]].+ Looks up the entity's kind from the store. Returns the result map.+ With :verbose true, also prints formatted output."+ [{:keys [store provider]} entity-id & {:keys [verbose]}]+ (binding [mu.fn/*enforce* false+ qp.i/*skip-middleware-because-app-db-access* true]+ (let [kind (or (store/index-kind-of store entity-id) :card)+ [_ result] (check-entity store provider kind entity-id)]+ (when verbose+ #_{:clj-kondo/ignore [:discouraged-var]}+ (println (format-result [entity-id result])))+ result)))++(comment+ ;; REPL workflow:+ (check "/Users/dan/projects/work/stats-remote-sync"+ "/tmp/metadata/metadata/databases")+ (setup+ "/Users/dan/projects/work/stats-remote-sync"+ "/Users/dan/projects/work/stats-remote-sync/databases")++ (check-one (setup "/Users/dan/projects/work/stats-remote-sync"+ "/tmp/metadata/metadata/databases")+ "2MNFkdc6EvgmUJU4E0ytF"))diff --git a/enterprise/backend/src/metabase_enterprise/checker/source.clj b/enterprise/backend/src/metabase_enterprise/checker/source.cljnew file mode 100644index 00000000000..a6ffee95911--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/source.clj@@ -0,0 +1,74 @@+(ns metabase-enterprise.checker.source+ "Protocols for resolving portable references to entity data.++ Two protocols separate the two concerns:++ - **SchemaSource** resolves database schema references (databases, tables, fields).+ Backed by the `--schema-dir` directory.+ - **AssetsSource** resolves serialized entity references (cards, snippets, transforms,+ segments). Backed by the `--export` directory.++ Portable references:+ - Database: string name+ - Table: [db-name schema-name table-name] where schema can be nil+ - Field: [db-name schema-name table-name field-name]+ - Card/snippet/transform/segment: entity-id string++ The checker assigns integer IDs; sources just say 'yes this exists, here's the data'+ or 'no, this reference is unresolved'.")++(set! *warn-on-reflection* true)++(defprotocol SchemaSource+ "Resolve database schema references to entity data."++ (resolve-database [this db-name]+ "Resolve database by name. Returns map with :name, :engine, :settings or nil.")++ (resolve-table [this table-path]+ "Resolve table by [db schema table]. Returns map with :name, :schema, :display-name, etc. or nil.")++ (resolve-field [this field-path]+ "Resolve field by [db schema table field]. Returns map with :name, :base-type, :semantic-type, etc. or nil.")++ (fields-for-table [this table-path]+ "Return a set of field paths belonging to the given table path [db schema table].")++ (all-field-paths [this]+ "Return all known field paths.")++ (all-database-names [this]+ "Return all known database names.")++ (all-table-paths [this]+ "Return all known table paths.")++ (tables-for-database [this db-name]+ "Return table paths belonging to the given database name."))++(defprotocol AssetsSource+ "Resolve serialized entity references to entity data."++ (resolve-card [this entity-id]+ "Resolve card by entity-id. Returns map with :name, :dataset-query, :result-metadata, etc. or nil.")++ (resolve-snippet [this entity-id]+ "Resolve native query snippet by entity-id. Returns map with :name, :content, etc. or nil.")++ (resolve-transform [this entity-id]+ "Resolve transform by entity-id. Returns map with :name, :source, etc. or nil.")++ (resolve-segment [this entity-id]+ "Resolve segment by entity-id. Returns map with :name, :definition, etc. or nil.")++ (resolve-dashboard [this entity-id]+ "Resolve dashboard by entity-id. Returns map with :name, :dashcards, :tabs, etc. or nil.")++ (resolve-collection [this entity-id]+ "Resolve collection by entity-id. Returns map with :name, :parent_id, etc. or nil.")++ (resolve-document [this entity-id]+ "Resolve document by entity-id. Returns map with :name, :document, etc. or nil.")++ (resolve-measure [this entity-id]+ "Resolve measure by entity-id. Returns map with :name, :definition, etc. or nil."))diff --git a/enterprise/backend/src/metabase_enterprise/checker/store.clj b/enterprise/backend/src/metabase_enterprise/checker/store.cljnew file mode 100644index 00000000000..9d7b4517eb4--- /dev/null+++ b/enterprise/backend/src/metabase_enterprise/checker/store.clj@@ -0,0 +1,267 @@+(ns metabase-enterprise.checker.store+ "The store knows what entities exist and can load them.++ It combines three things:+ - A **SchemaSource** for resolving databases, tables, and fields+ - An **AssetsSource** for resolving cards, snippets, transforms, and segments+ - A **file index** that enumerates all known entities by kind and ref++ On top of that it maintains:+ - A bidirectional **ID registry** (portable refs ↔ synthetic integer IDs,+ because lib requires integer IDs)+ - **Entity caches** (loaded data with :id stamped on)++ Entities are loaded lazily from sources and cached on first access.+ The store is an atom passed explicitly — no dynamic vars."+ (:require+ [metabase-enterprise.checker.source :as source]))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; Store creation+;;; ===========================================================================++(defn make-store+ "Create a fresh store for a checking session.++ `schema-source` is a SchemaSource for resolving databases/tables/fields.+ `assets-source` is an AssetsSource for resolving cards/snippets/transforms/segments.+ `index` is a file index: `{kind {ref file-path}}` — databases from the schema+ source, cards/dashboards/collections/transforms/segments/measures from assets."+ [schema-source assets-source index]+ (atom {:schema-source schema-source+ :assets-source assets-source+ :index index+ :id-counter 0+ :ref->id {} ; {kind {ref id}}+ :id->ref {} ; {kind {id ref}}+ :entities {}})) ; {kind {ref data}}++;;; ===========================================================================+;;; Index queries+;;; ===========================================================================++(defn all-refs+ "All known refs of a given kind from the file index."+ [store kind]+ (keys (get-in @store [:index kind])))++(defn exists?+ "Is `ref` of `kind` known to the store?+ Databases are checked via the schema source (not the assets index)."+ [store kind ref]+ (if (= kind :database)+ (contains? (set (source/all-database-names (:schema-source @store))) ref)+ (contains? (get-in @store [:index kind]) ref)))++(defn index-kind-of+ "Return the kind of an entity-id in the index, or nil if not found."+ [store entity-id]+ (cond+ (exists? store :collection entity-id) :collection+ (exists? store :card entity-id) :card+ (exists? store :dashboard entity-id) :dashboard+ (exists? store :document entity-id) :document+ (exists? store :measure entity-id) :measure+ (exists? store :segment entity-id) :segment+ (exists? store :snippet entity-id) :snippet+ (exists? store :transform entity-id) :transform+ :else nil))++(defn index-file+ "Get the file path for a ref of a given kind from the index, or nil."+ [store kind ref]+ (get-in @store [:index kind ref]))++(defn all-database-names+ "All database names from the schema source."+ [store]+ (source/all-database-names (:schema-source @store)))++(defn all-table-paths+ "All table paths from the schema source."+ [store]+ (source/all-table-paths (:schema-source @store)))++(defn all-field-paths+ "All field paths from the schema source."+ [store]+ (source/all-field-paths (:schema-source @store)))++(defn all-card-ids+ "All card entity-ids from the file index."+ [store]+ (all-refs store :card))++(defn tables-for-database+ "Table paths belonging to a specific database."+ [store db-name]+ (source/tables-for-database (:schema-source @store) db-name))++(defn fields-for-table+ "Field paths belonging to a specific table path."+ [store table-path]+ (source/fields-for-table (:schema-source @store) table-path))++;;; ===========================================================================+;;; ID registry — assign synthetic integer IDs to portable refs+;;; ===========================================================================++(defn- next-id! [store]+ (:id-counter (swap! store update :id-counter inc)))++(defn get-or-assign!+ "Return the integer ID for `ref` under `kind`, assigning one if needed."+ [store kind ref]+ (or (get-in @store [:ref->id kind ref])+ (let [id (next-id! store)]+ (swap! store (fn [s]+ (-> s+ (assoc-in [:ref->id kind ref] id)+ (assoc-in [:id->ref kind id] ref))))+ id)))++(defn id->ref+ "Look up the portable ref for an integer ID. Returns nil if unknown."+ [store kind id]+ (get-in @store [:id->ref kind id]))++(defn ref->id+ "Look up the integer ID for a portable ref. Returns nil if not yet assigned."+ [store kind ref]+ (get-in @store [:ref->id kind ref]))++;;; ===========================================================================+;;; Entity loading — lazy load from source, cache with assigned IDs+;;; ===========================================================================++(defn- schema-source [store]+ (:schema-source @store))++(defn- assets-source [store]+ (:assets-source @store))++(defn cached-entity+ "Look up a cached entity by kind and ref. Returns nil if not cached."+ [store kind ref]+ (get-in @store [:entities kind ref]))++(defn- cache-entity! [store kind ref data]+ (swap! store assoc-in [:entities kind ref] data)+ data)++;; --- Schema entities ---++(defn load-database!+ "Load and cache a database, assigning it an integer ID. Returns data or nil."+ [store db-name]+ (or (cached-entity store :database db-name)+ (when-let [data (source/resolve-database (schema-source store) db-name)]+ (let [id (get-or-assign! store :database db-name)]+ (cache-entity! store :database db-name (assoc data :id id))))))++(defn ensure-table-id!+ "Ensure a table path has an assigned integer ID and minimal metadata cached.+ Does NOT parse the table YAML — derives name/schema from the path.+ Use [[load-table!]] when full table data is needed."+ [store [db-name schema table-name :as table-path]]+ (or (cached-entity store :table table-path)+ (let [id (get-or-assign! store :table table-path)+ db-id (get-or-assign! store :database db-name)]+ (cache-entity! store :table table-path+ {:id id+ :db_id db-id+ :name table-name+ :schema schema}))))++(defn load-table!+ "Load and cache a table, assigning it and its database integer IDs.+ Parses the table YAML for full metadata."+ [store table-path]+ (let [cached (cached-entity store :table table-path)]+ (if (and cached (contains? cached :display_name))+ ;; Already fully loaded+ cached+ ;; Load from YAML, merging with any existing minimal cache+ (when-let [data (source/resolve-table (schema-source store) table-path)]+ (let [[db-name _ _] table-path+ id (get-or-assign! store :table table-path)+ db-id (get-or-assign! store :database db-name)]+ (cache-entity! store :table table-path (assoc data :id id :db_id db-id)))))))++(defn load-field!+ "Load and cache a field, assigning it and its table integer IDs."+ [store field-path]+ (or (cached-entity store :field field-path)+ (when-let [data (source/resolve-field (schema-source store) field-path)]+ (let [[db-name schema table-name _] field-path+ id (get-or-assign! store :field field-path)+ table-id (get-or-assign! store :table [db-name schema table-name])]+ (cache-entity! store :field field-path (assoc data :id id :table_id table-id))))))++;; --- Asset entities ---++(defn load-card!+ "Load and cache a card, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :card entity-id)+ (when-let [data (source/resolve-card (assets-source store) entity-id)]+ (let [id (get-or-assign! store :card entity-id)]+ (cache-entity! store :card entity-id (assoc data :id id))))))++(defn load-snippet!+ "Load and cache a snippet, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :snippet entity-id)+ (when-let [data (source/resolve-snippet (assets-source store) entity-id)]+ (let [id (get-or-assign! store :snippet entity-id)]+ (cache-entity! store :snippet entity-id (assoc data :id id))))))++(defn load-transform!+ "Load and cache a transform, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :transform entity-id)+ (when-let [data (source/resolve-transform (assets-source store) entity-id)]+ (let [id (get-or-assign! store :transform entity-id)]+ (cache-entity! store :transform entity-id (assoc data :id id))))))++(defn load-segment!+ "Load and cache a segment, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :segment entity-id)+ (when-let [data (source/resolve-segment (assets-source store) entity-id)]+ (let [id (get-or-assign! store :segment entity-id)]+ (cache-entity! store :segment entity-id (assoc data :id id))))))++(defn load-dashboard!+ "Load and cache a dashboard, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :dashboard entity-id)+ (when-let [data (source/resolve-dashboard (assets-source store) entity-id)]+ (let [id (get-or-assign! store :dashboard entity-id)]+ (cache-entity! store :dashboard entity-id (assoc data :id id))))))++(defn load-collection!+ "Load and cache a collection, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :collection entity-id)+ (when-let [data (source/resolve-collection (assets-source store) entity-id)]+ (let [id (get-or-assign! store :collection entity-id)]+ (cache-entity! store :collection entity-id (assoc data :id id))))))++(defn load-document!+ "Load and cache a document, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :document entity-id)+ (when-let [data (source/resolve-document (assets-source store) entity-id)]+ (let [id (get-or-assign! store :document entity-id)]+ (cache-entity! store :document entity-id (assoc data :id id))))))++(defn load-measure!+ "Load and cache a measure, assigning it an integer ID."+ [store entity-id]+ (or (cached-entity store :measure entity-id)+ (when-let [data (source/resolve-measure (assets-source store) entity-id)]+ (let [id (get-or-assign! store :measure entity-id)]+ (cache-entity! store :measure entity-id (assoc data :id id))))))diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj b/enterprise/backend/src/metabase_enterprise/dependencies/analysis.cljindex 7af468f8f83..2b6656a89f9 100644--- a/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj+++ b/enterprise/backend/src/metabase_enterprise/dependencies/analysis.clj@@ -26,19 +26,18 @@(deps.native/validate-native-query driver query)(into #{} (remove :soft?) (lib/find-bad-refs-with-source query))))-(defmulti check-entity- "Given a metadata provider, an entity type and an entity id, find any bad refs in that entity."+(defmulti ^:private -check-entity+ "Implementation multimethod for [[check-entity]]. Extend this to add support+ for new entity types."{:arglists '([metadata-provider entity-type entity-id])}(fn [_metadata-provider entity-type _entity-id]entity-type))-(defmethod check-entity :default+(defmethod -check-entity :default[_metadata-provider _entity-type _entity-id]nil)-(mu/defmethod check-entity :card :- [:set [:ref ::lib.schema.validate/error-with-source]]- "Given a `MetadataProvider` and a card ID, analyses the card's query to find any bad refs or other issues.- Returns any findings with source information, and `nil` for a clean query."+(mu/defmethod -check-entity :card :- [:set [:ref ::lib.schema.validate/error-with-source]][metadata-provider :- ::lib.schema.metadata/metadata-provider_entity-typecard-id :- ::lib.schema.id/card]@@ -46,10 +45,7 @@driver (:engine (lib.metadata/database query))](check-query driver query)))-(mu/defmethod check-entity :transform :- [:set [:ref ::lib.schema.validate/error-with-source]]- "Given a `MetadataProvider` and a transform ID, analyses the transform's query to find any bad refs or other issues.-- Returns any findings with source information, and `nil` for a clean transform."+(mu/defmethod -check-entity :transform :- [:set [:ref ::lib.schema.validate/error-with-source]][metadata-provider :- ::lib.schema.metadata/metadata-provider_entity-typetransform-id :- ::lib.schema.id/transform]@@ -67,7 +63,7 @@(cond-> (check-query driver query)duplicated-fields (into duplicated-fields))))-(mu/defmethod check-entity :segment :- [:set [:ref ::lib.schema.validate/error-with-source]]+(mu/defmethod -check-entity :segment :- [:set [:ref ::lib.schema.validate/error-with-source]][metadata-provider :- ::lib.schema.metadata/metadata-provider_entity-typesegment-id :- ::lib.schema.id/segment]@@ -76,3 +72,14 @@(lib/query metadata-provider))driver (:engine (lib.metadata/database query))](check-query driver query)))++(defn check-entity+ "Check a single entity for bad refs against a MetadataProvider.++ Returns a set of errors with source information, or nil if the entity is clean.+ Dispatches to entity-type-specific analysis (MBQL bad refs, native SQL validation,+ transform duplicate columns, etc.).++ Supported entity types: :card, :transform, :segment. Other types return nil."+ [metadata-provider entity-type entity-id]+ (-check-entity metadata-provider entity-type entity-id))diff --git a/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj b/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.cljindex d440a2d1b1d..231bc3588a7 100644--- a/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj+++ b/enterprise/backend/src/metabase_enterprise/dependencies/native_validation.clj@@ -26,10 +26,11 @@compiled (qp.compile/compile-with-inline-parameters with-params)](lib/native-query with-params (:query compiled))))-(defn- has-card-template-tags?- "Returns true if the query has any card-type template tags."+(defn- has-substitutable-template-tags?+ "Returns true if the query has any card or table template tags that need+ placeholder substitution before compilation."[query]- (some #(= (:type %) :card) (lib/all-template-tags query)))+ (some #(#{:card :table} (:type %)) (lib/all-template-tags query)))(def ^:private card-placeholder-prefix"Prefix used to generate placeholder table names for card references.@@ -44,9 +45,15 @@(when (str/starts-with? lower card-placeholder-prefix)(parse-long (subs lower (count card-placeholder-prefix))))))+(def ^:private table-placeholder-prefix+ "Prefix used to generate placeholder table names for table-type template tags.+ Table tag {{table_one}} becomes mb__validat_table__N in the SQL sent to SQLGlot."+ "mb__validat_table__")+(defn- compile-toplevel-query- "Compile a query replacing card references with placeholder table names.+ "Compile a query replacing card and table references with placeholder table names.Card refs like {{#1}} are replaced with mb__validat_card__1.+ Table refs like {{table_one}} are replaced with mb__validat_table__N.Returns the compiled native-only-query, or nil if the original SQL containsthe placeholder prefix (collision guard)."@@ -54,9 +61,12 @@(let [stage (first (:stages query))sql (:native stage)ttags (:template-tags stage)- card-tags (into {} (filter #(= (:type (val %)) :card)) ttags)]- (when-not (str/includes? sql card-placeholder-prefix)- (let [modified-sql+ card-tags (into {} (filter #(= (:type (val %)) :card)) ttags)+ table-tags (into {} (filter #(= (:type (val %)) :table)) ttags)]+ (when-not (or (str/includes? sql card-placeholder-prefix)+ (str/includes? sql table-placeholder-prefix))+ (let [;; Replace card tags+ sql-after-cards(reduce (fn [s [tag-name tag]](str/replace s(str "{{" tag-name "}}")@@ -64,24 +74,40 @@sqlcard-tags)- non-card-tags- (into {} (remove #(= (:type (val %)) :card)) ttags)+ ;; Replace table tags+ modified-sql+ (reduce (fn [s [tag-name tag]]+ (str/replace s+ (str "{{" tag-name "}}")+ (str table-placeholder-prefix (:table-id tag))))+ sql-after-cards+ table-tags)++ remaining-tags+ (into {} (remove #(#{:card :table} (:type (val %)))) ttags)modified-query(-> query(assoc-in [:stages 0 :native] modified-sql)- (assoc-in [:stages 0 :template-tags] non-card-tags))]+ (assoc-in [:stages 0 :template-tags] remaining-tags))](compile-query modified-query)))))+(defn- table-placeholder?+ "True if table-name is a table placeholder (e.g. \"mb__validat_table__65478\")."+ [table-name]+ (when table-name+ (str/starts-with? (u/lower-case-en (str table-name)) table-placeholder-prefix)))+(defn- table-deps"Returns the set of table dep maps (e.g. {:table \"products\" :schema \"public\"})referenced directly in the compiled native query.- Excludes card placeholder tables (mb__validat_card__N)."+ Excludes card and table placeholder tables."[driver compiled](into #{}(filter (fn [dep](and (:table dep)- (not (parse-card-placeholder (:table dep))))))+ (not (parse-card-placeholder (:table dep)))+ (not (table-placeholder? (:table dep))))))(driver/native-query-deps driver compiled)))(defn- extract-source-entity@@ -237,10 +263,13 @@[driver :- :keywordquery :- ::lib.schema/query](into #{}- ;; Strip :unknown source attribution — consumers should not distinguish between- ;; "we tried and couldn't determine the source" vs "no source info available".- (map #(cond-> % (= :unknown (:source-entity-type %)) (dissoc :source-entity-type :source-entity-id)))- (if (has-card-template-tags? query)+ (comp+ ;; Strip :unknown source attribution — consumers should not distinguish between+ ;; "we tried and couldn't determine the source" vs "no source info available".+ (map #(cond-> % (= :unknown (:source-entity-type %)) (dissoc :source-entity-type :source-entity-id)))+ ;; Remove errors referencing table placeholder names+ (remove #(some-> (:name %) (str/starts-with? table-placeholder-prefix))))+ (if (has-substitutable-template-tags? query)(if-let [compiled (compile-toplevel-query query)](let [errors (validate-with-sources driver compiled true)](if (empty? errors)@@ -254,13 +283,22 @@errors(fallback-enrich driver compiled errors))))))+(defn- has-table-template-tags?+ "Returns true if the query has any table-type template tags."+ [query]+ (some #(= (:type %) :table) (lib/all-template-tags query)))+(mu/defn native-result-metadata- "Compiles a (native) query and calculates its result metadata"+ "Compiles a (native) query and calculates its result metadata.+ Returns nil for queries with table-type template tags since the actual+ table (and therefore columns) is unknown at check time."[driver :- :keywordquery :- ::lib.schema/query]- (->> query- compile-query- (driver/native-result-metadata driver)))+ (when-not (has-table-template-tags? query)+ (let [compiled (if (has-substitutable-template-tags? query)+ (or (compile-toplevel-query query) (compile-query query))+ (compile-query query))]+ (driver/native-result-metadata driver compiled))))(mu/defn native-query-deps :- [:set[:ordiff --git a/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj b/enterprise/backend/src/metabase_enterprise/serialization/cmd.cljindex 9f3615724ed..a11a9713aa6 100644--- a/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj+++ b/enterprise/backend/src/metabase_enterprise/serialization/cmd.clj@@ -144,6 +144,9 @@(log/info (format "Export to '%s' complete!" path) (u/emoji "🚛💨 📦"))report))+(comment+ (v2-dump! "/tmp/serdes" {}))+(defn seed-entity-ids!"Add entity IDs for instances of serializable models that don't already have them.diff --git a/enterprise/backend/test/metabase_enterprise/checker/e2e_test.clj b/enterprise/backend/test/metabase_enterprise/checker/e2e_test.cljnew file mode 100644index 00000000000..880e5ba4fe9--- /dev/null+++ b/enterprise/backend/test/metabase_enterprise/checker/e2e_test.clj@@ -0,0 +1,137 @@+(ns metabase-enterprise.checker.e2e-test+ "End-to-end test running the checker against the serialization baseline export.++ This exercises the full pipeline: YAML parsing → index building → store →+ provider → query validation → structural checks → results.++ The baseline at test_resources/serialization_baseline is a real serdes export+ with cards, dashboards, transforms, snippets, and documents backed by+ the sample database schema."+ (:require+ [clojure.string :as str]+ [clojure.test :refer :all]+ [clojure.walk :as walk]+ [metabase-enterprise.checker.semantic :as checker]))++(set! *warn-on-reflection* true)++(def ^:private export-dir "test_resources/serialization_baseline")+(def ^:private schema-dir "test_resources/serialization_baseline/databases")++(def ^:private baseline-results+ "Checker results for the baseline export. Computed once, shared across all tests."+ (delay (:results (checker/check export-dir schema-dir))))++(defn- failures-report+ "Format failing results for assertion messages."+ [results status]+ (->> results+ (filter #(= status (checker/result-status (second %))))+ (map checker/format-error)+ (str/join "\n")))++(deftest baseline-export-no-errors-test+ (testing "The serialization baseline export passes all checks with zero errors"+ (let [results @baseline-results+ summary (checker/summarize-results results)]+ (is (=? {:ok pos-int?+ :errors 0+ :unresolved 0+ :native-errors 0+ :issues 0}+ summary))+ (is (> (:total summary) 45)+ "Should check cards plus non-card entities (dashboards, transforms, etc.)")+ (is (pos? (:total summary))+ "Should check at least one entity")+ (is (zero? (:errors summary))+ (str "Errors:\n" (failures-report results :error)))+ (is (zero? (:unresolved summary))+ (str "Unresolved:\n" (failures-report results :unresolved)))+ (is (zero? (:native-errors summary))+ (str "Native SQL errors:\n" (failures-report results :native-errors)))+ (is (zero? (:issues summary))+ (str "Issues:\n" (failures-report results :issues))))))++(deftest baseline-export-card-refs-populated-test+ (testing "Checked cards have populated ref data (tables, fields)"+ (let [card-results (->> @baseline-results vals (filter :refs))]+ (is (pos? (count card-results))+ "At least some cards should have refs")+ (is (some #(seq (get-in % [:refs :tables])) card-results)+ "At least one card should reference tables")+ (is (some #(seq (get-in % [:refs :fields])) card-results)+ "At least one card should reference fields"))))++;;; ===========================================================================+;;; Card shape assertions — verify detailed analysis on specific cards+;;; ===========================================================================++(defn- normalize [x] (walk/postwalk x (cond-> x (sequential? x) set)))++(deftest joins-card-shape-test+ (testing "MBQL card with 3 joins resolves all tables and fields"+ (let [result (get @baseline-results "wpW9JO5MsHQd0-7WrR7gN")]+ (is (=? (normalize+ {:card-id pos-int?+ :name "Joins - Strategies and Compound Conditions",+ :entity-id "wpW9JO5MsHQd0-7WrR7gN",+ :refs {:tables ["Sample Database.PUBLIC.PRODUCTS"+ "Sample Database.PUBLIC.REVIEWS"+ "Sample Database.PUBLIC.ORDERS"+ "Sample Database.PUBLIC.PEOPLE"],+ :fields ["Sample Database.PUBLIC.ORDERS.ID"+ "Sample Database.PUBLIC.ORDERS.USER_ID"+ "Sample Database.PUBLIC.ORDERS.PRODUCT_ID"+ "Sample Database.PUBLIC.ORDERS.SUBTOTAL"+ "Sample Database.PUBLIC.ORDERS.TAX"+ "Sample Database.PUBLIC.ORDERS.TOTAL"+ "Sample Database.PUBLIC.ORDERS.DISCOUNT"+ "Sample Database.PUBLIC.ORDERS.CREATED_AT"+ "Sample Database.PUBLIC.ORDERS.QUANTITY"+ "Sample Database.PUBLIC.PRODUCTS.ID"+ "Sample Database.PUBLIC.PRODUCTS.EAN"+ "Sample Database.PUBLIC.PRODUCTS.TITLE"+ "Sample Database.PUBLIC.PRODUCTS.CATEGORY"+ "Sample Database.PUBLIC.PRODUCTS.VENDOR"+ "Sample Database.PUBLIC.PRODUCTS.PRICE"+ "Sample Database.PUBLIC.PRODUCTS.RATING"+ "Sample Database.PUBLIC.PRODUCTS.CREATED_AT"+ "Sample Database.PUBLIC.PEOPLE.NAME"+ "Sample Database.PUBLIC.PEOPLE.STATE"]}})+ (normalize result)))+ ;; REVIEWS fields are NOT in the list because the join has `fields: none`+ ;; — only join condition fields from REVIEWS are used, not selected.+ ;; The REVIEWS table IS resolved though (checked above).+ (is (-> result :source-cards empty?)))))++;;;; note: these are all sets here for equality's sake. But they traditionally are vectors++(deftest measure-metric-segment-refs-test+ (testing "Card with measure, metric, and segment references tracks them all"+ (let [result (get @baseline-results "kBjQ5VXJ5z3vYSW72J6qa")]+ (is (=? {:card-id pos-int?,+ :name "Metric, Measure, and Segment References",+ :entity-id "kBjQ5VXJ5z3vYSW72J6qa",+ :refs {:tables ["Sample Database.PUBLIC.ORDERS"],+ :source-cards ["Total Revenue"],+ :measures ["DBap2523KuN4Lt-cYrUjF"],+ :metrics ["IW8kbqZVaMxdGtCM2F4U6"],+ :segments ["OaFXAvEzPvu9ZXW5PUMRa"]}}+ result)))))++(deftest transitive-entity-refs-test+ (testing "Source card reference transitively surfaces the inner card's snippets, measures, segments"+ (let [;; "Native Query - Card and Snippet References" sources from+ ;; "Basic Aggregations" AND uses a snippet.+ snippet-card (get @baseline-results "HiBFSt0BNx5s5MxVDLLKB")]+ ;; The snippet card itself uses a snippet and sources from Basic Aggregations.+ ;; Basic Aggregations references ORDERS table — that should appear transitively.+ (is (=? {:card-id pos-int?+ :name "Native Query - Card and Snippet References",+ :entity-id "HiBFSt0BNx5s5MxVDLLKB",+ :refs {:source-cards ["Basic Aggregations"],+ :tables ["Sample Database.PUBLIC.ORDERS"],+ :fields ["Sample Database.PUBLIC.ORDERS.CREATED_AT"],+ :snippets ["Active Product Filter"]}}+ snippet-card)))))diff --git a/enterprise/backend/test/metabase_enterprise/checker/format/serdes_test.clj b/enterprise/backend/test/metabase_enterprise/checker/format/serdes_test.cljnew file mode 100644index 00000000000..36a5de28947--- /dev/null+++ b/enterprise/backend/test/metabase_enterprise/checker/format/serdes_test.clj@@ -0,0 +1,396 @@+(ns metabase-enterprise.checker.format.serdes-test+ "Tests for the serdes format module — YAML extraction, file indexing, and source resolution."+ (:require+ [clojure.java.io :as io]+ [clojure.test :refer [deftest is testing]]+ [metabase-enterprise.checker.format.serdes :as serdes]+ [metabase-enterprise.checker.source :as source]+ [metabase.util.yaml :as yaml]))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; Helpers+;;; ===========================================================================++(def ^:private fixtures-dir "test_resources/yaml_checks")++(defn- fixtures-path []+ (let [f (io/file fixtures-dir)]+ (if (.isAbsolute f)+ fixtures-dir+ (.getPath (io/file (System/getProperty "user.dir") fixtures-dir)))))++(defn- with-temp-dir+ "Create a temp directory, call (f dir-path), then clean up."+ [f]+ (let [dir (java.io.File/createTempFile "serdes-test" "")]+ (.delete dir)+ (.mkdirs dir)+ (try+ (f (.getPath dir))+ (finally+ (doseq [fl (reverse (file-seq dir))]+ (.delete ^java.io.File fl))))))++(defn- write-yaml!+ "Write a YAML file, creating parent directories as needed."+ [dir & path-and-data]+ (let [data (last path-and-data)+ path-segs (butlast path-and-data)+ ^java.io.File file (apply io/file dir (map str path-segs))]+ (.mkdirs (.getParentFile file))+ (spit file (yaml/generate-string data))+ (.getPath file)))++;;; ===========================================================================+;;; YAML extraction tests (pure, no files needed)+;;; ===========================================================================++(deftest extract-entity-id-from-fixture-test+ (testing "extract-entity-id reads entity_id from a card YAML"+ (let [card-file (io/file (fixtures-path) "collections" "cards" "simple-orders_simple_orders.yaml")]+ (is (= "simple-orders" (serdes/extract-entity-id (.getPath card-file)))))))++(deftest extract-model-from-fixture-test+ (testing "extract-model reads the serdes model from a card YAML"+ (let [card-file (io/file (fixtures-path) "collections" "cards" "simple-orders_simple_orders.yaml")]+ (is (= "Card" (serdes/extract-model (.getPath card-file)))))))++(deftest extract-entity-id-from-temp-file-test+ (testing "extract-entity-id with various formats"+ (with-temp-dir+ (fn [dir]+ (let [f (write-yaml! dir "test.yaml" {:name "X" :entity_id "xYz123AbC"})]+ (is (= "xYz123AbC" (serdes/extract-entity-id f))))))))++;;; ===========================================================================+;;; Index tests — databases are indexed, tables/fields are on-demand+;;; ===========================================================================++(deftest build-file-index-databases-test+ (testing "build-file-index indexes databases from fixtures"+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]+ (is (contains? (:database index) "Test Database"))+ (is (contains? (:database index) "SQLite DB")))))++(deftest build-file-index-cards-test+ (testing "build-file-index indexes cards by entity_id"+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]+ (is (contains? (:card index) "simple-orders"))+ (is (contains? (:card index) "native-orders"))+ (is (contains? (:card index) "orders-with-products")))))++(deftest build-file-index-no-duplicates-test+ (testing "fixture index has no duplicates"+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]+ (is (nil? (:duplicates index))))))++(deftest build-database-dir-index-test+ (testing "build-database-dir-index indexes databases and provides db-name->dir mapping"+ (let [db-dir (str (fixtures-path) "/databases")+ {:keys [index db-name->dir]} (serdes/build-database-dir-index db-dir)]+ (is (contains? (:database index) "Test Database"))+ (is (contains? (:database index) "SQLite DB"))+ (is (contains? db-name->dir "Test Database"))+ (is (contains? db-name->dir "SQLite DB"))+ ;; Tables are NOT in the index — they're resolved on demand+ (is (nil? (:table index)))+ ;; Should not contain cards (databases dir has no collections)+ (is (nil? (:card index))))))++;;; ===========================================================================+;;; db-name->dir mapping tests+;;; ===========================================================================++(deftest db-name-to-dir-mapping-test+ (testing "db-name->dir maps real database names to directory names"+ (with-temp-dir+ (fn [dir]+ ;; Create a database with a slugified directory name but real name in YAML+ (write-yaml! dir "databases" "my_fancy_db" "my_fancy_db.yaml"+ {:name "My Fancy DB" :engine "h2"})+ (let [{:keys [db-name->dir]} (serdes/build-database-dir-index (str dir "/databases"))]+ (is (= "my_fancy_db" (get db-name->dir "My Fancy DB")))))))++ (testing "db-name->dir includes ALL databases, not just some"+ (with-temp-dir+ (fn [dir]+ (doseq [i (range 10)]+ (let [slug (str "db_" i)]+ (write-yaml! dir "databases" slug (str slug ".yaml")+ {:name (str "Database " i) :engine "h2"})))+ (let [{:keys [db-name->dir]} (serdes/build-database-dir-index (str dir "/databases"))]+ (is (= 10 (count db-name->dir)))+ (doseq [i (range 10)]+ (is (= (str "db_" i) (get db-name->dir (str "Database " i))))))))))++;;; ===========================================================================+;;; SchemaSource protocol — on-demand resolution tests+;;; ===========================================================================++(deftest source-resolves-database-test+ (testing "SerdesSource resolves databases by name"+ (let [source (serdes/make-source (fixtures-path))+ db (source/resolve-database source "Test Database")]+ (is (some? db))+ (is (= "Test Database" (:name db))))))++(deftest source-resolves-table-on-demand-test+ (testing "SerdesSource resolves tables by path on demand (no pre-indexing)"+ (let [source (serdes/make-source (fixtures-path))+ table (source/resolve-table source ["Test Database" "public" "orders"])]+ (is (some? table))+ (is (= "orders" (:name table))))+ (testing "schema-less tables"+ (let [source (serdes/make-source (fixtures-path))+ table (source/resolve-table source ["SQLite DB" nil "orders"])]+ (is (some? table))+ (is (= "orders" (:name table)))))))++(deftest source-resolves-field-on-demand-test+ (testing "SerdesSource resolves fields by path on demand"+ (let [source (serdes/make-source (fixtures-path))+ field (source/resolve-field source ["Test Database" "public" "orders" "id"])]+ (is (some? field))+ (is (= "id" (:name field))))+ (testing "schema-less fields"+ (let [source (serdes/make-source (fixtures-path))+ field (source/resolve-field source ["SQLite DB" nil "orders" "id"])]+ (is (some? field))+ (is (= "id" (:name field)))))))++(deftest source-resolves-card-test+ (testing "SerdesSource resolves cards by entity-id"+ (let [source (serdes/make-source (fixtures-path))+ card (source/resolve-card source "simple-orders")]+ (is (some? card))+ (is (= "Simple Orders" (:name card))))))++(deftest source-returns-nil-for-unknown-test+ (testing "SerdesSource returns nil for unknown entities"+ (let [source (serdes/make-source (fixtures-path))]+ (is (nil? (source/resolve-database source "Nonexistent")))+ (is (nil? (source/resolve-table source ["DB" "x" "y"])))+ (is (nil? (source/resolve-field source ["DB" "x" "y" "z"])))+ (is (nil? (source/resolve-card source "no-such-card"))))))++;;; ===========================================================================+;;; fields-for-table and all-table-paths — on-demand enumeration+;;; ===========================================================================++(deftest fields-for-table-test+ (testing "fields-for-table returns field paths for a table"+ (let [source (serdes/make-source (fixtures-path))+ fields (source/fields-for-table source ["Test Database" "public" "orders"])]+ (is (set? fields))+ (is (contains? fields ["Test Database" "public" "orders" "id"]))+ (is (contains? fields ["Test Database" "public" "orders" "total"]))))+ (testing "fields-for-table works for schema-less tables"+ (let [source (serdes/make-source (fixtures-path))+ fields (source/fields-for-table source ["SQLite DB" nil "orders"])]+ (is (set? fields))+ (is (contains? fields ["SQLite DB" nil "orders" "id"]))))+ (testing "fields-for-table returns nil for unknown table"+ (let [source (serdes/make-source (fixtures-path))]+ (is (nil? (source/fields-for-table source ["Nope" "x" "y"]))))))++(deftest all-table-paths-test+ (testing "all-table-paths enumerates all tables across databases"+ (let [source (serdes/make-source (fixtures-path))+ tables (source/all-table-paths source)]+ (is (seq tables))+ (is (some #(= ["Test Database" "public" "orders"] %) tables))+ (is (some #(= ["SQLite DB" nil "orders"] %) tables)))))++(deftest all-database-names-test+ (testing "all-database-names returns database names"+ (let [source (serdes/make-source (fixtures-path))]+ (is (= #{"Test Database" "SQLite DB"} (set (source/all-database-names source)))))))++;;; ===========================================================================+;;; Slugified directory names — critical correctness tests+;;; ===========================================================================++(deftest slugified-db-dir-resolves-correctly-test+ (testing "databases with slugified directory names resolve correctly"+ (with-temp-dir+ (fn [dir]+ (let [db-dir (str dir "/databases")]+ ;; Create a database with slugified dir but real name in YAML+ (write-yaml! dir "databases" "analytics_data_warehouse" "analytics_data_warehouse.yaml"+ {:name "Analytics Data Warehouse" :engine "postgres"})+ ;; Create a table under it+ (write-yaml! dir "databases" "analytics_data_warehouse" "schemas" "public" "tables" "orders" "orders.yaml"+ {:name "orders" :schema "public"+ :serdes/meta [{:id "Analytics Data Warehouse" :model "Database"}+ {:id "public" :model "Schema"}+ {:id "orders" :model "Table"}]})+ ;; Create a field+ (write-yaml! dir "databases" "analytics_data_warehouse" "schemas" "public" "tables" "orders" "fields" "id.yaml"+ {:name "id" :base_type "type/Integer"+ :serdes/meta [{:id "Analytics Data Warehouse" :model "Database"}+ {:id "public" :model "Schema"}+ {:id "orders" :model "Table"}+ {:id "id" :model "Field"}]})+ (let [source (serdes/make-database-source db-dir)]+ ;; Database resolves by real name+ (is (some? (source/resolve-database source "Analytics Data Warehouse")))+ ;; Table resolves using real db name+ (is (some? (source/resolve-table source ["Analytics Data Warehouse" "public" "orders"])))+ ;; Field resolves using real db name+ (let [field (source/resolve-field source ["Analytics Data Warehouse" "public" "orders" "id"])]+ (is (some? field))+ (is (= "id" (:name field))))+ ;; fields-for-table works+ (let [fields (source/fields-for-table source ["Analytics Data Warehouse" "public" "orders"])]+ (is (contains? fields ["Analytics Data Warehouse" "public" "orders" "id"])))))))))++;;; ===========================================================================+;;; Performance guards — indexing should NOT parse field/table YAMLs+;;; ===========================================================================++(deftest index-does-not-contain-tables-or-fields-test+ (testing "the file index only contains databases — tables and fields are resolved on demand"+ (let [db-dir (str (fixtures-path) "/databases")+ {:keys [index]} (serdes/build-database-dir-index db-dir)]+ (is (seq (:database index)) "databases should be in the index")+ (is (nil? (:table index)) "tables should NOT be in the index")+ (is (nil? (:field index)) "fields should NOT be in the index"))))++;;; ===========================================================================+;;; Cache shape and lazy resolution+;;; ===========================================================================++(def ^:private not-indexed :metabase-enterprise.checker.format.serdes/not-indexed)++(deftest schema-model-shape-test+ (testing "schema model starts with ::not-indexed for each schema, populates lazily"+ (with-temp-dir+ (fn [dir]+ (write-yaml! dir "databases" "db" "db.yaml" {:name "DB" :engine "h2"})+ (write-yaml! dir "databases" "db" "schemas" "s1" "tables" "t1" "t1.yaml"+ {:name "T1" :schema "s1"})+ (write-yaml! dir "databases" "db" "schemas" "s1" "tables" "t1" "fields" "f1.yaml"+ {:name "F1" :base_type "type/Integer"})+ (write-yaml! dir "databases" "db" "schemas" "s2" "tables" "t2" "t2.yaml"+ {:name "T2" :schema "s2"})+ (write-yaml! dir "databases" "db" "schemas" "s2" "tables" "t2" "fields" "f2.yaml"+ {:name "F2" :base_type "type/Text"})+ (let [source (serdes/make-database-source (str dir "/databases"))]+ ;; Initial: db-file set, all schemas ::not-indexed+ (is (=? {"DB" {:db-file string?+ :schemas {"s1" not-indexed+ "s2" not-indexed}}}+ @(.-schema-model source)))+ ;; After resolving a table in s1: s1 indexed with tables, s2 untouched+ (source/resolve-table source ["DB" "s1" "T1"])+ (is (=? {"DB" {:schemas {"s1" {"T1" {:table-file string?+ :fields not-indexed}}+ "s2" not-indexed}}}+ @(.-schema-model source)))+ ;; After resolving fields for T1: T1 fields indexed, s2 still untouched+ (source/fields-for-table source ["DB" "s1" "T1"])+ (is (=? {"DB" {:schemas {"s1" {"T1" {:table-file string?+ :fields {"F1" string?}}}+ "s2" not-indexed}}}+ @(.-schema-model source))))))))++(deftest case-insensitive-schema-resolution-test+ (testing "schemas with different casing on disk vs YAML resolve correctly"+ (with-temp-dir+ (fn [dir]+ ;; Dir name is lowercase, YAML schema is uppercase (H2 convention)+ (write-yaml! dir "databases" "db" "db.yaml" {:name "DB" :engine "h2"})+ (write-yaml! dir "databases" "db" "schemas" "public" "tables" "orders" "orders.yaml"+ {:name "ORDERS" :schema "PUBLIC"})+ (write-yaml! dir "databases" "db" "schemas" "public" "tables" "orders" "fields" "id.yaml"+ {:name "ID" :base_type "type/Integer"})+ (let [source (serdes/make-database-source (str dir "/databases"))]+ ;; Resolve by real names (uppercase) — schema re-keys from "public" to "PUBLIC"+ (is (some? (source/resolve-table source ["DB" "PUBLIC" "ORDERS"])))+ (is (some? (source/resolve-field source ["DB" "PUBLIC" "ORDERS" "ID"])))+ (is (=? {"DB" {:schemas {"PUBLIC" {"ORDERS" {:fields {"ID" string?}}}}}}+ @(.-schema-model source))))))))++;;; ===========================================================================+;;; Segments and measures — indexed from export dir, NOT schema dir+;;; ===========================================================================++(deftest export-dir-indexes-segments-and-measures-test+ (testing "build-file-index (export dir) finds segments and measures under databases/"+ (let [{:keys [index]} (serdes/build-file-index (fixtures-path))]+ (is (contains? (:segment index) "big-orders-segment")+ "segment from with-schema table should be indexed")+ (is (contains? (:segment index) "recent-orders-segment")+ "segment from schema-less table should be indexed")+ (is (contains? (:measure index) "total-revenue-measure")+ "measure should be indexed"))))++(deftest schema-dir-does-not-index-segments-test+ (testing "build-database-dir-index (schema dir) does NOT index segments or measures"+ (let [db-dir (str (fixtures-path) "/databases")+ {:keys [index]} (serdes/build-database-dir-index db-dir)]+ (is (nil? (:segment index)) "schema dir should not contain segments")+ (is (nil? (:measure index)) "schema dir should not contain measures"))))++(deftest segments-resolve-from-export-source-test+ (testing "segments can be resolved from the export source"+ (let [source (serdes/make-source (fixtures-path))]+ (let [seg (source/resolve-segment source "big-orders-segment")]+ (is (some? seg))+ (is (= "Big Orders" (:name seg))))+ (let [seg (source/resolve-segment source "recent-orders-segment")]+ (is (some? seg))+ (is (= "Recent Orders" (:name seg)))))))++(deftest measures-resolve-from-export-source-test+ (testing "measures can be resolved from the export source"+ (let [source (serdes/make-source (fixtures-path))+ m (source/resolve-measure source "total-revenue-measure")]+ (is (some? m))+ (is (= "Total Revenue" (:name m))))))++;;; ===========================================================================+;;; Temp file edge cases+;;; ===========================================================================++(deftest build-file-index-empty-dir-test+ (testing "build-file-index on empty directory returns empty index"+ (with-temp-dir+ (fn [dir]+ (let [{:keys [index]} (serdes/build-file-index dir)]+ (is (nil? (:database index)))+ (is (nil? (:card index))))))))++(def ^:private card-yaml-template+ "name: %s\nentity_id: %s\nserdes/meta:\n- id: %s\n model: Card\n")++(deftest build-file-index-detects-duplicates-test+ (testing "build-file-index reports duplicate entity_ids"+ (with-temp-dir+ (fn [dir]+ ;; Create two cards with the same entity_id in different directories+ (let [^java.io.File f1 (io/file dir "collections" "a" "cards" "dup.yaml")+ ^java.io.File f2 (io/file dir "collections" "b" "cards" "dup.yaml")]+ (.mkdirs (.getParentFile f1))+ (.mkdirs (.getParentFile f2))+ (spit f1 (format card-yaml-template "Card A" "dup-id" "dup-id"))+ (spit f2 (format card-yaml-template "Card B" "dup-id" "dup-id"))+ (let [{:keys [index]} (serdes/build-file-index dir)]+ (is (seq (:duplicates index)) "Should detect duplicate entity_ids")+ (is (= "dup-id" (:ref (first (:duplicates index)))))))))))++(deftest source-index-accessor-test+ (testing "source-index returns the assets index (no databases — those are in the schema model)"+ (let [source (serdes/make-source (fixtures-path))+ index (serdes/source-index source)]+ (is (map? index))+ (is (not (contains? index :database)) "databases are in the schema model, not the assets index")+ (is (contains? index :card)))))++(deftest all-card-ids-test+ (testing "all-card-ids returns all card entity-ids"+ (let [source (serdes/make-source (fixtures-path))]+ (is (>= (count (serdes/all-card-ids source)) 5))+ (is (contains? (set (serdes/all-card-ids source)) "simple-orders")))))diff --git a/enterprise/backend/test/metabase_enterprise/checker/semantic_test.clj b/enterprise/backend/test/metabase_enterprise/checker/semantic_test.cljnew file mode 100644index 00000000000..68064b532a1--- /dev/null+++ b/enterprise/backend/test/metabase_enterprise/checker/semantic_test.clj@@ -0,0 +1,1215 @@+(ns metabase-enterprise.checker.semantic-test+ "Tests for the semantic checker.++ Uses real YAML fixture files from test_resources/yaml_checks for most tests.+ Uses in-memory sources for edge cases like unresolved refs."+ (:require+ [clojure.java.io :as io]+ [clojure.test :refer [deftest is testing]]+ [metabase-enterprise.checker.format.serdes :as serdes-format]+ [metabase-enterprise.checker.provider :as provider]+ [metabase-enterprise.checker.semantic :as checker]+ [metabase-enterprise.checker.source :as source]+ [metabase-enterprise.checker.store :as store]+ [metabase-enterprise.checker.test-helpers :as helpers]+ [metabase.lib.metadata.protocols :as lib.metadata.protocols]))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; Test Fixture Path+;;; ===========================================================================++(def ^:private test-fixtures-dir+ "Path to test fixtures directory (real YAML files)."+ "test_resources/yaml_checks")++(defn- fixtures-path []+ (let [f (io/file test-fixtures-dir)]+ (if (.isAbsolute f)+ test-fixtures-dir+ (.getPath (io/file (System/getProperty "user.dir") test-fixtures-dir)))))++(defn- make-test-source []+ (serdes-format/make-source (fixtures-path)))++(defn- check-all [source]+ (checker/check-entities source source (serdes-format/source-index source)))++(defn- check-specific [source entity-ids]+ (checker/check-entities source source (serdes-format/source-index source) entity-ids))++;;; ===========================================================================+;;; Tests Using Real YAML Files+;;; ===========================================================================++(deftest source-index-test+ (testing "Source correctly indexes databases and cards; tables/fields are on-demand"+ (let [source (make-test-source)+ index (serdes-format/source-index source)]+ ;; Databases are in the schema model, not the assets index+ (is (= #{"Test Database" "SQLite DB"} (set (source/all-database-names source))))+ ;; Cards are in the assets index+ (is (= 5 (count (:card index))) "Should have 5 cards")+ ;; Tables and fields are resolved on demand+ (is (= 4 (count (source/all-table-paths source))) "Should have 4 tables via source")+ (is (= 11 (count (source/all-field-paths source))) "Should have 11 fields via source"))))++(deftest simple-mbql-query-test+ (testing "Simple MBQL query on orders table validates successfully"+ (let [source (make-test-source)+ results (check-all source)+ result (get results "simple-orders")]+ (is (some? result) "Card should be found")+ (is (= "Simple Orders" (:name result)) "Card name should match")+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))+ (is (empty? (:unresolved result)) "Should not have unresolved refs")+ (is (empty? (:bad-refs result)) "Should not have bad refs from lib")+ ;; Check that table refs were extracted+ (is (seq (get-in result [:refs :tables])) "Should have table refs"))))++(deftest native-query-test+ (testing "Native SQL query validates successfully"+ (let [source (make-test-source)+ results (check-all source)+ result (get results "native-orders")]+ (is (some? result) "Card should be found")+ (is (= "Native Orders" (:name result)) "Card name should match")+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))+ (is (empty? (:unresolved result)) "Should not have unresolved refs"))))++(deftest mbql-with-joins-test+ (testing "MBQL query with joins validates successfully"+ (let [source (make-test-source)+ results (check-all source)+ result (get results "orders-with-products")]+ (is (some? result) "Card should be found")+ (is (= "Orders With Products" (:name result)) "Card name should match")+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))+ (is (empty? (:unresolved result)) "Should not have unresolved refs")+ ;; Check that both tables are referenced+ (let [tables (get-in result [:refs :tables])]+ (is (some #(re-find #"orders" %) tables) "Should reference orders table")+ (is (some #(re-find #"products" %) tables) "Should reference products table")))))++(deftest all-cards-checked-test+ (testing "All cards in fixtures are checked"+ (let [source (make-test-source)+ results (check-all source)]+ (is (>= (count results) 5) "Should check at least 5 cards")+ (is (contains? results "simple-orders"))+ (is (contains? results "native-orders"))+ (is (contains? results "orders-with-products"))+ (is (contains? results "schemaless-query"))+ (is (contains? results "orders-with-fk")))))++;;; ===========================================================================+;;; Schema-less Database Tests+;;;+;;; Databases like SQLite don't have schemas, so tables are stored directly+;;; under databases/<db>/tables/ instead of databases/<db>/schemas/<schema>/tables/+;;; ===========================================================================++(deftest schemaless-database-index-test+ (testing "Schema-less databases are indexed correctly with nil schema"+ (let [source (make-test-source)]+ (is (= #{"Test Database" "SQLite DB"} (set (source/all-database-names source))))+ (is (= 4 (count (source/all-table-paths source))) "Should have 4 tables via source"))))++(deftest schemaless-query-test+ (testing "Query on schema-less database validates successfully"+ (let [source (make-test-source)+ results (check-all source)+ result (get results "schemaless-query")]+ (is (some? result) "Card should be found")+ (is (= "Schemaless Query" (:name result)) "Card name should match")+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))+ (is (empty? (:unresolved result)) "Should not have unresolved refs")+ (is (empty? (:bad-refs result)) "Should not have bad refs"))))++;;; ===========================================================================+;;; FK Resolution Tests+;;;+;;; Foreign key references in field metadata and result_metadata must be+;;; converted from path vectors to integer IDs.+;;; ===========================================================================++(deftest fk-in-field-metadata-test+ (testing "FK target in field metadata is resolved to integer ID"+ (let [source (make-test-source)+ index (serdes-format/source-index source)+ provider (provider/make-provider (store/make-store source source index))+ ;; Get the product_id field which has FK to products.id+ fields (lib.metadata.protocols/metadatas+ provider {:lib/type :metadata/column})+ fk-field (first (filter #(= "product_id" (:name %)) fields))]+ (is (some? fk-field) "Should find product_id field")+ (is (= :type/FK (:semantic-type fk-field)) "Should be FK type")+ (is (pos-int? (:fk-target-field-id fk-field))+ "FK target should be resolved to positive integer"))))++(deftest fk-in-result-metadata-test+ (testing "FK target in card result_metadata is resolved to integer ID"+ (let [source (make-test-source)+ results (check-all source)+ result (get results "orders-with-fk")]+ (is (some? result) "Card should be found")+ (is (= "Orders With FK" (:name result)) "Card name should match")+ (is (nil? (:error result)) (str "Should not have errors: " (:error result)))+ (is (empty? (:unresolved result)) "Should not have unresolved refs")+ (is (empty? (:bad-refs result)) "Should not have bad refs"))))++;;; ===========================================================================+;;; Tests Using In-Memory Source for Edge Cases+;;;+;;; ===========================================================================+;;; MBQL normalization tests+;;; ===========================================================================++(deftest template-tag-default-not-keywordized-test+ (testing "Template tag :default values are not incorrectly keywordized as MBQL operators"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["DB" "PUBLIC" "ORDERS" "CREATED_AT"]+ {:name "CREATED_AT" :base_type "type/DateTimeWithLocalTZ"+ :database_type "TIMESTAMP WITH TIME ZONE"}}+ :cards {"tMpLtAgDeFaUlT00card1"+ {:name "Field Filter Card"+ :entity_id "tMpLtAgDeFaUlT00card1"+ :dataset_query+ {:database "DB"+ :stages [{:native "SELECT * FROM ORDERS WHERE {{date_filter}}"+ :template-tags+ {"date_filter"+ {:type "dimension"+ :dimension ["field" {} ["DB" "PUBLIC" "ORDERS" "CREATED_AT"]]+ :default "past30days"+ :display-name "Date Filter"+ :name "date_filter"+ :widget-type "date/all-options"}}+ "lib/type" "mbql.stage/native"}]+ "lib/type" "mbql/query"}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["tMpLtAgDeFaUlT00card1"])+ result (get results "tMpLtAgDeFaUlT00card1")]+ (is (nil? (:error result))+ (str "Should not error: " (:error result))))))++;;; ===========================================================================+;;; Create a simple in-memory source to test unresolved references+;;; without needing temp files.+;;; ===========================================================================++(deftest unresolved-table-in-card-test+ (testing "Card referencing nonexistent table is detected"+ (let [entities {:databases {"Test Database" {:name "Test Database"+ :engine "h2"}}+ :tables {} ; No tables!+ :fields {}+ :cards {"test-card" {:name "Test Card"+ :entity_id "test-card"+ :dataset_query {:database "Test Database"+ :type "query"+ :query {:source-table ["Test Database" "public" "orders"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["test-card"])+ result (get results "test-card")]+ (is (some? result) "Card should be found")+ (is (= "Test Card" (:name result)) "Card name should match")+ (is (seq (:unresolved result)) "Should have unresolved refs")+ (is (some #(= :table (:type %)) (:unresolved result))+ "Should have unresolved table ref"))))++(deftest unresolved-database-in-card-test+ (testing "Card referencing nonexistent database is detected"+ (let [entities {:databases {} ; No databases!+ :tables {}+ :fields {}+ :cards {"test-card" {:name "Test Card"+ :entity_id "test-card"+ :dataset_query {:database "Nonexistent Database"+ :type "query"+ :query {:source-table ["Nonexistent Database" "public" "orders"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["test-card"])+ result (get results "test-card")]+ (is (some? result) "Card should be found")+ (is (= "Test Card" (:name result)) "Card name should match")+ ;; Either has an error about missing database, or unresolved refs+ (is (or (:error result)+ (some #(= :database (:type %)) (:unresolved result)))+ "Should have unresolved database or error"))))++(deftest check-specific-cards-test+ (testing "Can check specific cards by entity-id"+ (let [source (make-test-source)+ results (check-specific source ["simple-orders" "native-orders"])]+ (is (>= (count results) 2) "Should check at least 2 cards")+ (is (contains? results "simple-orders"))+ (is (contains? results "native-orders"))+ (is (not (contains? results "orders-with-products"))))))++;;; ===========================================================================+;;; Results Processing Tests+;;; ===========================================================================++(deftest result-status-test+ (testing "Result status is computed correctly"+ (is (= :ok (checker/result-status {})))+ (is (= :error (checker/result-status {:error "Something went wrong"})))+ (is (= :unresolved (checker/result-status {:unresolved [{:type :table}]})))+ (is (= :issues (checker/result-status {:bad-refs [{:type :field}]})))))++(deftest summarize-results-test+ (testing "Results are summarized correctly"+ (let [results {"card1" {}+ "card2" {:error "oops"}+ "card3" {:unresolved [{:type :table}]}+ "card4" {:bad-refs [{:type :field}]}}+ summary (checker/summarize-results results)]+ (is (= 4 (:total summary)))+ (is (= 1 (:ok summary)))+ (is (= 1 (:errors summary)))+ (is (= 1 (:unresolved summary)))+ (is (= 1 (:issues summary))))))++;;; ===========================================================================+;;; Format-error Tests (LLM-friendly error output)+;;; ===========================================================================++(deftest format-error-returns-nil-for-ok-test+ (testing "format-error returns nil for OK results"+ (is (nil? (checker/format-error ["card-1" {}])))+ (is (nil? (checker/format-error ["card-1" {:refs {:tables ["t"]}}])))))++(deftest format-error-includes-card-identity-test+ (testing "format-error includes card name and entity-id"+ (let [output (checker/format-error ["abc123" {:name "My Card" :error "boom"}])]+ (is (some? output))+ (is (re-find #"My Card" output))+ (is (re-find #"abc123" output)))))++(deftest format-error-includes-unresolved-refs-test+ (testing "format-error includes unresolved field/table refs"+ (let [output (checker/format-error+ ["eid" {:name "Bad Card"+ :unresolved [{:type :field :path ["db" "public" "t" "bad_col"]}+ {:type :table :path ["db" "public" "missing"]}]}])]+ (is (re-find #"unresolved field" output))+ (is (re-find #"bad_col" output))+ (is (re-find #"unresolved table" output))+ (is (re-find #"missing" output)))))++(deftest format-error-includes-bad-refs-test+ (testing "format-error includes bad-refs"+ (let [output (checker/format-error+ ["eid" {:name "Issue Card"+ :bad-refs [{:type :missing-column :name "gone"}]}])]+ (is (re-find #"bad ref" output))+ (is (re-find #"missing-column" output)))))++(deftest format-error-includes-error-message-test+ (testing "format-error includes the error message"+ (let [output (checker/format-error+ ["eid" {:name "Error Card" :error "Unknown database: foo"}])]+ (is (re-find #"error: Unknown database: foo" output)))))++(deftest format-result-non-card-entity-test+ (testing "format-result shows kind for non-card entities instead of Card ID"+ (let [output (checker/format-result+ ["eid" {:name "My Dashboard" :entity-id "eid" :kind :dashboard+ :unresolved [{:type :collection :entity-id "bad" :message "not found"}]}])]+ (is (re-find #"Kind: dashboard" output))+ (is (not (re-find #"Card ID" output))))))++(deftest format-error-non-card-entity-test+ (testing "format-error uses kind name for non-card entities"+ (let [output (checker/format-error+ ["eid" {:name "My Transform" :entity-id "eid" :kind :transform+ :error "Unknown database: foo"}])]+ (is (re-find #"^transform:" output))+ (is (not (re-find #"^card:" output))))))++;;; ===========================================================================+;;; Sentinel ID Tests — unresolved refs get IDs so queries can be constructed+;;; ===========================================================================++(deftest sentinel-id-for-unresolved-field-test+ (testing "Card with unresolved field gets sentinel ID — query builds, field flagged"+ (let [entities {:databases {"Test DB" {:name "Test DB" :engine "h2"}}+ :tables {["Test DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["Test DB" "PUBLIC" "ORDERS" "ID"] {:name "ID"+ :base_type "type/BigInteger"+ :database_type "BIGINT"+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}+ ;; Card references NONEXISTENT_FIELD which doesn't exist+ :cards {"bad-field" {:name "Bad Field Card"+ :entity_id "bad-field"+ :type "question"+ :dataset_query {:database "Test DB"+ :type "query"+ :query {:source-table ["Test DB" "PUBLIC" "ORDERS"]+ :breakout [[:field+ ["Test DB" "PUBLIC" "ORDERS" "NONEXISTENT_FIELD"]+ {:base-type :type/Text}]]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["bad-field"])+ result (get results "bad-field")]+ (is (some? result))+ ;; Should NOT have an :error (query should build successfully with sentinel)+ (is (nil? (:error result)) (str "Should not blow up: " (:error result)))+ ;; Should have unresolved refs tracking the bad field+ (is (seq (:unresolved result)) "Should track unresolved field")+ (is (some #(= :field (:type %)) (:unresolved result))+ "Should have unresolved field ref"))))++(deftest sentinel-id-for-unresolved-table-in-join-test+ (testing "Card with unresolved table in join gets sentinel ID — query builds"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {}+ ;; Card joins to a nonexistent table+ :cards {"bad-join" {:name "Bad Join Card"+ :entity_id "bad-join"+ :type "question"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "ORDERS"]+ :joins [{:source-table ["DB" "PUBLIC" "NONEXISTENT"]+ :alias "n"+ :condition [:= 1 1]}]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["bad-join"])+ result (get results "bad-join")]+ (is (some? result))+ ;; Should NOT have an :error (query should build with sentinel)+ (is (nil? (:error result)) (str "Should not blow up: " (:error result)))+ ;; Should track the unresolved table+ (is (seq (:unresolved result)) "Should track unresolved table")+ (is (some #(= :table (:type %)) (:unresolved result))+ "Should have unresolved table ref"))))++;;; ===========================================================================+;;; Native SQL validation via deps.analysis+;;;+;;; These tests verify that deps.analysis/check-entity is properly invoked+;;; for native SQL cards (requires sql-tools.init to be loaded).+;;; ===========================================================================++(deftest native-card-bad-sql-produces-native-errors-test+ (testing "Card with bad SQL produces native-errors, not validation-exception-error"+ (let [entities {:databases {"Test DB" {:name "Test DB" :engine "h2"}}+ :tables {["Test DB" "PUBLIC" "ORDERS"]+ {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["Test DB" "PUBLIC" "ORDERS" "ID"]+ {:name "ID" :base_type "type/BigInteger"+ :database_type "BIGINT"+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}+ :cards {"bad-sql" {:name "Bad SQL"+ :entity_id "bad-sql"+ :type "question"+ :dataset_query {:database "Test DB"+ :type "native"+ :native {:query "SELLECT * FROM ORDERS"}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["bad-sql"])+ result (get results "bad-sql")]+ (is (some? result))+ (is (or (seq (:native-errors result))+ (:error result))+ "Bad SQL should produce native-errors or error, not validation-exception-error")+ (is (empty? (filter #(= :validation-exception-error (:type %))+ (:bad-refs result)))+ "Should not have validation-exception-error in bad-refs"))))++(deftest native-card-valid-sql-no-errors-test+ (testing "Card with valid native SQL produces no errors"+ (let [entities {:databases {"Test DB" {:name "Test DB" :engine "h2"}}+ :tables {["Test DB" "PUBLIC" "ORDERS"]+ {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["Test DB" "PUBLIC" "ORDERS" "ID"]+ {:name "ID" :base_type "type/BigInteger"+ :database_type "BIGINT"+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}+ ["Test DB" "PUBLIC" "ORDERS" "TOTAL"]+ {:name "TOTAL" :base_type "type/Float"+ :database_type "DOUBLE PRECISION"+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}+ :cards {"good-sql" {:name "Good SQL"+ :entity_id "good-sql"+ :type "question"+ :dataset_query {:database "Test DB"+ :type "native"+ :native {:query "SELECT ID, TOTAL FROM ORDERS"}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["good-sql"])+ result (get results "good-sql")]+ (is (some? result))+ (is (nil? (:error result)) (str "Should not error: " (:error result)))+ (is (empty? (:native-errors result)) "Valid SQL should have no native errors")+ (is (empty? (:bad-refs result)) "Valid SQL should have no bad refs"))))++;;; ===========================================================================+;;; Computed result-metadata tests+;;;+;;; When YAML has no result_metadata, the checker computes it from the query+;;; so deps.analysis can verify card subquery column references.+;;; ===========================================================================++(deftest mbql-card-computes-result-metadata-test+ (testing "MBQL card without result_metadata computes columns from query"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["DB" "PUBLIC" "ORDERS" "ID"]+ {:name "ID" :base_type "type/BigInteger" :database_type "BIGINT"+ :table_id ["DB" "PUBLIC" "ORDERS"]}+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"+ :table_id ["DB" "PUBLIC" "ORDERS"]}}+ ;; Card with MBQL query but no result_metadata+ :cards {"mbql-card" {:name "MBQL Card"+ :entity_id "mbql-card"+ :type "question"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "ORDERS"]}}}+ ;; Native card referencing the MBQL card's columns+ "native-card" {:name "Native Ref"+ :entity_id "native-card"+ :type "question"+ :dataset_query {:database "DB"+ :type "native"+ :native {:query "SELECT ID FROM {{#1-mbql}}"+ :template-tags {"#1-mbql" {:type :card+ :name "#1-mbql"+ :card-id "mbql-card"}}}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["native-card"])+ result (get results "native-card")]+ (is (some? result))+ ;; ID exists in ORDERS and in the MBQL card's computed result-metadata,+ ;; so it should NOT be flagged as missing+ (is (not (some #(and (= :missing-column (:type %))+ (= "ID" (:name %)))+ (:native-errors result)))+ "ID should not be flagged — it exists in the MBQL card's computed columns"))))++(deftest mbql-card-named-aggregation-in-result-metadata-test+ (testing "Named aggregation columns appear in computed result-metadata"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["DB" "PUBLIC" "ORDERS" "TOTAL"]+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"+ :table_id ["DB" "PUBLIC" "ORDERS"]}}+ ;; Card with named aggregation but no result_metadata+ :cards {"agg-card" {:name "Agg Card"+ :entity_id "agg-card"+ :type "question"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "ORDERS"]+ :aggregation [[:sum {:name "total_revenue"+ :display-name "Total Revenue"}+ [:field+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]+ {:base-type :type/Float}]]]}}}+ ;; Native card referencing the named aggregation column+ "ref-card" {:name "Ref Card"+ :entity_id "ref-card"+ :type "question"+ :dataset_query {:database "DB"+ :type "native"+ :native {:query "SELECT total_revenue FROM {{#1-agg}}"+ :template-tags {"#1-agg" {:type :card+ :name "#1-agg"+ :card-id "agg-card"}}}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["ref-card"])+ result (get results "ref-card")]+ (is (some? result))+ (is (not (some #(and (= :missing-column (:type %))+ (= "total_revenue" (:name %)))+ (:native-errors result)))+ "total_revenue should not be flagged — it's a named aggregation in the source card"))))++(deftest native-card-without-result-metadata-no-false-positives-test+ (testing "Native card without result_metadata doesn't produce false positive errors"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["DB" "PUBLIC" "ORDERS" "ID"]+ {:name "ID" :base_type "type/BigInteger" :database_type "BIGINT"+ :table_id ["DB" "PUBLIC" "ORDERS"]}+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"+ :table_id ["DB" "PUBLIC" "ORDERS"]}}+ :cards {"sql-card" {:name "SQL Card"+ :entity_id "sql-card"+ :type "question"+ :dataset_query {:database "DB"+ :type "native"+ :native {:query "SELECT ID, TOTAL FROM ORDERS"}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["sql-card"])+ result (get results "sql-card")]+ (is (some? result))+ (is (nil? (:error result)))+ (is (empty? (:bad-refs result)) "Valid native SQL should have no bad refs"))))++;;; ===========================================================================+;;; Measure and Segment reference tests+;;; ===========================================================================++(deftest measure-ref-resolved-test+ (testing "Card referencing a known measure resolves without errors"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Uses Measure"+ :entity_id "card-1"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]+ :aggregation [["measure" "mSrAvgProdPrice00008x"]]}}}}+ :measures {"mSrAvgProdPrice00008x" {:name "Avg Product Price"+ :entity_id "mSrAvgProdPrice00008x"}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (nil? (:error result)) (str "Should not error: " (:error result)))+ (is (empty? (filter #(= :measure (:type %)) (or (:unresolved result) [])))+ "Should not have unresolved measure refs"))))++(deftest measure-ref-missing-test+ (testing "Card referencing unknown measure is flagged"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Bad Measure Ref"+ :entity_id "card-1"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]+ :aggregation [["measure" "xXxNoExIsT0MeAsUrExx1"]]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (some #(= :measure (:type %)) (or (:unresolved result) []))+ "Should have unresolved measure ref"))))++(deftest segment-ref-resolved-test+ (testing "Card referencing a known segment resolves without errors"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Uses Segment"+ :entity_id "card-1"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]+ :filter ["segment" "aB3kLmN9pQrStUvWxYz1a"]}}}}+ :segments {"aB3kLmN9pQrStUvWxYz1a" {:name "Known Segment"+ :entity_id "aB3kLmN9pQrStUvWxYz1a"}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (nil? (:error result)) (str "Should not error: " (:error result)))+ (is (empty? (filter #(= :segment (:type %)) (or (:unresolved result) [])))+ "Should not have unresolved segment refs"))))++(deftest segment-ref-missing-test+ (testing "Card referencing unknown segment is flagged"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Bad Segment Ref"+ :entity_id "card-1"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]+ :filter ["segment" "xXxNoExIsT0SeGmEnTx12"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (some #(= :segment (:type %)) (or (:unresolved result) []))+ "Should have unresolved segment ref"))))++;;; ===========================================================================+;;; Transitive entity ref tests+;;; ===========================================================================++(deftest transitive-snippet-refs-test+ (testing "Card sourcing from another card that uses a snippet surfaces that snippet transitively"+ ;; Entity-ids must be 21-char NanoIDs to pass portable-id? checks+ (let [inner-eid "tRaNsInNeRsNiPpEt0001"+ outer-eid "tRaNsOuTeRsNiPpEt0001"+ snip-eid "tRaNsSnIpPeTiD0000001"+ entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :fields {["DB" "PUBLIC" "ORDERS" "ID"] {:name "ID" :base_type "type/BigInteger"+ :database_type "BIGINT"}}+ :cards {inner-eid {:name "Inner Card"+ :entity_id inner-eid+ :dataset_query {:database "DB"+ :stages [{:native "SELECT * FROM ORDERS WHERE {{snippet: My Filter}}"+ :template-tags {"snippet: My Filter"+ {:type "snippet"+ :snippet-name "My Filter"+ :snippet-id snip-eid}}+ "lib/type" "mbql.stage/native"}]+ "lib/type" "mbql/query"}+ :result_metadata [{:name "ID" :base_type "type/BigInteger"}]}+ outer-eid {:name "Outer Card"+ :entity_id outer-eid+ :dataset_query {:database "DB"+ :stages [{:source-card inner-eid+ "lib/type" "mbql.stage/mbql"}]+ "lib/type" "mbql/query"}}}+ :snippets {snip-eid {:name "My Filter" :content "1=1"}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index [outer-eid inner-eid])+ outer (get results outer-eid)]+ (is (= ["Inner Card"] (get-in outer [:refs :source-cards]))+ "Should list inner card as source")+ (is (= ["My Filter"] (get-in outer [:refs :snippets]))+ "Should transitively surface the inner card's snippet"))))++(deftest transitive-measure-refs-test+ (testing "Card sourcing from another card that uses measures/segments surfaces them transitively"+ (let [inner-eid "tRaNsInNeRmEaSuRe0001"+ outer-eid "tRaNsOuTeRmEaSuRe0001"+ msr-eid "tRaNsMeAsUrEiD0000001"+ seg-eid "tRaNsSeGmEnTiD0000001"+ entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {inner-eid {:name "Inner Metrics"+ :entity_id inner-eid+ :dataset_query {:database "DB"+ :stages [{:source-table ["DB" "PUBLIC" "T"]+ :aggregation [["measure" msr-eid]]+ :filters [["segment" seg-eid]]+ "lib/type" "mbql.stage/mbql"}]+ "lib/type" "mbql/query"}+ :result_metadata [{:name "count" :base_type "type/Integer"}]}+ outer-eid {:name "Outer Query"+ :entity_id outer-eid+ :dataset_query {:database "DB"+ :stages [{:source-card inner-eid+ "lib/type" "mbql.stage/mbql"}]+ "lib/type" "mbql/query"}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ index (assoc index+ :measure {msr-eid :memory}+ :segment {seg-eid :memory})+ results (checker/check-entities schema assets index [outer-eid inner-eid])+ outer (get results outer-eid)]+ (is (= ["Inner Metrics"] (get-in outer [:refs :source-cards])))+ (is (= [msr-eid] (get-in outer [:refs :measures]))+ "Should transitively surface the inner card's measure")+ (is (= [seg-eid] (get-in outer [:refs :segments]))+ "Should transitively surface the inner card's segment"))))++;;; ===========================================================================+;;; Collection reference tests+;;; ===========================================================================++(deftest valid-collection-id-test+ (testing "Card with valid collection_id passes"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Good Card"+ :entity_id "card-1"+ :collection_id "coll-1"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ ;; Index includes the collection+ index (assoc index :collection {"coll-1" :memory})+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (nil? (:error result)))+ (is (empty? (:unresolved result)) "Valid collection_id should not produce unresolved refs"))))++(deftest missing-collection-id-test+ (testing "Card with collection_id pointing to nonexistent entity is flagged"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Bad Collection Card"+ :entity_id "card-1"+ :collection_id "nonexistent"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (seq (:unresolved result)) "Missing collection should be flagged")+ (is (some #(= :collection (:type %)) (:unresolved result))+ "Should have unresolved collection ref"))))++(deftest collection-id-points-to-card-test+ (testing "Card with collection_id pointing to another card (not a collection) is flagged"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Card With Bad Collection"+ :entity_id "card-1"+ :collection_id "card-2"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}+ "card-2" {:name "Other Card"+ :entity_id "card-2"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (seq (:unresolved result)) "collection_id pointing to a card should be flagged")+ (is (some #(and (= :collection (:type %))+ (re-find #"card" (or (:message %) "")))+ (:unresolved result))+ "Error should mention it points to a card"))))++(deftest nil-collection-id-test+ (testing "Card with nil collection_id is fine"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Root Card"+ :entity_id "card-1"+ :collection_id nil+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (nil? (:error result)))+ (is (empty? (:unresolved result)) "nil collection_id should be fine"))))++;;; ===========================================================================+;;; Dashboard ID reference tests+;;; ===========================================================================++(deftest card-with-missing-dashboard-id-test+ (testing "Card with dashboard_id pointing to nonexistent dashboard is flagged"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Orphaned Card"+ :entity_id "card-1"+ :dashboard_id "nonexistent-dash"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (seq (:unresolved result)) "Missing dashboard_id should be flagged")+ (is (some #(= :dashboard (:type %)) (:unresolved result))+ "Should have unresolved dashboard ref"))))++(deftest card-with-valid-dashboard-id-test+ (testing "Card with dashboard_id pointing to a known dashboard passes"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Dashboard Card"+ :entity_id "card-1"+ :dashboard_id "dash-1"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ index (assoc index :dashboard {"dash-1" :memory})+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (nil? (:error result)))+ (is (empty? (:unresolved result)) "Valid dashboard_id should not produce unresolved refs"))))++(deftest card-with-null-dashboard-id-test+ (testing "Card with nil dashboard_id is fine"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Standalone Card"+ :entity_id "card-1"+ :dashboard_id nil+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (nil? (:error result)))+ (is (empty? (:unresolved result)) "nil dashboard_id should be fine"))))++(deftest card-with-dashboard-id-pointing-to-card-test+ (testing "Card with dashboard_id pointing to another card (not a dashboard) is flagged"+ (let [entities {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "T"] {:name "T" :schema "PUBLIC"}}+ :fields {}+ :cards {"card-1" {:name "Bad Dash Ref"+ :entity_id "card-1"+ :dashboard_id "card-2"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}+ "card-2" {:name "Other Card"+ :entity_id "card-2"+ :dataset_query {:database "DB"+ :type "query"+ :query {:source-table ["DB" "PUBLIC" "T"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["card-1"])+ result (get results "card-1")]+ (is (seq (:unresolved result)) "dashboard_id pointing to a card should be flagged")+ (is (some #(and (= :dashboard (:type %))+ (re-find #"card" (or (:message %) "")))+ (:unresolved result))+ "Error should mention it points to a card"))))++(deftest dashboard-bad-collection-id-test+ (testing "Dashboard with invalid collection_id is flagged"+ (let [entities {:databases {} :tables {} :fields {} :cards {}+ :dashboards {"dash-1" {:name "My Dashboard"+ :entity_id "dash-1"+ :collection_id "nonexistent"}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index)]+ (is (contains? results "dash-1") "Dashboard should be in results")+ (let [result (get results "dash-1")]+ (is (seq (:unresolved result)) "Dashboard with bad collection_id should be flagged")+ (is (some #(= :collection (:type %)) (:unresolved result)))))))++(deftest dashboard-valid-collection-id-not-flagged-test+ (testing "Dashboard with valid collection_id passes"+ (let [entities {:databases {} :tables {} :fields {} :cards {}+ :dashboards {"dash-1" {:name "Good Dashboard"+ :entity_id "dash-1"+ :collection_id "coll-1"}}+ :collections {"coll-1" {:name "My Collection"+ :entity_id "coll-1"}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index)]+ (is (= :ok (checker/result-status (get results "dash-1")))+ "Dashboard with valid collection_id should be OK"))))++;;; ===========================================================================+;;; Dashboard semantic validation tests+;;; ===========================================================================++(defn- with-temp-dashboard+ "Run checks with a dashboard entity provided via in-memory source.+ `dashboard-data` is a map representing the dashboard.+ `extra-index` is merged into the index (e.g. to add cards).+ Calls `(f results)` with the check results."+ [dashboard-data extra-index f]+ (let [entity-id (:entity_id dashboard-data)+ entities {:databases {}+ :tables {}+ :fields {}+ :cards {}+ :dashboards {entity-id dashboard-data}}+ [schema assets base-index] (helpers/make-sources-and-index entities)+ index (merge base-index+ {:collection {}+ :card {}}+ extra-index)+ results (checker/check-entities schema assets index)]+ (f results)))++(deftest dashboard-card-ref-valid-test+ (testing "Dashboard with valid card_id refs passes"+ (with-temp-dashboard+ {:name "Good Dashboard"+ :entity_id "dash-1"+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]+ :dashcards [{:entity_id "dc-1"+ :card_id "card-1"+ :dashboard_tab_id ["dash-1" "tab-1"]+ :row 0 :col 0 :size_x 12 :size_y 6}]}+ {:card {"card-1" :memory}}+ (fn [results]+ (is (= :ok (checker/result-status (get results "dash-1")))+ "Dashboard with valid refs should be OK")))))++(deftest dashboard-card-ref-missing-test+ (testing "Dashboard with card_id pointing to unknown card is flagged"+ (with-temp-dashboard+ {:name "Bad Card Ref"+ :entity_id "dash-1"+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]+ :dashcards [{:entity_id "dc-1"+ :card_id "nonexistent-card"+ :dashboard_tab_id ["dash-1" "tab-1"]+ :row 0 :col 0 :size_x 12 :size_y 6}]}+ {}+ (fn [results]+ (is (contains? results "dash-1"))+ (let [result (get results "dash-1")]+ (is (some #(= :dashcard-card-ref (:type %)) (:unresolved result))+ "Should flag missing card ref"))))))++(deftest dashboard-virtual-card-null-card-id-test+ (testing "Dashboard with null card_id (heading/text) is fine"+ (with-temp-dashboard+ {:name "Headings Dashboard"+ :entity_id "dash-1"+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]+ :dashcards [{:entity_id "dc-1"+ :card_id nil+ :dashboard_tab_id ["dash-1" "tab-1"]+ :row 0 :col 0 :size_x 24 :size_y 1}]}+ {}+ (fn [results]+ (is (= :ok (checker/result-status (get results "dash-1")))+ "Virtual card with null card_id should be fine")))))++(deftest dashboard-tab-ref-invalid-test+ (testing "Dashboard with dashcard referencing nonexistent tab is flagged"+ (with-temp-dashboard+ {:name "Bad Tab Ref"+ :entity_id "dash-1"+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]+ :dashcards [{:entity_id "dc-1"+ :card_id nil+ :dashboard_tab_id ["dash-1" "wrong-tab"]+ :row 0 :col 0 :size_x 12 :size_y 6}]}+ {}+ (fn [results]+ (is (contains? results "dash-1"))+ (let [result (get results "dash-1")]+ (is (some #(= :dashcard-tab-ref (:type %)) (:unresolved result))+ "Should flag bad tab ref"))))))++(deftest dashboard-grid-overflow-test+ (testing "Dashboard with dashcard extending beyond grid width is flagged"+ (with-temp-dashboard+ {:name "Grid Overflow"+ :entity_id "dash-1"+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]+ :dashcards [{:entity_id "dc-1"+ :card_id nil+ :dashboard_tab_id ["dash-1" "tab-1"]+ :row 0 :col 20 :size_x 12 :size_y 6}]}+ {}+ (fn [results]+ (is (contains? results "dash-1"))+ (let [result (get results "dash-1")]+ (is (some #(= :dashcard-grid (:type %)) (:unresolved result))+ "Should flag grid overflow"))))))++(deftest dashboard-grid-negative-col-test+ (testing "Dashboard with negative col is flagged"+ (with-temp-dashboard+ {:name "Negative Col"+ :entity_id "dash-1"+ :tabs []+ :dashcards [{:entity_id "dc-1"+ :card_id nil+ :row 0 :col -1 :size_x 12 :size_y 6}]}+ {}+ (fn [results]+ (is (contains? results "dash-1"))+ (let [result (get results "dash-1")]+ (is (some #(= :dashcard-grid (:type %)) (:unresolved result))+ "Should flag negative col"))))))++(deftest dashboard-valid-full-width-test+ (testing "Dashboard with full-width card (col=0, size_x=24) passes"+ (with-temp-dashboard+ {:name "Full Width"+ :entity_id "dash-1"+ :tabs [{:entity_id "tab-1" :name "Tab 1" :position 0}]+ :dashcards [{:entity_id "dc-1"+ :card_id nil+ :dashboard_tab_id ["dash-1" "tab-1"]+ :row 0 :col 0 :size_x 24 :size_y 1}]}+ {}+ (fn [results]+ (is (= :ok (checker/result-status (get results "dash-1")))+ "Full-width card should be valid")))))++;;; ===========================================================================+;;; Transform query validation tests+;;; ===========================================================================++(defn- with-temp-transform+ "Run checks with a transform entity provided via in-memory source.+ `transform-data` is a map representing the transform.+ `extra-index` is merged into the index (e.g. to add databases).+ Calls `(f results)` with the check results."+ [transform-data extra-index f]+ (let [entity-id (:entity_id transform-data)+ entities {:databases (get extra-index :_databases {})+ :tables (get extra-index :_tables {})+ :fields (get extra-index :_fields {})+ :cards {}+ :transforms {entity-id transform-data}}+ [schema assets base-index] (helpers/make-sources-and-index entities)+ index (merge base-index+ {:collection {}+ :card {}}+ (dissoc extra-index :_databases :_tables :_fields))+ results (checker/check-entities schema assets index)]+ (f results)))++(deftest transform-bad-native-sql-test+ (testing "Transform with invalid native SQL is flagged"+ (with-temp-transform+ {:name "Bad SQL Transform"+ :entity_id "tx-1"+ :source_database_id "Test DB"+ :source {:type "query"+ :query {"lib/type" "mbql/query"+ :database "Test DB"+ :stages [{"lib/type" "mbql.stage/native"+ :native "SELET * FROM ORDERS"}]}}+ :target {:type "table" :schema "TRANSFORMS" :name "bad_transform"}+ :serdes/meta [{:id "tx-1" :model "Transform"}]}+ {:_databases {"Test DB" {:name "Test DB" :engine "h2"}}+ :_tables {["Test DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :_fields {}+ :database {"Test DB" :memory}}+ (fn [results]+ (is (contains? results "tx-1") "Transform with bad SQL should be in results")+ (let [result (get results "tx-1")]+ (is (or (seq (:native-errors result))+ (:error result)+ (seq (:unresolved result)))+ "Should have native errors, error, or unresolved refs"))))))++(deftest transform-valid-native-sql-test+ (testing "Transform with valid native SQL passes"+ (with-temp-transform+ {:name "Good SQL Transform"+ :entity_id "tx-1"+ :source_database_id "Test DB"+ :source {:type "query"+ :query {"lib/type" "mbql/query"+ :database "Test DB"+ :stages [{"lib/type" "mbql.stage/native"+ :native "SELECT ID, TOTAL FROM ORDERS"}]}}+ :target {:type "table" :schema "TRANSFORMS" :name "good_transform"}+ :serdes/meta [{:id "tx-1" :model "Transform"}]}+ {:_databases {"Test DB" {:name "Test DB" :engine "h2"}}+ :_tables {["Test DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}}+ :_fields {["Test DB" "PUBLIC" "ORDERS" "ID"]+ {:name "ID" :base_type "type/BigInteger" :database_type "BIGINT"+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}+ ["Test DB" "PUBLIC" "ORDERS" "TOTAL"]+ {:name "TOTAL" :base_type "type/Float" :database_type "DOUBLE PRECISION"+ :table_id ["Test DB" "PUBLIC" "ORDERS"]}}+ :database {"Test DB" :memory}}+ (fn [results]+ (is (= :ok (checker/result-status (get results "tx-1")))+ "Transform with valid SQL should be OK")))))++(deftest transform-mbql-unresolved-table-test+ (testing "Transform with MBQL referencing nonexistent table is flagged"+ (with-temp-transform+ {:name "Bad MBQL Transform"+ :entity_id "tx-1"+ :source_database_id "Test DB"+ :source {:type "query"+ :query {"lib/type" "mbql/query"+ :database "Test DB"+ :stages [{"lib/type" "mbql.stage/mbql"+ :source-table ["Test DB" "PUBLIC" "NONEXISTENT"]}]}}+ :target {:type "table" :schema "TRANSFORMS" :name "bad_mbql_transform"}+ :serdes/meta [{:id "tx-1" :model "Transform"}]}+ {:_databases {"Test DB" {:name "Test DB" :engine "h2"}}+ :_tables {}+ :_fields {}+ :database {"Test DB" :memory}}+ (fn [results]+ (is (contains? results "tx-1") "Transform with unresolved table should be in results")+ (let [result (get results "tx-1")]+ (is (seq (:unresolved result)) "Should have unresolved refs"))))))++(deftest transform-missing-database-test+ (testing "Transform referencing nonexistent database is flagged"+ (with-temp-transform+ {:name "Bad DB Transform"+ :entity_id "tx-1"+ :source_database_id "Nonexistent DB"+ :source {:type "query"+ :query {"lib/type" "mbql/query"+ :database "Nonexistent DB"+ :stages [{"lib/type" "mbql.stage/native"+ :native "SELECT 1"}]}}+ :target {:type "table" :schema "TRANSFORMS" :name "bad_db"}+ :serdes/meta [{:id "tx-1" :model "Transform"}]}+ {}+ (fn [results]+ (is (contains? results "tx-1") "Transform with missing DB should be in results")+ (let [result (get results "tx-1")]+ (is (some #(= :database (:type %)) (:unresolved result))+ "Should flag missing database"))))))++;;; ===========================================================================+;;; Multi-database tests+;;;+;;; The provider's `(database)` method must return the correct database for the+;;; entity being checked, not just the first database in the store.+;;; ===========================================================================++(deftest multi-database-cards-test+ (testing "Cards from different databases each get the correct database metadata"+ (let [entities {:databases {"H2 DB" {:name "H2 DB" :engine "h2"}+ "Postgres DB" {:name "Postgres DB" :engine "postgres"}}+ :tables {["H2 DB" "PUBLIC" "ORDERS"]+ {:name "ORDERS" :schema "PUBLIC"}+ ["Postgres DB" "public" "customers"]+ {:name "customers" :schema "public"}}+ :fields {["H2 DB" "PUBLIC" "ORDERS" "ID"]+ {:name "ID" :base_type "type/BigInteger"+ :database_type "BIGINT"+ :table_id ["H2 DB" "PUBLIC" "ORDERS"]}+ ["Postgres DB" "public" "customers" "id"]+ {:name "id" :base_type "type/BigInteger"+ :database_type "bigint"+ :table_id ["Postgres DB" "public" "customers"]}}+ :cards {"h2-card" {:name "H2 Card"+ :entity_id "h2-card"+ :type "question"+ :dataset_query {:database "H2 DB"+ :type "query"+ :query {:source-table ["H2 DB" "PUBLIC" "ORDERS"]}}}+ "postgres-card" {:name "Postgres Card"+ :entity_id "postgres-card"+ :type "question"+ :dataset_query {:database "Postgres DB"+ :type "query"+ :query {:source-table ["Postgres DB" "public" "customers"]}}}}}+ [schema assets index] (helpers/make-sources-and-index entities)+ results (checker/check-entities schema assets index ["h2-card" "postgres-card"])+ h2-result (get results "h2-card")+ pg-result (get results "postgres-card")]+ (is (some? h2-result) "H2 card should be checked")+ (is (some? pg-result) "Postgres card should be checked")+ (is (nil? (:error h2-result)) (str "H2 card should not error: " (:error h2-result)))+ (is (nil? (:error pg-result)) (str "Postgres card should not error: " (:error pg-result)))+ (is (empty? (:bad-refs h2-result)) "H2 card should have no bad refs")+ (is (empty? (:bad-refs pg-result)) "Postgres card should have no bad refs"))))++;;; ===========================================================================+;;; REPL Helpers+;;; ===========================================================================++(comment+ ;; Run all tests+ (clojure.test/run-tests 'metabase-enterprise.checker.semantic-test)++ ;; Check fixtures path+ (fixtures-path)++ ;; Manually check all cards in fixtures+ (def source (make-test-source))+ (check-all source))diff --git a/enterprise/backend/test/metabase_enterprise/checker/store_test.clj b/enterprise/backend/test/metabase_enterprise/checker/store_test.cljnew file mode 100644index 00000000000..708cdef8ab2--- /dev/null+++ b/enterprise/backend/test/metabase_enterprise/checker/store_test.clj@@ -0,0 +1,211 @@+(ns metabase-enterprise.checker.store-test+ "Tests for the checker store — ID registry, index queries, entity loading/caching."+ (:require+ [clojure.test :refer [deftest is testing]]+ [metabase-enterprise.checker.store :as store]+ [metabase-enterprise.checker.test-helpers :as helpers]))++(set! *warn-on-reflection* true)++;;; ===========================================================================+;;; Test data+;;; ===========================================================================++(def ^:private test-entities+ {:databases {"DB" {:name "DB" :engine "h2"}}+ :tables {["DB" "PUBLIC" "ORDERS"] {:name "ORDERS" :schema "PUBLIC"}+ ["DB" "PUBLIC" "PRODUCTS"] {:name "PRODUCTS" :schema "PUBLIC"}}+ :fields {["DB" "PUBLIC" "ORDERS" "ID"] {:name "ID" :base_type "type/BigInteger"}+ ["DB" "PUBLIC" "ORDERS" "TOTAL"] {:name "TOTAL" :base_type "type/Float"}+ ["DB" "PUBLIC" "PRODUCTS" "ID"] {:name "ID" :base_type "type/BigInteger"}+ ["DB" "PUBLIC" "PRODUCTS" "CATEGORY"] {:name "CATEGORY" :base_type "type/Text"}}+ :cards {"card-1" {:name "Card 1" :entity_id "card-1"}}})++(defn- make-test-store []+ (let [[schema assets index] (helpers/make-sources-and-index test-entities)]+ (store/make-store schema assets index)))++;;; ===========================================================================+;;; Index query tests+;;; ===========================================================================++(deftest all-refs-test+ (let [store (make-test-store)]+ (testing "all-refs returns refs for each kind"+ (is (= ["DB"] (store/all-database-names store)))+ (is (= #{"card-1"} (set (store/all-card-ids store))))+ (is (= 2 (count (store/all-table-paths store))))+ (is (= 4 (count (store/all-field-paths store)))))))++(deftest exists?-test+ (let [store (make-test-store)]+ (testing "known refs return true"+ (is (store/exists? store :database "DB"))+ (is (store/exists? store :table ["DB" "PUBLIC" "ORDERS"]))+ (is (store/exists? store :card "card-1")))+ (testing "unknown refs return false"+ (is (not (store/exists? store :database "Nonexistent")))+ (is (not (store/exists? store :table ["DB" "PUBLIC" "MISSING"])))+ (is (not (store/exists? store :card "no-such-card"))))))++(deftest index-kind-of-test+ (let [store (make-test-store)]+ (testing "returns the kind for known entity-ids"+ (is (= :card (store/index-kind-of store "card-1"))))+ (testing "returns nil for unknown entity-ids"+ (is (nil? (store/index-kind-of store "unknown"))))))++(deftest index-kind-of-multi-kind-test+ (testing "index-kind-of distinguishes between kinds"+ (let [empty-entities {:databases {} :tables {} :fields {} :cards {}}+ [schema assets _] (helpers/make-sources-and-index empty-entities)+ index {:card {"eid-1" :memory}+ :dashboard {"eid-2" :memory}+ :collection {"eid-3" :memory}+ :document {"eid-4" :memory}}+ store (store/make-store schema assets index)]+ (is (= :card (store/index-kind-of store "eid-1")))+ (is (= :dashboard (store/index-kind-of store "eid-2")))+ (is (= :collection (store/index-kind-of store "eid-3")))+ (is (= :document (store/index-kind-of store "eid-4"))))))++(deftest index-file-test+ (let [store (make-test-store)]+ (testing "returns the file path for known refs"+ (is (= :memory (store/index-file store :database "DB")))+ (is (= :memory (store/index-file store :card "card-1"))))+ (testing "returns nil for unknown refs"+ (is (nil? (store/index-file store :database "Nonexistent"))))))++(deftest fields-for-table-test+ (let [store (make-test-store)]+ (testing "returns field paths for a known table"+ (is (= #{["DB" "PUBLIC" "ORDERS" "ID"]+ ["DB" "PUBLIC" "ORDERS" "TOTAL"]}+ (store/fields-for-table store ["DB" "PUBLIC" "ORDERS"]))))+ (testing "returns field paths for another table"+ (is (= #{["DB" "PUBLIC" "PRODUCTS" "ID"]+ ["DB" "PUBLIC" "PRODUCTS" "CATEGORY"]}+ (store/fields-for-table store ["DB" "PUBLIC" "PRODUCTS"]))))+ (testing "returns nil for unknown table"+ (is (nil? (store/fields-for-table store ["DB" "PUBLIC" "MISSING"]))))))++;;; ===========================================================================+;;; ID registry tests+;;; ===========================================================================++(deftest get-or-assign-returns-stable-ids-test+ (let [store (make-test-store)]+ (testing "first call assigns an ID"+ (let [id (store/get-or-assign! store :database "DB")]+ (is (pos-int? id))))+ (testing "second call returns the same ID"+ (let [id1 (store/get-or-assign! store :database "DB")+ id2 (store/get-or-assign! store :database "DB")]+ (is (= id1 id2))))))++(deftest different-refs-get-different-ids-test+ (let [store (make-test-store)]+ (testing "different refs get different IDs"+ (let [id1 (store/get-or-assign! store :table ["DB" "PUBLIC" "ORDERS"])+ id2 (store/get-or-assign! store :table ["DB" "PUBLIC" "PRODUCTS"])]+ (is (not= id1 id2))))))++(deftest id-ref-roundtrip-test+ (let [store (make-test-store)]+ (testing "id->ref and ref->id are inverses"+ (let [ref ["DB" "PUBLIC" "ORDERS"]+ id (store/get-or-assign! store :table ref)]+ (is (= ref (store/id->ref store :table id)))+ (is (= id (store/ref->id store :table ref)))))))++(deftest ref->id-returns-nil-before-assignment-test+ (let [store (make-test-store)]+ (testing "ref->id returns nil for unassigned refs"+ (is (nil? (store/ref->id store :table ["DB" "PUBLIC" "ORDERS"]))))+ (testing "id->ref returns nil for unassigned IDs"+ (is (nil? (store/id->ref store :table 999))))))++(deftest ids-are-unique-across-kinds-test+ (let [store (make-test-store)]+ (testing "IDs don't collide across different kinds"+ (let [db-id (store/get-or-assign! store :database "DB")+ table-id (store/get-or-assign! store :table ["DB" "PUBLIC" "ORDERS"])+ card-id (store/get-or-assign! store :card "card-1")]+ (is (= 3 (count (distinct [db-id table-id card-id]))))))))++;;; ===========================================================================+;;; Entity loading and caching tests+;;; ===========================================================================++(deftest load-database-test+ (let [store (make-test-store)]+ (testing "load-database! returns data with assigned :id"+ (let [data (store/load-database! store "DB")]+ (is (some? data))+ (is (= "DB" (:name data)))+ (is (= "h2" (:engine data)))+ (is (pos-int? (:id data)))))+ (testing "load-database! returns nil for unknown database"+ (is (nil? (store/load-database! store "Nonexistent"))))))++(deftest load-table-test+ (let [store (make-test-store)]+ (testing "load-table! returns data with :id and :db_id"+ (let [data (store/load-table! store ["DB" "PUBLIC" "ORDERS"])]+ (is (some? data))+ (is (= "ORDERS" (:name data)))+ (is (pos-int? (:id data)))+ (is (pos-int? (:db_id data)))))+ (testing "load-table! returns nil for unknown table"+ (is (nil? (store/load-table! store ["DB" "PUBLIC" "MISSING"]))))))++(deftest load-field-test+ (let [store (make-test-store)]+ (testing "load-field! returns data with :id and :table_id"+ (let [data (store/load-field! store ["DB" "PUBLIC" "ORDERS" "ID"])]+ (is (some? data))+ (is (= "ID" (:name data)))+ (is (pos-int? (:id data)))+ (is (pos-int? (:table_id data)))))+ (testing "load-field! returns nil for unknown field"+ (is (nil? (store/load-field! store ["DB" "PUBLIC" "ORDERS" "MISSING"]))))))++(deftest load-card-test+ (let [store (make-test-store)]+ (testing "load-card! returns data with assigned :id"+ (let [data (store/load-card! store "card-1")]+ (is (some? data))+ (is (= "Card 1" (:name data)))+ (is (pos-int? (:id data)))))+ (testing "load-card! returns nil for unknown card"+ (is (nil? (store/load-card! store "nonexistent"))))))++(deftest caching-returns-same-object-test+ (let [store (make-test-store)]+ (testing "loading the same entity twice returns the cached version"+ (let [first-load (store/load-database! store "DB")+ second-load (store/load-database! store "DB")]+ (is (identical? first-load second-load))))))++(deftest cached-entity-test+ (let [store (make-test-store)]+ (testing "cached-entity returns nil before loading"+ (is (nil? (store/cached-entity store :database "DB"))))+ (testing "cached-entity returns data after loading"+ (store/load-database! store "DB")+ (is (some? (store/cached-entity store :database "DB"))))))++(deftest load-table-assigns-database-id-test+ (let [store (make-test-store)]+ (testing "loading a table also assigns an ID to its database"+ (store/load-table! store ["DB" "PUBLIC" "ORDERS"])+ (is (some? (store/ref->id store :database "DB"))+ "Database should have an assigned ID after loading a table"))))++(deftest load-field-assigns-table-id-test+ (let [store (make-test-store)]+ (testing "loading a field also assigns an ID to its table"+ (store/load-field! store ["DB" "PUBLIC" "ORDERS" "ID"])+ (is (some? (store/ref->id store :table ["DB" "PUBLIC" "ORDERS"]))+ "Table should have an assigned ID after loading a field"))))diff --git a/enterprise/backend/test/metabase_enterprise/checker/test_helpers.clj b/enterprise/backend/test/metabase_enterprise/checker/test_helpers.cljnew file mode 100644index 00000000000..dcc3c8ca4cc--- /dev/null+++ b/enterprise/backend/test/metabase_enterprise/checker/test_helpers.clj@@ -0,0 +1,68 @@+(ns metabase-enterprise.checker.test-helpers+ "Shared test helpers for the checker module."+ (:require+ [metabase-enterprise.checker.source :as source]))++(defn make-schema-source+ "Create an in-memory SchemaSource from maps of databases, tables, and fields."+ [{:keys [databases tables fields]}]+ (let [fields-by-table (reduce-kv (fn [m field-path _]+ (let [table-path (subvec field-path 0 3)]+ (update m table-path (fnil conj #{}) field-path)))+ {}+ fields)]+ (reify+ source/SchemaSource+ (resolve-database [_ db-name] (get databases db-name))+ (resolve-table [_ table-path] (get tables table-path))+ (resolve-field [_ field-path] (get fields field-path))+ (fields-for-table [_ table-path] (get fields-by-table table-path))+ (all-field-paths [_] (keys fields))+ (all-database-names [_] (keys databases))+ (all-table-paths [_] (keys tables))+ (tables-for-database [_ db-name] (filterv #(= (first %) db-name) (keys tables))))))++(defn make-assets-source+ "Create an in-memory AssetsSource from maps of entities.+ Supports :cards, :snippets, :transforms, :segments, :dashboards,+ :collections, :documents, :measures."+ [{:keys [cards snippets transforms segments dashboards collections documents measures]}]+ (reify+ source/AssetsSource+ (resolve-card [_ entity-id] (get cards entity-id))+ (resolve-snippet [_ entity-id] (get snippets entity-id))+ (resolve-transform [_ entity-id] (get transforms entity-id))+ (resolve-segment [_ entity-id] (get segments entity-id))+ (resolve-dashboard [_ entity-id] (get dashboards entity-id))+ (resolve-collection [_ entity-id] (get collections entity-id))+ (resolve-document [_ entity-id] (get documents entity-id))+ (resolve-measure [_ entity-id] (get measures entity-id))))++(defn make-schema-index+ "Create a file index for an in-memory schema source.+ Values are :memory since resolution goes through the source, not files."+ [{:keys [databases tables fields]}]+ {:database (zipmap (keys databases) (repeat :memory))+ :table (zipmap (keys tables) (repeat :memory))+ :field (zipmap (keys fields) (repeat :memory))})++(defn make-assets-index+ "Create a file index for an in-memory assets source.+ Values are :memory since resolution goes through the source, not files."+ [{:keys [cards snippets transforms segments dashboards collections documents measures]}]+ (cond-> {:card (zipmap (keys cards) (repeat :memory))}+ snippets (assoc :snippet (zipmap (keys snippets) (repeat :memory)))+ transforms (assoc :transform (zipmap (keys transforms) (repeat :memory)))+ segments (assoc :segment (zipmap (keys segments) (repeat :memory)))+ dashboards (assoc :dashboard (zipmap (keys dashboards) (repeat :memory)))+ collections (assoc :collection (zipmap (keys collections) (repeat :memory)))+ documents (assoc :document (zipmap (keys documents) (repeat :memory)))+ measures (assoc :measure (zipmap (keys measures) (repeat :memory)))))++(defn make-sources-and-index+ "Convenience: split an entities map into schema-source, assets-source, and merged index.+ Returns [schema-source assets-source index]."+ [entities]+ [(make-schema-source entities)+ (make-assets-source entities)+ (merge (make-schema-index entities) (make-assets-index entities))])diff --git a/src/metabase/core/bootstrap.clj b/src/metabase/core/bootstrap.cljindex e0eeac050bd..eb79d3ed4cd 100644--- a/src/metabase/core/bootstrap.clj+++ b/src/metabase/core/bootstrap.clj@@ -1,7 +1,8 @@(ns metabase.core.bootstrap(:gen-class)(:require- [clojure.java.io :as io]))+ [clojure.java.io :as io]+ [clojure.tools.cli :as cli]))(set! *warn-on-reflection* true)@@ -26,11 +27,46 @@;; ensure the [[clojure.tools.logging]] logger factory is the log4j2 version (slf4j is far slower and identified first)(System/setProperty "clojure.tools.logging.factory" "clojure.tools.logging.impl/log4j2-factory")+;;; ===========================================================================+;;; Standalone modes+;;;+;;; These modes don't need the full Metabase infrastructure (app-db, events,+;;; etc.) and can run with minimal namespace loading for fast startup.+;;;+;;; Usage: java -jar metabase.jar --mode checker [checker-specific args...]+;;; ===========================================================================++#_{:clj-kondo/ignore [:discouraged-var]}+(def output!+ "Alias for println so can suppress warning in one place"+ println)++(defn- run-standalone-mode+ "Errors if --mode is specified but not recognized.+ The mode's -main function is responsible for calling System/exit."+ [mode args]+ (let [startup (case mode+ ;; schema checker was moved out to js. Perhaps this will get a long running server mode checker? Or+ ;; perhaps just this one.+ "checker" 'metabase-enterprise.checker.cli/entrypoint+ nil)]+ (if startup+ ((requiring-resolve startup) args)+ (do (binding [*out* *err*]+ (output! (str "Unknown mode: " mode))+ (output! "Available modes: checker"))+ (System/exit 1)))))+(defn -main"Main entrypoint. Invokes [[metabase.core.core/entrypoint]]"[& args];; We need to install the classloader here before other namespaces are loaded since they could launch threads on load.- ;; If a thread is spun uo and put back into a pool before this happens that pool will have a poisoned thread unable to+ ;; If a thread is spun up and put back into a pool before this happens that pool will have a poisoned thread unable to;; see classes in our classloader.((requiring-resolve 'metabase.classloader.core/the-classloader))- (apply (requiring-resolve 'metabase.core.core/entrypoint) args))+ ;; Check for standalone modes first - these skip loading metabase.core.core+ (let [{:keys [options]} (cli/parse-opts args [[nil "--mode MODE"]])+ mode (:mode options)]+ (if mode+ (run-standalone-mode mode args)+ (apply (requiring-resolve 'metabase.core.core/entrypoint) args))))diff --git a/src/metabase/models/serialization.clj b/src/metabase/models/serialization.cljindex 75b6f6cc2e4..1252723ff11 100644--- a/src/metabase/models/serialization.clj+++ b/src/metabase/models/serialization.clj@@ -70,6 +70,7 @@[metabase.lib.schema.parameter :as lib.schema.parameter][metabase.lib.util.match :as lib.util.match][metabase.models.interface :as mi]+ [metabase.models.serialization.resolve :as resolve][metabase.models.visualization-settings :as mb.viz][metabase.util :as u][metabase.util.date-2 :as u.date]@@ -242,7 +243,7 @@(defn identity-hash?"Returns true if s is a valid identity hash string."[s]- (boolean (re-matches #"^[0-9a-fA-F]{8}$" s)))+ (resolve/identity-hash? s));; ## Memoizing `hydrated-hash`;;@@ -450,7 +451,7 @@(defmethod make-spec :default [_ _] nil)-(defn ^:dynamic ^::cache *make-spec*+(defn ^:dynamic *make-spec*"Cachable wrapper around [[make-spec]] that is memoized inside [[with-cache]]."[model-name opts](make-spec model-name opts))@@ -849,9 +850,7 @@(defn entity-id?"Checks if the given string is a 21-character NanoID. Useful for telling entity IDs apart from identity hashes."[id-str]- (boolean (and id-str- (string? id-str)- (re-matches #"^[A-Za-z0-9_-]{21}$" id-str))))+ (resolve/entity-id? id-str));; TODO: Clean up this [[identity-hash]] infrastructure once the `seed_entity_ids` issue is fixed. See above on the;; details of the two hashing schemes.@@ -929,32 +928,33 @@:unique-name-fns (atom {})}));;; # Utilities for implementing serdes-;;; Note that many of these use `^::cache` to cache their lookups during deserialization. This greatly reduces the-;;; number of database lookups, since many entities might belong to eg. a single collection.+;;; These wrapper functions delegate to the current resolver (set by [[with-cache]]).+;;; When no resolver is bound, they fall back to the database-backed resolver.++;; TODO: `requiring-resolve` is needed here because resolve.db requires this ns+;; (for `generate-path`, `field-hierarchy`, `lookup-by-id`, `recursively-find-field-q`).+;; Moving those into resolve.db (or a shared utils ns) would break the cycle and+;; let us require resolve.db directly.+(defn- export-resolver []+ (or resolve/*export-resolver*+ @(requiring-resolve 'metabase.models.serialization.resolve.db/default-export-resolver)))++(defn- import-resolver []+ (or resolve/*import-resolver*+ @(requiring-resolve 'metabase.models.serialization.resolve.db/default-import-resolver)));;; ## General foreign keys-(mu/defn ^:dynamic ^::cache *export-fk*+(mu/defn ^:dynamic *export-fk*"Given a numeric foreign key and its model (symbol, name or IModel), looks up the entity by ID and gets its entity IDor identity hash.- Unusual parameter order means this can be used as `(update x :some_id export-fk 'SomeModel)`.+ Unusual parameter order means this can be used as `(update x :some_id *export-fk* 'SomeModel)`.NOTE: This works for both top-level and nested entities. Top-level entities like `Card` are returned as just aportable ID string.. Nested entities are returned as a vector of such ID strings."[id :- [:maybe pos-int?]model :- ::model-keyword-or-symbol]- (when id- (let [model-name (name model)- entity (t2/select-one model (first (t2/primary-keys model)) id)- path (when entity- (mapv :id (generate-path model-name entity)))]- (cond- (nil? entity) (throw (ex-info "FK target not found" {:model model- :id id- :skip true- ::type :target-not-found}))- (= (count path) 1) (first path)- :else path))))+ (resolve/export-fk (export-resolver) id model))(defmacro ^:private fk-elide"If a call to `*export-fk*` inside of this fails, do not export the whole data structure"@@ -967,7 +967,7 @@(throw e#))nil)))-(mu/defn ^:dynamic ^::cache *import-fk*+(mu/defn ^:dynamic *import-fk*"Given an identifier, and the model it represents (symbol, name or IModel), looks up the correspondingentity and gets its primary key.@@ -980,67 +980,56 @@Unusual parameter order means this can be used as `(update x :some_id import-fk 'SomeModel)`."[eidmodel :- ::model-keyword-or-symbol]- (when eid- (let [eid (if (vector? eid)- (last eid)- eid)- entity (lookup-by-id model eid)]- (if entity- (get entity (first (t2/primary-keys model)))- (throw (ex-info "Could not find foreign key target - bad serdes dependencies or other serialization error"- {:entity_id eid :model (name model)}))))))--(mu/defn ^:dynamic ^::cache *export-fk-keyed*+ (resolve/import-fk (import-resolver) eid model))++(mu/defn ^:dynamic *export-fk-keyed*"Given a numeric ID, look up a different identifying field for that entity, and return it as a portable ID.Eg. `Database.name`.- [[import-fk-keyed]] is the inverse.- Unusual parameter order lets this be called as, for example, `(update x :db_id export-fk-keyed :model/Database :name)`.+ [[*import-fk-keyed*]] is the inverse.+ Unusual parameter order lets this be called as, for example, `(update x :db_id *export-fk-keyed* :model/Database :name)`.Note: This assumes the primary key is called `:id`."[idmodel :- ::model-keyword-or-symbolfield]- (t2/select-one-fn field model :id id))+ (resolve/export-fk-keyed (export-resolver) id model field))-(defn ^:dynamic ^::cache *import-fk-keyed*+(defn ^:dynamic *import-fk-keyed*"Given a single, portable, identifying field and the model it refers to, this resolves the entity and returns itsnumeric `:id`.Eg. `Database.name`.Unusual parameter order lets this be called as, for example,- `(update x :creator_id import-fk-keyed :model/Database :name)`."+ `(update x :creator_id *import-fk-keyed* :model/Database :name)`."[portable model field]- (t2/select-one-pk model field portable))+ (resolve/import-fk-keyed (import-resolver) portable model field));;; ## Users-(mu/defn ^:dynamic ^::cache *export-user*+(mu/defn ^:dynamic *export-user*"Exports a user as the email address.- This just calls [[export-fk-keyed]], but the counterpart [[import-user]] is more involved. This is a unique function+ This just calls [[*export-fk-keyed*]], but the counterpart [[*import-user*]] is more involved. This is a unique functionso they form a pair."[id :- [:maybe ::lib.schema.id/user]]- (when id (*export-fk-keyed* id 'User :email)))+ (resolve/export-user (export-resolver) id))-(mu/defn ^:dynamic ^::cache *import-user*+(mu/defn ^:dynamic *import-user*"Imports a user by their email address.If a user with that email address exists, returns its primary key.If no such user exists, creates a dummy inactive one with the default settings, blank name, and randomized password.Does not send any invite emails."[email :- [:maybe string?]]- (when email- (or (*import-fk-keyed* email 'User :email)- ;; Need to break a circular dependency here.- (:id ((resolve 'metabase.users.models.user/serdes-synthesize-user!) {:email email :is_active false})))))+ (resolve/import-user (import-resolver) email));;; ## Databases-(defn ^:dynamic ^::cache *export-database-fk*+(defn ^:dynamic *export-database-fk*"Given a numeric database ID, return its name as a portable reference.[[*import-database-fk*]] is the inverse."[id](when id(t2/select-one-fn :name [:model/Database :id :name] :id id)))-(defn ^:dynamic ^::cache *import-database-fk*+(defn ^:dynamic *import-database-fk*"Given a portable database name, resolve it back to a numeric ID.[[*export-database-fk*]] is the inverse."[db-name]@@ -1048,29 +1037,22 @@;;; ## Tables-(mu/defn ^:dynamic ^::cache *export-table-fk*+(mu/defn ^:dynamic *export-table-fk*"Given a numeric `table_id`, return a portable table reference.If the `table_id` is `nil`, return `nil`. This is legal for a native question.That has the form `[db-name schema table-name]`, where the `schema` might be nil.- [[import-table-fk]] is the inverse."+ [[*import-table-fk*]] is the inverse."[table-id :- [:maybe ::lib.schema.id/table]](when table-id(let [{:keys [db_id name schema]} (t2/select-one [:model/Table :id :db_id :name :schema] :id table-id)db-name (*export-database-fk* db_id)][db-name schema name])))-(mu/defn ^:dynamic ^::cache *import-table-fk*- "Given a `table_id` as exported by [[export-table-fk]], resolve it back into a numeric `table_id`.+(mu/defn ^:dynamic *import-table-fk*+ "Given a `table_id` as exported by [[*export-table-fk*]], resolve it back into a numeric `table_id`.The input might be nil, in which case so is the output. This is legal for a native question."- [[db-name schema table-name :as table-id] :- [:maybe [:tuple string? [:maybe string?] string?]]]- (when table-id- (if-let [db-id (t2/select-one-fn :id :model/Database :name db-name)]- (or (t2/select-one-fn :id :model/Table :name table-name :schema schema :db_id db-id)- (throw (ex-info (format "table id present, but no table found: %s" table-id)- {:table-id table-id})))- (throw (ex-info (format "table id present, but database not found: %s" table-id)- {:table-id table-id- :database-names (sort (t2/select-fn-vec :name :model/Table))})))))+ [[_db-name _schema _table-name :as table-id] :- [:maybe [:tuple string? [:maybe string?] string?]]]+ (resolve/import-table-fk (import-resolver) table-id))(defn table->path"Given a `table_id` as exported by [[export-table-fk]], turn it into a `[{:model ...}]` path for the Table.@@ -1102,7 +1084,9 @@;;; ## Fields-(defn- field-hierarchy [id]+(defn field-hierarchy+ "Returns the field hierarchy (field + parents) for a field ID. Used by resolvers."+ [id](reverse(t2/select :model/Field{:with-recursive [[[:parents {:columns [:id :name :parent_id :table_id]}]@@ -1128,26 +1112,20 @@[:= :name field][:= :parent_id (recursively-find-field-q table-id rest)]]}))-(mu/defn ^:dynamic ^::cache *export-field-fk*+(mu/defn ^:dynamic *export-field-fk*"Given a numeric `field_id`, return a portable field reference.That has the form `[db-name schema table-name field-name]`, where the `schema` might be nil.[[*import-field-fk*]] is the inverse."[field-id :- [:maybe ::lib.schema.id/field]](when field-id- (let [{field-name :name :keys [parent_id table_id]} (t2/select-one [:model/Field :name :parent_id :table_id] :id field-id)]- (if (nil? parent_id)- (conj (*export-table-fk* table_id) field-name)- (let [fields (field-hierarchy field-id)- [db-name schema table-name] (*export-table-fk* table_id)]- (into [db-name schema table-name] (map :name fields)))))))--(mu/defn ^:dynamic ^::cache *import-field-fk* :- [:maybe pos-int?]+ (let [fields (field-hierarchy field-id)+ [db-name schema table-name] (*export-table-fk* (:table_id (first fields)))]+ (into [db-name schema table-name] (map :name fields)))))++(mu/defn ^:dynamic *import-field-fk*"Given a `field_id` as exported by [[*export-field-fk*]], resolve it back into a numeric `field_id`."- [[db-name schema table-name & fields :as field-id] :- [:maybe [:cat string? [:maybe string?] string? #_fields [:+ string?]]]]- (when field-id- (let [table-id (*import-table-fk* [db-name schema table-name])- field-q (recursively-find-field-q table-id (reverse fields))]- (t2/select-one-pk :model/Field field-q))))+ [[_db-name _schema _table-name & _fields :as field-id] :- [:maybe [:cat string? [:maybe string?] string? #_fields [:+ string?]]]]+ (resolve/import-field-fk (import-resolver) field-id))(defn field->path"Given a `field_id` as exported by [[export-field-fk]], turn it into a `[{:model ...}]` path for the Field.@@ -1936,12 +1914,8 @@;;; ## Memoizing appdb lookups(defmacro with-cache- "Runs body with all functions marked with ::cache re-bound to memoized versions for performance."+ "Runs body with resolvers bound to cached (memoized) versions for performance."[& body]- (let [ns* 'metabase.models.serialization]- `(binding ~(reduce into []- (for [[var-sym var] (ns-interns ns*)- :when (::cache (meta var))- :let [fq-sym (symbol (name ns*) (name var-sym))]]- [fq-sym `(memoize ~fq-sym)]))- ~@body)))+ `(binding [resolve/*export-resolver* ((requiring-resolve 'metabase.models.serialization.resolve.db/cached-export-resolver))+ resolve/*import-resolver* ((requiring-resolve 'metabase.models.serialization.resolve.db/cached-import-resolver))]+ ~@body))diff --git a/src/metabase/models/serialization/resolve.clj b/src/metabase/models/serialization/resolve.cljnew file mode 100644index 00000000000..84f7b8786b3--- /dev/null+++ b/src/metabase/models/serialization/resolve.clj@@ -0,0 +1,252 @@+(ns metabase.models.serialization.resolve+ "Protocols, dynamic vars, and portable-ID utilities for serdes FK resolution.++ This namespace has NO toucan2 dependency — it can be used by lightweight consumers+ (like the checker) that don't need the database."+ (:require+ ^{:clj-kondo/ignore [:discouraged-namespace]} [metabase.legacy-mbql.schema :as mbql.s] ;; legacy usages -- do not use in new code+ [metabase.lib.core :as lib]+ [metabase.lib.schema.id :as lib.schema.id]+ [metabase.lib.schema.parameter :as lib.schema.parameter]+ [metabase.lib.util.match :as lib.util.match]+ [metabase.models.visualization-settings :as mb.viz]+ [metabase.util.malli :as mu]))++(set! *warn-on-reflection* true)++;;; ============================================================+;;; Protocols+;;; ============================================================++(defprotocol SerdesExportResolver+ "Resolve database IDs to portable references during export."+ (export-fk [this id model] "Given a numeric FK and model, return a portable ID (string or vector).")+ (export-fk-keyed [this id model field] "Given a numeric ID, look up a different identifying field.")+ (export-user [this id] "Export a user as their email address.")+ (export-table-fk [this table-id] "Given a numeric table_id, return [db-name schema table-name].")+ (export-field-fk [this field-id] "Given a numeric field_id, return [db-name schema table-name field-name]."))++(defprotocol SerdesImportResolver+ "Resolve portable references back to database IDs during import."+ (import-fk [this eid model] "Given a portable ID and model, return the numeric PK.")+ (import-fk-keyed [this portable model field] "Given a portable identifying field value, return the numeric :id.")+ (import-user [this email] "Import a user by email, creating if needed. Returns PK.")+ (import-table-fk [this path] "Given [db-name schema table-name], return numeric table_id.")+ (import-field-fk [this path] "Given [db-name schema table-name field-name], return numeric field_id."))++;;; ============================================================+;;; Dynamic vars — bound to resolver instances+;;; ============================================================++(def ^:dynamic *export-resolver*+ "The current `SerdesExportResolver` instance. Bound during export."+ nil)++(def ^:dynamic *import-resolver*+ "The current `SerdesImportResolver` instance. Bound during import."+ nil)++;;; ============================================================+;;; Pure predicates+;;; ============================================================++(defn entity-id?+ "Checks if the given string is a 21-character NanoID."+ [id-str]+ (boolean (and id-str+ (string? id-str)+ (re-matches #"^[A-Za-z0-9_-]{21}$" id-str))))++(defn identity-hash?+ "Returns true if s is a valid identity hash string."+ [s]+ (boolean (re-matches #"^[0-9a-fA-F]{8}$" s)))++(defn- portable-id?+ "True if the provided string is either an Entity ID or identity-hash string."+ [s]+ (and (string? s)+ (or (entity-id? s)+ (identity-hash? s))))++;;; ============================================================+;;; import-mbql — depends only on protocols, lib.util.match, lib.schema.id+;;; ============================================================++(defn- mbql-fully-qualified-names->ids*+ [resolver entity]+ (lib.util.match/replace-lite entity+ [#{:field "field"} (opts :guard map?) (fully-qualified-name :guard vector?)]+ [:field (mbql-fully-qualified-names->ids* resolver opts)+ (import-field-fk resolver fully-qualified-name)]++ ;; legacy field refs, still used in parameters and result metadata `field_ref`+ [#{:field "field"} (fully-qualified-name :guard vector?) (opts :guard (some-fn map? nil))]+ [:field (import-field-fk resolver fully-qualified-name) (some->> opts (mbql-fully-qualified-names->ids* resolver))]++ ;; MBQL 3 `:field-id` can (allegedly) still show up sometimes? Support it just in case.+ [(tag :guard #{:field :field-id "field" "field-id"}) (id :guard vector?)]+ [:field (import-field-fk resolver id) nil]++ ;; source-field is also used within parameter mapping dimensions+ ;; example relevant clause - [:field 2 {:source-field 1}]+ {:source-field (fully-qualified-name :guard vector?)}+ (assoc &match :source-field (import-field-fk resolver fully-qualified-name))++ {:database (fully-qualified-name :guard string?)}+ (-> &match+ (assoc :database (if (= fully-qualified-name "database/__virtual")+ lib.schema.id/saved-questions-virtual-database-id+ (import-fk-keyed resolver fully-qualified-name :model/Database :name)))+ (->> (mbql-fully-qualified-names->ids* resolver)))++ {:card-id (entity-id :guard portable-id?)}+ (-> &match+ (assoc :card-id (import-fk resolver entity-id 'Card))+ (->> (mbql-fully-qualified-names->ids* resolver)))++ [#{:metric "metric"} opts (entity-id :guard portable-id?)]+ [:metric (mbql-fully-qualified-names->ids* resolver opts)+ (import-fk resolver entity-id 'Card)]++ [#{:segment "segment"} opts (entity-id :guard portable-id?)]+ [:segment (mbql-fully-qualified-names->ids* resolver opts)+ (import-fk resolver entity-id 'Segment)]++ [#{:measure "measure"} opts (entity-id :guard portable-id?)]+ [:measure (mbql-fully-qualified-names->ids* resolver opts)+ (import-fk resolver entity-id 'Measure)]++ ;; support legacy MBQL 4 refs for things like the serialized Audit v2 queries+ [#{:metric "metric"} (entity-id :guard portable-id?)]+ [:metric (import-fk resolver entity-id 'Card)]++ [#{:segment "segment"} (entity-id :guard portable-id?)]+ [:segment (import-fk resolver entity-id 'Segment)]++ [#{:measure "measure"} (entity-id :guard portable-id?)]+ [:measure (import-fk resolver entity-id 'Measure)]++ {:source-table (_ :guard vector?)}+ (-> &match+ (update :source-table (partial import-table-fk resolver))+ (->> (mbql-fully-qualified-names->ids* resolver)))++ {:source_table (_ :guard vector?)}+ (-> &match+ (update :source_table (partial import-table-fk resolver))+ (->> (mbql-fully-qualified-names->ids* resolver)))++ ;; support legacy MBQL 4 for the Audit v2 queries+ {:source-table (id :guard portable-id?)}+ (-> &match+ (assoc :source-table (str "card__" (import-fk resolver id 'Card)))+ (->> (mbql-fully-qualified-names->ids* resolver)))++ {:source-card (id :guard portable-id?)}+ (-> &match+ (assoc :source-card (import-fk resolver id 'Card))+ (->> (mbql-fully-qualified-names->ids* resolver)))++ {:snippet-id (id :guard portable-id?)}+ (-> &match+ (assoc :snippet-id (import-fk resolver id 'NativeQuerySnippet))+ (->> (mbql-fully-qualified-names->ids* resolver)))))++(defn import-mbql+ "Given an MBQL expression with portable IDs, convert back to numeric IDs.+ Uses `*import-resolver*` if bound, otherwise requires a resolver argument."+ ([exported]+ (import-mbql *import-resolver* exported))+ ([resolver exported]+ (mbql-fully-qualified-names->ids* resolver exported)))++(mu/defn- mbql-clause-tag :- [:maybe [:enum :field :dimension :metric :segment :measure]]+ "Is given form an MBQL entity reference?"+ [form]+ (when (and (vector? form)+ (#{:field :dimension :metric :segment :measure} (keyword (first form))))+ (keyword (first form))))++(defn- normalize [mbql]+ (let [tag (mbql-clause-tag mbql)+ schema (case tag+ :field [:multi+ {:dispatch #(and (vector? %)+ (map? (second %)))}+ [true :mbql.clause/field]+ [false ::mbql.s/field]] ; legacy MBQL clause+ :dimension ::lib.schema.parameter/dimension+ :metric :mbql.clause/metric+ :segment :mbql.clause/segment+ :measure :mbql.clause/measure+ #_else nil)]+ (cond->> mbql+ schema (lib/normalize schema mbql))))++(defn- mbql-id->fully-qualified-name+ [resolver mbql]+ (lib.util.match/replace-lite (normalize mbql)+ ;; `pos-int?` guard is here to make the operation idempotent+ [:field (opts :guard map?) (id :guard pos-int?)]+ [:field (mbql-id->fully-qualified-name resolver opts) (export-field-fk resolver id)]++ ;; legacy (MBQL 4) field refs are still supported in parameter targets and in result metadata `field_ref`...+ [:field (id :guard pos-int?) (opts :guard (some-fn map? nil?))]+ [:field (export-field-fk resolver id) (mbql-id->fully-qualified-name resolver opts)]++ ;; MBQL 3 `:field-id` can (allegedly) still show up sometimes? Support it just in case.+ [(tag :guard #{:field :field-id}) (id :guard pos-int?)]+ [tag (export-field-fk resolver id)]++ {:source-table (id :guard pos-int?)}+ (assoc &match :source-table (export-table-fk resolver id))++ ;; source-field is also used within parameter mapping dimensions+ ;; example relevant clause - [:field 2 {:source-field 1}]+ {:source-field (id :guard pos-int?)}+ (assoc &match :source-field (export-field-fk resolver id))++ [:dimension (dim :guard vector?)]+ [:dimension (mbql-id->fully-qualified-name resolver dim)]++ [:metric opts (id :guard pos-int?)]+ [:metric (mbql-id->fully-qualified-name resolver opts) (export-fk resolver id 'Card)]++ [:segment opts (id :guard pos-int?)]+ [:segment (mbql-id->fully-qualified-name resolver opts) (export-fk resolver id 'Segment)]++ [:measure opts (id :guard pos-int?)]+ [:measure (mbql-id->fully-qualified-name resolver opts) (export-fk resolver id 'Measure)]))++(defn export-mbql+ "Given an MBQL expression, convert it to an EDN structure and turn the non-portable Database, Table and Field IDs+ inside it into portable references."+ ([entity] (export-mbql *export-resolver* entity))+ ([resolver entity]+ (lib.util.match/replace-lite entity+ (_ :guard mbql-clause-tag)+ (mbql-id->fully-qualified-name resolver &match)++ (_ :guard sequential?)+ (mapv export-mbql &match)++ (_ :guard map?)+ (reduce-kv+ (fn [entity k _v]+ (let [f (case k+ :database (fn [db-id]+ (if (= db-id lib.schema.id/saved-questions-virtual-database-id)+ "database/__virtual"+ (export-fk-keyed resolver db-id :model/Database :name)))+ (:card_id :card-id) #(export-fk resolver % :model/Card) ; attributes that refer to db fields use `_`; template-tags use `-`+ (:source_table :source-table) #(export-table-fk resolver %)+ (:source_card :source-card) #(export-fk resolver % :model/Card)+ ::mb.viz/param-mapping-source #(export-field-fk resolver %)+ :segment #(export-fk resolver % :model/Segment)+ :snippet-id #(export-fk resolver % :model/NativeQuerySnippet)+ :lib/metadata (constantly nil)+ #_else (partial export-mbql resolver))]+ (update entity k f)))+ &match+ &match))))diff --git a/src/metabase/models/serialization/resolve/db.clj b/src/metabase/models/serialization/resolve/db.cljnew file mode 100644index 00000000000..89f52e575fd--- /dev/null+++ b/src/metabase/models/serialization/resolve/db.clj@@ -0,0 +1,157 @@+(ns metabase.models.serialization.resolve.db+ "Database-backed implementations of the serdes resolver protocols.++ These are the default resolvers used during normal serdes export/import+ against the application database."+ (:require+ [metabase.models.serialization :as serdes]+ [metabase.models.serialization.resolve :as resolve]+ [toucan2.core :as t2]))++(set! *warn-on-reflection* true)++;;; ============================================================+;;; Export functions+;;; ============================================================++(defn export-fk+ "Given a numeric foreign key and its model, looks up the entity by ID and gets its entity ID or identity hash."+ [id model]+ (when id+ (let [model-name (name model)+ entity (t2/select-one model (first (t2/primary-keys model)) id)+ path (when entity+ (mapv :id (serdes/generate-path model-name entity)))]+ (cond+ (nil? entity) (throw (ex-info "FK target not found" {:model model+ :id id+ :skip true+ :metabase.models.serialization/type :target-not-found}))+ (= (count path) 1) (first path)+ :else path))))++(defn export-fk-keyed+ "Given a numeric ID, look up a different identifying field for that entity."+ [id model field]+ (t2/select-one-fn field model :id id))++(defn export-user+ "Export a user as their email address."+ [resolver id]+ (when id (resolve/export-fk-keyed resolver id 'User :email)))++(defn export-table-fk+ "Given a numeric table_id, return a portable table reference [db-name schema table-name]."+ [table-id]+ (when table-id+ (let [{:keys [db_id name schema]} (t2/select-one :model/Table :id table-id)+ db-name (t2/select-one-fn :name :model/Database :id db_id)]+ [db-name schema name])))++(defn export-field-fk+ "Given a numeric field_id, return a portable field reference [db-name schema table-name field-name]."+ [resolver field-id]+ (when field-id+ (let [fields (serdes/field-hierarchy field-id)+ [db-name schema field-name] (resolve/export-table-fk resolver (:table_id (first fields)))]+ (into [db-name schema field-name] (map :name fields)))))++;;; ============================================================+;;; Import functions+;;; ============================================================++(defn import-fk+ "Given a portable ID and model, return the numeric PK."+ [eid model]+ (when eid+ (let [eid (if (vector? eid) (last eid) eid)+ entity (serdes/lookup-by-id model eid)]+ (if entity+ (get entity (first (t2/primary-keys model)))+ (throw (ex-info "Could not find foreign key target - bad serdes dependencies or other serialization error"+ {:entity_id eid :model (name model)}))))))++(defn import-fk-keyed+ "Given a portable identifying field value, return the numeric :id."+ [portable model field]+ (t2/select-one-pk model field portable))++(defn import-user+ "Import a user by email, creating if needed. Returns PK."+ [resolver email]+ (when email+ (or (resolve/import-fk-keyed resolver email 'User :email)+ ;; Need to break a circular dependency here.+ (:id ((clojure.core/resolve 'metabase.users.models.user/serdes-synthesize-user!) {:email email :is_active false})))))++(defn import-table-fk+ "Given [db-name schema table-name], return numeric table_id."+ [[db-name schema table-name :as table-id]]+ (when table-id+ (if-let [db-id (t2/select-one-fn :id :model/Database :name db-name)]+ (or (t2/select-one-fn :id :model/Table :name table-name :schema schema :db_id db-id)+ (throw (ex-info (format "table id present, but no table found: %s" table-id)+ {:table-id table-id})))+ (throw (ex-info (format "table id present, but database not found: %s" table-id)+ {:table-id table-id+ :database-names (sort (t2/select-fn-vec :name :model/Table))})))))++(defn import-field-fk+ "Given [db-name schema table-name field-name ...], return numeric field_id."+ [resolver [db-name schema table-name & fields :as field-id]]+ (when field-id+ (let [table-id (resolve/import-table-fk resolver [db-name schema table-name])+ field-q (serdes/recursively-find-field-q table-id (reverse fields))]+ (t2/select-one-pk :model/Field field-q))))++;;; ============================================================+;;; Resolver constructors+;;; ============================================================++(def default-export-resolver+ "A stateless database-backed export resolver."+ (reify resolve/SerdesExportResolver+ (export-fk [_ id model] (export-fk id model))+ (export-fk-keyed [_ id model field] (export-fk-keyed id model field))+ (export-user [this id] (export-user this id))+ (export-table-fk [_ table-id] (export-table-fk table-id))+ (export-field-fk [this field-id] (export-field-fk this field-id))))++(def default-import-resolver+ "A stateless database-backed import resolver."+ (reify resolve/SerdesImportResolver+ (import-fk [_ eid model] (import-fk eid model))+ (import-fk-keyed [_ portable model field] (import-fk-keyed portable model field))+ (import-user [this email] (import-user this email))+ (import-table-fk [_ path] (import-table-fk path))+ (import-field-fk [this path] (import-field-fk this path))))++(defn cached-export-resolver+ "Returns a database-backed export resolver with memoized lookups."+ []+ (let [export-fk* (memoize export-fk)+ export-fk-keyed* (memoize export-fk-keyed)+ export-user* (memoize export-user)+ export-table-fk* (memoize export-table-fk)+ export-field-fk* (memoize export-field-fk)]+ (reify resolve/SerdesExportResolver+ (export-fk [_ id model] (export-fk* id model))+ (export-fk-keyed [_ id model field] (export-fk-keyed* id model field))+ (export-user [this id] (export-user* this id))+ (export-table-fk [_ table-id] (export-table-fk* table-id))+ (export-field-fk [this field-id] (export-field-fk* this field-id)))))++(defn cached-import-resolver+ "Returns a database-backed import resolver with memoized lookups."+ []+ (let [import-fk* (memoize import-fk)+ import-fk-keyed* (memoize import-fk-keyed)+ import-user* (memoize import-user)+ import-table-fk* (memoize import-table-fk)+ import-field-fk* (memoize import-field-fk)]+ (reify resolve/SerdesImportResolver+ (import-fk [_ eid model] (import-fk* eid model))+ (import-fk-keyed [_ portable model field] (import-fk-keyed* portable model field))+ (import-user [this email] (import-user* this email))+ (import-table-fk [_ path] (import-table-fk* path))+ (import-field-fk [this path] (import-field-fk* this path)))))diff --git a/test_resources/yaml_checks/collections/cards/native-orders_native_orders.yaml b/test_resources/yaml_checks/collections/cards/native-orders_native_orders.yamlnew file mode 100644index 00000000000..bc67d6d9905--- /dev/null+++ b/test_resources/yaml_checks/collections/cards/native-orders_native_orders.yaml@@ -0,0 +1,81 @@+name: Native Orders+description: A native SQL query+entity_id: native-orders+created_at: '2026-01-01T00:00:00.000000Z'+creator_id: [email protected]+display: table+archived: false+collection_id: null+collection_preview: true+collection_position: null+query_type: native+database_id: Test Database+table_id: null+enable_embedding: false+embedding_params: null+made_public_by_id: null+public_uuid: null+parameters: []+parameter_mappings: []+dataset_query:+ database: Test Database+ native:+ query: select * from orders+ type: native+result_metadata:+- base_type: type/BigInteger+ database_type: BIGINT+ display_name: ID+ effective_type: type/BigInteger+ field_ref:+ - field+ - id+ - base-type: type/BigInteger+ name: id+ semantic_type: type/PK+ source: native+- base_type: type/Float+ database_type: DOUBLE PRECISION+ display_name: total+ effective_type: type/Float+ field_ref:+ - field+ - total+ - base-type: type/Float+ name: total+ source: native+- base_type: type/Integer+ database_type: INTEGER+ display_name: product_id+ effective_type: type/Integer+ field_ref:+ - field+ - product_id+ - base-type: type/Integer+ name: product_id+ source: native+- base_type: type/DateTime+ database_type: TIMESTAMP+ display_name: created_at+ effective_type: type/DateTime+ field_ref:+ - field+ - created_at+ - base-type: type/DateTime+ name: created_at+ semantic_type: type/CreationTimestamp+ source: native+visualization_settings:+ column_settings: null+card_schema: 23+serdes/meta:+- id: native-orders+ label: native_orders+ model: Card+archived_directly: false+dashboard_id: null+document_id: null+embedding_type: null+metabase_version: v0.55.1-SNAPSHOT+source_card_id: null+type: questiondiff --git a/test_resources/yaml_checks/collections/cards/orders-with-fk_orders_with_fk.yaml b/test_resources/yaml_checks/collections/cards/orders-with-fk_orders_with_fk.yamlnew file mode 100644index 00000000000..4cd56b57269--- /dev/null+++ b/test_resources/yaml_checks/collections/cards/orders-with-fk_orders_with_fk.yaml@@ -0,0 +1,66 @@+name: Orders With FK+description: Query with FK in result_metadata+entity_id: orders-with-fk+created_at: '2026-01-01T00:00:00.000000Z'+creator_id: [email protected]+display: table+archived: false+query_type: query+database_id: SQLite DB+table_id:+- SQLite DB+- null+- orders+dataset_query:+ database: SQLite DB+ query:+ source-table:+ - SQLite DB+ - null+ - orders+ type: query+result_metadata:+- base_type: type/Integer+ display_name: ID+ effective_type: type/Integer+ field_ref:+ - field+ - - SQLite DB+ - null+ - orders+ - id+ - null+ id:+ - SQLite DB+ - null+ - orders+ - id+ name: id+ semantic_type: type/PK+- base_type: type/Integer+ display_name: User ID+ effective_type: type/Integer+ field_ref:+ - field+ - - SQLite DB+ - null+ - orders+ - user_id+ - null+ fk_target_field_id:+ - SQLite DB+ - null+ - users+ - id+ id:+ - SQLite DB+ - null+ - orders+ - user_id+ name: user_id+ semantic_type: type/FK+serdes/meta:+- id: orders-with-fk+ label: orders_with_fk+ model: Card+type: questiondiff --git a/test_resources/yaml_checks/collections/cards/orders-with-products_orders_with_products.yaml b/test_resources/yaml_checks/collections/cards/orders-with-products_orders_with_products.yamlnew file mode 100644index 00000000000..4f03f128b5f--- /dev/null+++ b/test_resources/yaml_checks/collections/cards/orders-with-products_orders_with_products.yaml@@ -0,0 +1,197 @@+name: Orders With Products+description: Orders joined with products table+entity_id: orders-with-products+created_at: '2026-01-01T00:00:00.000000Z'+creator_id: [email protected]+display: table+archived: false+collection_id: null+collection_preview: true+collection_position: null+query_type: query+database_id: Test Database+table_id:+- Test Database+- public+- orders+enable_embedding: false+embedding_params: null+made_public_by_id: null+public_uuid: null+parameters: []+parameter_mappings: []+dataset_query:+ database: Test Database+ query:+ fields:+ - - field+ - - Test Database+ - public+ - orders+ - id+ - base-type: type/BigInteger+ - - field+ - - Test Database+ - public+ - orders+ - total+ - base-type: type/Float+ - - field+ - - Test Database+ - public+ - products+ - title+ - base-type: type/Text+ join-alias: Products+ - - field+ - - Test Database+ - public+ - products+ - price+ - base-type: type/Float+ join-alias: Products+ joins:+ - alias: Products+ condition:+ - =+ - - field+ - - Test Database+ - public+ - orders+ - product_id+ - base-type: type/Integer+ - - field+ - - Test Database+ - public+ - products+ - id+ - base-type: type/BigInteger+ join-alias: Products+ fields: all+ source-table:+ - Test Database+ - public+ - products+ strategy: left-join+ source-table:+ - Test Database+ - public+ - orders+ type: query+result_metadata:+- active: true+ base_type: type/BigInteger+ database_type: BIGINT+ display_name: ID+ effective_type: type/BigInteger+ field_ref:+ - field+ - - Test Database+ - public+ - orders+ - id+ - null+ id:+ - Test Database+ - public+ - orders+ - id+ name: id+ position: 0+ semantic_type: type/PK+ source: fields+ table_id:+ - Test Database+ - public+ - orders+ visibility_type: normal+- active: true+ base_type: type/Float+ database_type: DOUBLE PRECISION+ display_name: Total+ effective_type: type/Float+ field_ref:+ - field+ - - Test Database+ - public+ - orders+ - total+ - null+ id:+ - Test Database+ - public+ - orders+ - total+ name: total+ position: 1+ source: fields+ table_id:+ - Test Database+ - public+ - orders+ visibility_type: normal+- active: true+ base_type: type/Text+ database_type: VARCHAR+ display_name: Products -> Title+ effective_type: type/Text+ field_ref:+ - field+ - - Test Database+ - public+ - products+ - title+ - join-alias: Products+ id:+ - Test Database+ - public+ - products+ - title+ name: title+ position: 0+ semantic_type: type/Title+ source: fields+ table_id:+ - Test Database+ - public+ - products+ visibility_type: normal+- active: true+ base_type: type/Float+ database_type: DOUBLE PRECISION+ display_name: Products -> Price+ effective_type: type/Float+ field_ref:+ - field+ - - Test Database+ - public+ - products+ - price+ - join-alias: Products+ id:+ - Test Database+ - public+ - products+ - price+ name: price+ position: 1+ source: fields+ table_id:+ - Test Database+ - public+ - products+ visibility_type: normal+visualization_settings:+ column_settings: null+card_schema: 23+serdes/meta:+- id: orders-with-products+ label: orders_with_products+ model: Card+archived_directly: false+dashboard_id: null+document_id: null+embedding_type: null+metabase_version: v0.55.1-SNAPSHOT+source_card_id: null+type: questiondiff --git a/test_resources/yaml_checks/collections/cards/schemaless-query_schemaless_query.yaml b/test_resources/yaml_checks/collections/cards/schemaless-query_schemaless_query.yamlnew file mode 100644index 00000000000..18555cd1930--- /dev/null+++ b/test_resources/yaml_checks/collections/cards/schemaless-query_schemaless_query.yaml@@ -0,0 +1,61 @@+name: Schemaless Query+description: Query on schema-less SQLite database+entity_id: schemaless-query+created_at: '2026-01-01T00:00:00.000000Z'+creator_id: [email protected]+display: table+archived: false+query_type: query+database_id: SQLite DB+table_id:+- SQLite DB+- null+- users+dataset_query:+ database: SQLite DB+ query:+ source-table:+ - SQLite DB+ - null+ - users+ type: query+result_metadata:+- base_type: type/Integer+ display_name: ID+ effective_type: type/Integer+ field_ref:+ - field+ - - SQLite DB+ - null+ - users+ - id+ - null+ id:+ - SQLite DB+ - null+ - users+ - id+ name: id+ semantic_type: type/PK+- base_type: type/Text+ display_name: Name+ effective_type: type/Text+ field_ref:+ - field+ - - SQLite DB+ - null+ - users+ - name+ - null+ id:+ - SQLite DB+ - null+ - users+ - name+ name: name+ semantic_type: type/Name+serdes/meta:+- id: schemaless-query+ label: schemaless_query+ model: Card+type: questiondiff --git a/test_resources/yaml_checks/collections/cards/simple-orders_simple_orders.yaml b/test_resources/yaml_checks/collections/cards/simple-orders_simple_orders.yamlnew file mode 100644index 00000000000..cc209b395f4--- /dev/null+++ b/test_resources/yaml_checks/collections/cards/simple-orders_simple_orders.yaml@@ -0,0 +1,153 @@+name: Simple Orders+description: A simple query on orders table+entity_id: simple-orders+created_at: '2026-01-01T00:00:00.000000Z'+creator_id: [email protected]+display: table+archived: false+collection_id: null+collection_preview: true+collection_position: null+query_type: query+database_id: Test Database+table_id:+- Test Database+- public+- orders+enable_embedding: false+embedding_params: null+made_public_by_id: null+public_uuid: null+parameters: []+parameter_mappings: []+dataset_query:+ database: Test Database+ query:+ source-table:+ - Test Database+ - public+ - orders+ type: query+result_metadata:+- active: true+ base_type: type/BigInteger+ database_type: BIGINT+ display_name: ID+ effective_type: type/BigInteger+ field_ref:+ - field+ - - Test Database+ - public+ - orders+ - id+ - null+ id:+ - Test Database+ - public+ - orders+ - id+ name: id+ position: 0+ semantic_type: type/PK+ source: fields+ table_id:+ - Test Database+ - public+ - orders+ visibility_type: normal+- active: true+ base_type: type/Float+ database_type: DOUBLE PRECISION+ display_name: Total+ effective_type: type/Float+ field_ref:+ - field+ - - Test Database+ - public+ - orders+ - total+ - null+ id:+ - Test Database+ - public+ - orders+ - total+ name: total+ position: 1+ source: fields+ table_id:+ - Test Database+ - public+ - orders+ visibility_type: normal+- active: true+ base_type: type/Integer+ database_type: INTEGER+ display_name: Product ID+ effective_type: type/Integer+ field_ref:+ - field+ - - Test Database+ - public+ - orders+ - product_id+ - null+ fk_target_field_id:+ - Test Database+ - public+ - products+ - id+ id:+ - Test Database+ - public+ - orders+ - product_id+ name: product_id+ position: 2+ semantic_type: type/FK+ source: fields+ table_id:+ - Test Database+ - public+ - orders+ visibility_type: normal+- active: true+ base_type: type/DateTime+ database_type: TIMESTAMP+ display_name: Created At+ effective_type: type/DateTime+ field_ref:+ - field+ - - Test Database+ - public+ - orders+ - created_at+ - null+ id:+ - Test Database+ - public+ - orders+ - created_at+ name: created_at+ position: 3+ semantic_type: type/CreationTimestamp+ source: fields+ table_id:+ - Test Database+ - public+ - orders+ visibility_type: normal+visualization_settings:+ column_settings: null+card_schema: 23+serdes/meta:+- id: simple-orders+ label: simple_orders+ model: Card+archived_directly: false+dashboard_id: null+document_id: null+embedding_type: null+metabase_version: v0.55.1-SNAPSHOT+source_card_id: null+type: questiondiff --git a/test_resources/yaml_checks/databases/SQLite DB/SQLite DB.yaml b/test_resources/yaml_checks/databases/SQLite DB/SQLite DB.yamlnew file mode 100644index 00000000000..bd75c443074--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/SQLite DB.yaml@@ -0,0 +1,7 @@+name: SQLite DB+engine: sqlite+created_at: '2026-01-01T00:00:00.000000Z'+settings: null+serdes/meta:+- id: SQLite DB+ model: Databasediff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/id.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/id.yamlnew file mode 100644index 00000000000..34fcc5de514--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/id.yaml@@ -0,0 +1,43 @@+name: id+display_name: ID+description: Primary key+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- SQLite DB+- null+- orders+database_type: INTEGER+base_type: type/Integer+effective_type: null+semantic_type: type/PK+database_is_auto_increment: true+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 0+custom_position: 0+database_position: 0+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: SQLite DB+ model: Database+- id: orders+ model: Table+- id: id+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: false+database_is_pk: true+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/user_id.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/user_id.yamlnew file mode 100644index 00000000000..53ab777bb7e--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/fields/user_id.yaml@@ -0,0 +1,47 @@+name: user_id+display_name: User ID+description: Foreign key to users table+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- SQLite DB+- null+- orders+database_type: INTEGER+base_type: type/Integer+effective_type: null+semantic_type: type/FK+database_is_auto_increment: false+database_required: false+fk_target_field_id:+- SQLite DB+- null+- users+- id+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 1+custom_position: 0+database_position: 1+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: SQLite DB+ model: Database+- id: orders+ model: Table+- id: user_id+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/orders.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/orders.yamlnew file mode 100644index 00000000000..9e6bd0b675e--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/orders.yaml@@ -0,0 +1,13 @@+name: orders+display_name: Orders+description: Orders table+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: null+schema: null+db_id: SQLite DB+serdes/meta:+- id: SQLite DB+ model: Database+- id: orders+ model: Tablediff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/orders/segments/recent_orders.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/segments/recent_orders.yamlnew file mode 100644index 00000000000..4e44ec6479a--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/orders/segments/recent_orders.yaml@@ -0,0 +1,21 @@+name: Recent Orders+entity_id: recent-orders-segment+definition:+ database: SQLite DB+ stages:+ - filters:+ - - ">"+ - {}+ - - field+ - {}+ - created_at+ - 2025-01-01+ source-table:+ - SQLite DB+ - null+ - orders+ lib/type: mbql.stage/mbql+ lib/type: mbql/query+serdes/meta:+- id: recent-orders-segment+ model: Segmentdiff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/id.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/id.yamlnew file mode 100644index 00000000000..6a2e07a3e2d--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/id.yaml@@ -0,0 +1,43 @@+name: id+display_name: ID+description: Primary key+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- SQLite DB+- null+- users+database_type: INTEGER+base_type: type/Integer+effective_type: null+semantic_type: type/PK+database_is_auto_increment: true+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 0+custom_position: 0+database_position: 0+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: SQLite DB+ model: Database+- id: users+ model: Table+- id: id+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: false+database_is_pk: true+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/name.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/name.yamlnew file mode 100644index 00000000000..a48af8a14f8--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/users/fields/name.yaml@@ -0,0 +1,43 @@+name: name+display_name: Name+description: User name+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- SQLite DB+- null+- users+database_type: TEXT+base_type: type/Text+effective_type: null+semantic_type: type/Name+database_is_auto_increment: false+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 1+custom_position: 0+database_position: 1+has_field_values: auto-list+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: SQLite DB+ model: Database+- id: users+ model: Table+- id: name+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/SQLite DB/tables/users/users.yaml b/test_resources/yaml_checks/databases/SQLite DB/tables/users/users.yamlnew file mode 100644index 00000000000..090f0d17259--- /dev/null+++ b/test_resources/yaml_checks/databases/SQLite DB/tables/users/users.yaml@@ -0,0 +1,13 @@+name: users+display_name: Users+description: User accounts table+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: null+schema: null+db_id: SQLite DB+serdes/meta:+- id: SQLite DB+ model: Database+- id: users+ model: Tablediff --git a/test_resources/yaml_checks/databases/Test Database/Test Database.yaml b/test_resources/yaml_checks/databases/Test Database/Test Database.yamlnew file mode 100644index 00000000000..fce6f68cae9--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/Test Database.yaml@@ -0,0 +1,33 @@+name: Test Database+description: A test database for CI checker tests+engine: h2+dbms_version:+ flavor: H2+ version: 2.1.214 (2022-06-13)+ semantic-version:+ - 2+ - 1+created_at: '2026-01-01T00:00:00.000000Z'+creator_id: [email protected]+timezone: UTC+auto_run_queries: true+refingerprint: null+is_full_sync: true+is_on_demand: false+is_sample: false+is_audit: false+is_attached_dwh: false+metadata_sync_schedule: 0 0 * * * ? *+cache_field_values_schedule: 0 0 0 * * ? *+settings: null+caveats: null+points_of_interest: null+initial_sync_status: complete+serdes/meta:+- id: Test Database+ model: Database+provider_name: null+router_database_id: null+uploads_enabled: false+uploads_schema_name: null+uploads_table_prefix: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/created_at.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/created_at.yamlnew file mode 100644index 00000000000..29f6ed79e2e--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/created_at.yaml@@ -0,0 +1,45 @@+name: created_at+display_name: Created At+description: When the order was created+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- orders+database_type: TIMESTAMP+base_type: type/DateTime+effective_type: null+semantic_type: type/CreationTimestamp+database_is_auto_increment: false+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 3+custom_position: 0+database_position: 3+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: orders+ model: Table+- id: created_at+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/id.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/id.yamlnew file mode 100644index 00000000000..523095c67a9--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/id.yaml@@ -0,0 +1,45 @@+name: id+display_name: ID+description: Primary key for orders+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- orders+database_type: BIGINT+base_type: type/BigInteger+effective_type: null+semantic_type: type/PK+database_is_auto_increment: true+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 0+custom_position: 0+database_position: 0+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: orders+ model: Table+- id: id+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: false+database_is_pk: true+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/product_id.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/product_id.yamlnew file mode 100644index 00000000000..dad77cd1f6b--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/product_id.yaml@@ -0,0 +1,49 @@+name: product_id+display_name: Product ID+description: Foreign key to products table+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- orders+database_type: INTEGER+base_type: type/Integer+effective_type: null+semantic_type: type/FK+database_is_auto_increment: false+database_required: false+fk_target_field_id:+- Test Database+- public+- products+- id+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 2+custom_position: 0+database_position: 2+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: orders+ model: Table+- id: product_id+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/total.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/total.yamlnew file mode 100644index 00000000000..7dc2eb020e0--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/fields/total.yaml@@ -0,0 +1,45 @@+name: total+display_name: Total+description: Order total amount+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- orders+database_type: DOUBLE PRECISION+base_type: type/Float+effective_type: null+semantic_type: null+database_is_auto_increment: false+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 1+custom_position: 0+database_position: 1+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: orders+ model: Table+- id: total+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/measures/total_revenue.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/measures/total_revenue.yamlnew file mode 100644index 00000000000..09908d42fa7--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/measures/total_revenue.yaml@@ -0,0 +1,20 @@+name: Total Revenue+entity_id: total-revenue-measure+definition:+ database: Test Database+ stages:+ - aggregation:+ - - sum+ - {}+ - - field+ - {}+ - total+ source-table:+ - Test Database+ - public+ - orders+ lib/type: mbql.stage/mbql+ lib/type: mbql/query+serdes/meta:+- id: total-revenue-measure+ model: Measurediff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/orders.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/orders.yamlnew file mode 100644index 00000000000..6fb5c39a56f--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/orders.yaml@@ -0,0 +1,34 @@+name: orders+display_name: Orders+description: Order transactions table+created_at: '2026-01-01T00:00:00.000000Z'+db_id: Test Database+schema: public+entity_type: entity/TransactionTable+active: true+is_upload: false+field_order: database+visibility_type: null+show_in_getting_started: false+initial_sync_status: complete+points_of_interest: null+caveats: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: orders+ model: Table+archived_at: null+collection_id: null+data_authority: unconfigured+data_layer: internal+data_source: null+database_require_filter: null+deactivated_at: null+is_published: false+is_writable: true+owner_email: null+owner_user_id: null+transform_id: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/segments/big_orders.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/segments/big_orders.yamlnew file mode 100644index 00000000000..bfec8397f15--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/orders/segments/big_orders.yaml@@ -0,0 +1,21 @@+name: Big Orders+entity_id: big-orders-segment+definition:+ database: Test Database+ stages:+ - filters:+ - - ">"+ - {}+ - - field+ - {}+ - total+ - 100+ source-table:+ - Test Database+ - public+ - orders+ lib/type: mbql.stage/mbql+ lib/type: mbql/query+serdes/meta:+- id: big-orders-segment+ model: Segmentdiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/id.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/id.yamlnew file mode 100644index 00000000000..cd6fb4cb53c--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/id.yaml@@ -0,0 +1,45 @@+name: id+display_name: ID+description: Primary key for products+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- products+database_type: BIGINT+base_type: type/BigInteger+effective_type: null+semantic_type: type/PK+database_is_auto_increment: true+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 0+custom_position: 0+database_position: 0+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: products+ model: Table+- id: id+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: false+database_is_pk: true+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/price.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/price.yamlnew file mode 100644index 00000000000..18c888b5383--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/price.yaml@@ -0,0 +1,45 @@+name: price+display_name: Price+description: Product price+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- products+database_type: DOUBLE PRECISION+base_type: type/Float+effective_type: null+semantic_type: null+database_is_auto_increment: false+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 2+custom_position: 0+database_position: 2+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: products+ model: Table+- id: price+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/title.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/title.yamlnew file mode 100644index 00000000000..87e204b4c19--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/fields/title.yaml@@ -0,0 +1,45 @@+name: title+display_name: Title+description: Product title+created_at: '2026-01-01T00:00:00.000000Z'+active: true+visibility_type: normal+table_id:+- Test Database+- public+- products+database_type: VARCHAR+base_type: type/Text+effective_type: null+semantic_type: type/Title+database_is_auto_increment: false+database_required: false+fk_target_field_id: null+dimensions: []+json_unfolding: false+parent_id: null+coercion_strategy: null+preview_display: true+position: 1+custom_position: 0+database_position: 1+has_field_values: null+settings: null+caveats: null+points_of_interest: null+nfc_path: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: products+ model: Table+- id: title+ model: Field+database_default: null+database_indexed: null+database_is_generated: false+database_is_nullable: true+database_is_pk: false+database_partitioned: nulldiff --git a/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/products.yaml b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/products.yamlnew file mode 100644index 00000000000..1c541117070--- /dev/null+++ b/test_resources/yaml_checks/databases/Test Database/schemas/public/tables/products/products.yaml@@ -0,0 +1,34 @@+name: products+display_name: Products+description: Product catalog+created_at: '2026-01-01T00:00:00.000000Z'+db_id: Test Database+schema: public+entity_type: entity/ProductTable+active: true+is_upload: false+field_order: database+visibility_type: null+show_in_getting_started: false+initial_sync_status: complete+points_of_interest: null+caveats: null+serdes/meta:+- id: Test Database+ model: Database+- id: public+ model: Schema+- id: products+ model: Table+archived_at: null+collection_id: null+data_authority: unconfigured+data_layer: internal+data_source: null+database_require_filter: null+deactivated_at: null+is_published: false+is_writable: true+owner_email: null+owner_user_id: null+transform_id: null