The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Remove need for versioned Liquibase changeset ids (#78531)
Nathan Voxland2a7df44f0271
diff --git a/.claude/agents/platform-backend-expert.md b/.claude/agents/platform-backend-expert.mdindex ee78b80278e..cc8cd65843e 100644--- a/.claude/agents/platform-backend-expert.md+++ b/.claude/agents/platform-backend-expert.md@@ -19,7 +19,7 @@ All OSS, under `src/metabase/` unless noted.|---|---|| App DB | `metabase.app-db.core` (facade), `.connection`, `.connection-pool-setup`, `.data-source`, `.env`, `.setup`, `.query`, `.query-cancelation`, `.cluster-lock`, `.encryption`, `.h2`, `.update-h2`, `.quartz`, `.transient-error`, `.sql-errors`, `.db` || Query guards | `metabase.app-db.value-guard`, `metabase.app-db.honeysql-guard`; lint `hooks.metabase.toucan.db-ns` (test: `.clj-kondo/test/hooks/metabase/toucan/db_ns_test.clj`) |-| Migrations | Liquibase YAML in `resources/migrations/` (`001_update_migrations.yaml`, `056`-`059` files, then one file per change in per-release dirs `060/` onward); `metabase.app-db.liquibase` (+ `.liquibase.h2`, `.liquibase.mysql`); `metabase.app-db.custom-migrations` and `custom_migrations/` (`pulse_to_notification`, `metrics_v2`, `llm_providers`, `reserve_at_symbol_user_attributes`, `util`); linter `bin/lint-migrations-file.sh` |+| Migrations | Liquibase YAML in `resources/migrations/` (`001_update_migrations.yaml`, `056`-`059` files, per-release dirs `060/`-`065/` with `vNN.` ids that already shipped; new changesets are version-less in year dirs like `2026/`, backported as the same file to release branches back to 58); `metabase.app-db.liquibase` (+ `.liquibase.h2`, `.liquibase.mysql`); `metabase.app-db.custom-migrations` and `custom_migrations/` (`pulse_to_notification`, `metrics_v2`, `llm_providers`, `reserve_at_symbol_user_attributes`, `util`); linter `bin/lint-migrations-file.sh` || CLI | `metabase.cmd.*`: `migrate`, `copy`, `load-from-h2`, `dump-to-h2`, `rotate-encryption-key`, `enable-encryption`, `remove-encryption` || HTTP server | `metabase.server.core`, `.instance`, `.handler` (middleware stack), `.routes`, `.streaming-response`, `.streaming-response.thread-pool`, `.settings`, `.db`; 18 middlewares in `metabase.server.middleware.*` (`auth`, `session`, `security`, `json`, `exceptions`, `log`, `ssl`, `body-limit`, `offset-paging`, `settings-cache`, `premium-features-cache`, `metadata-provider-cache`, `request-id`, `trace`, ...) || API framework | `metabase.api.macros` (`defendpoint`), `.macros.defendpoint.open-api`, `.macros.defendpoint.closed-schemas`, `.macros.scope`, `metabase.api.open-api`, `metabase.api.common`, `metabase.api.response`; routes in `metabase.api-routes.*` |@@ -48,7 +48,7 @@ All OSS, under `src/metabase/` unless noted.## How to work-1. Migrations: follow the "Add app-DB migrations as one file per change" section of the `backend-module-conventions` skill (file layout, IDs, backports, scoped preconditions). Copy the shape of a recent file in the newest dir. Run `bin/lint-migrations-file.sh`. Test with the `test-migrations` macro (`metabase.app-db.schema-migrations-test.impl`); `metabase.app-db.schema-migrations-test` and `metabase.app-db.custom-migrations-test` have examples.+1. Migrations: follow the `app-db-migrations` skill (`.claude/skills/app-db-migrations/SKILL.md`) for the version-less file layout, ids, rollback, and type rules. Rolling back in development is by deployment (`clojure -M:dev:migrate rollback last-deployment`, `dev.migrate/rollback!`); the release `migrate down` steps back one recorded Metabase major and refuses dev-migrated databases. Run `bin/lint-migrations-file.sh`. Test with the `test-migrations` macro (`metabase.app-db.schema-migrations-test.impl`); `metabase.app-db.schema-migrations-test` and `metabase.app-db.custom-migrations-test` have examples.2. Guards and lint: run `metabase.app-db.value-guard-test`, `metabase.app-db.query-test`, and `hooks.metabase.toucan.db-ns-test` (kondo hook tests are on the dev classpath, so `./bin/test-agent :only '[hooks.metabase.toucan.db-ns-test]'` works; `:only '[".clj-kondo/test"]'` runs all of them). A hook change alters lint output for every module, so run `./bin/mage kondo-ratchets` after it.3. Middleware and server: read the stack in `metabase.server.handler`, then the one middleware. Tests live in `test/metabase/server/` (`handler-test`, `streaming-response-test`, `middleware/`).4. API framework: run `metabase.api.macros-test` and `metabase.api.open-api-test`. A change to `defendpoint` touches every endpoint, so also load a few `*.api` namespaces.diff --git a/.claude/skills/app-db-migrations/SKILL.md b/.claude/skills/app-db-migrations/SKILL.mdnew file mode 100644index 00000000000..ef5f064c92c--- /dev/null+++ b/.claude/skills/app-db-migrations/SKILL.md@@ -0,0 +1,176 @@+---+name: app-db-migrations+description: Add, test, run and roll back application-database (Liquibase) migrations. Use when adding a changeset, changing a table or column in the Metabase app DB, writing a custom (Clojure) migration, or rolling migrations back in development.+---++# Application DB migrations++Schema changes to the Metabase application database are Liquibase changesets in YAML under `resources/migrations/`.+Every changeset in the repository is linted by `./bin/lint-migrations-file.sh` (`mage lint-migrations`); run it before+pushing.++## Where a new changeset goes++New changesets are **version-less** and live in a year directory. Create one with+`./bin/mage new-migration <snake_case_name>`, which writes:++```+resources/migrations/2026/20261012_short_snake_case_name.yaml # changeset id: a random one, e.g. k3x9qa+```++- The directory is the current year; the file name is `<yyyymmdd>_<snake_case_name>.yaml` (lower-case letters,+ digits and underscores only). One file per change or per closely related group of changes; several changesets in+ one file are fine.+- The changeset **id is a random identifier with no version**, e.g. `k3x9qa`: `new-migration` generates one for the+ first changeset, checking it is unused in `resources/migrations`. For more changesets in the same file, make up+ another the same way and grep to confirm it is unused. The lint accepts lower-case letters, digits and underscores+ with at least one letter -- not an ISO date or timestamp like the old ids carried (`2026-09-18T12:00:00`), not+ `v66.…`, not all digits, and no leading `v`: releases from before version-less support read the schema's version+ from the highest `id LIKE 'v%'`.+- The Metabase version a changeset ships in is decided by the branch it is merged into, not by anything in the file.++Do **not** start new work in the versioned files (`migrations/0NN/…` with `vNN.` ids, or `0NN_update_migrations.yaml`).+The lint still accepts `vNN.` ids up to v66, so changesets already in flight for 66 can land, and refuses them from+v67 on. Don't add versioned changesets on release branches either: changelog order puts every versioned file before+the year directories, so a versioned changeset added after a version-less one would run before it on a fresh install+but after it on an upgraded one.++## Backporting a changeset++Version-less support is backported to patch releases of every supported major back to v58, so a+version-less changeset is backported like any other change: cherry-pick the **same file** onto each release branch.+Liquibase identifies a changeset by file path, id and author, so the upgraded customer's database already has it as+run. A few rules follow from the same changeset meeting different histories:++- **The file must be byte-identical on every branch.** Liquibase checksums ran changesets, and a different body on+ another branch fails checksum validation when a customer upgrades onto it. Fix a mistake with a new changeset.+- **Ship it on every branch from the oldest target up to master.** A customer can upgrade from any patch of an older+ major to any later one.+- **Use only schema that exists in the oldest major it targets.**+- **It runs at different points in different histories.** On a fresh install of a newer major it runs after that+ major's `vNN.` changesets; on a database upgraded from an older major's patch it already ran before them. Make sure+ none of the later majors' changesets conflict with it either way round, e.g. by both creating the same object.+- Customers on a patch from before the backport still boot against a database it ran on: those releases ignore+ changelog rows they don't know, and Metabase keeps a marker row last in `databasechangelog` so they read the schema's+ version correctly (see `metabase.app-db.liquibase.versions/record-legacy-version-tracking!`).++## Writing the changeset++```yaml+databaseChangeLog:+ - changeSet:+ id: k3x9qa+ author: yourgithubhandle+ comment: Add entity_id to glossary+ changes:+ - addColumn:+ tableName: glossary+ columns:+ - column:+ name: entity_id+ type: varchar(21)+ remarks: NanoID for serialization+ rollback:+ - dropColumn:+ tableName: glossary+ columnName: entity_id+```++Rules the lint enforces (and reviewers expect):++- **One change per changeset** (several `sql` changes are allowed only when each targets a different `dbms`).+- `comment` is required; `createTable` needs `remarks`, and by convention so does every column.+- Use the portable types: `${boolean.type}`, `${timestamp_type}`, `${text.type}`, `${blob.type}`. Never bare+ `boolean`, `text`, `blob`, `datetime` or `timestamp without time zone`.+- **Rollback**: include a `rollback` key unless the change type supports Liquibase auto-rollback (`createTable`,+ `addColumn`, `createIndex`, `addForeignKeyConstraint`, `renameColumn`, …). `sql`, `dropColumn`, `dropTable`,+ `modifyDataType` and the like need an explicit rollback (an empty `rollback:` is allowed when there is genuinely+ nothing to undo).+- Preconditions that check a constraint or index must be scoped to a table (`indexExists` / `primaryKeyExists` /+ `uniqueConstraintExists` need `tableName`, `foreignKeyConstraintExists` needs `foreignKeyTableName`), or Liquibase+ snapshots the whole schema to answer them. Preconditions are rarely needed at all: rely on Liquibase's changeset+ tracking.+- No `deleteCascade` on an `addColumn` constraint (add the foreign key with `addForeignKeyConstraint` instead).+- Migrations must work on H2, PostgreSQL and MySQL/MariaDB; anything database-specific gets a `dbms:` filter.+- **Shipped changesets are immutable.** Fix a mistake with a new changeset; never edit or delete one that a release+ contains. `runOnChange: true` changesets (views) are the one exception and may be edited in place.++## Retiring a shipped changeset++Never just delete a shipped changeset: instances keep its row in `DATABASECHANGELOG` forever, Liquibase then logs it+as "not rolled back", and any replacement that recreates the same object fails with a duplicate-object error. To+retire one when you must, add a new changeset that runs `DELETE FROM DATABASECHANGELOG WHERE id IN (...)` for the+retired ids (`resources/migrations/059_update_migrations.yaml::v59.2026-01-09T11:30:00` retires+`v58.2026-01-09T12:00:00`).++## Data migrations (Clojure)++Prefer SQL changesets. Anything that SQL can't express goes in `src/metabase/app_db/custom_migrations.clj` (larger+ones get their own namespace under `custom_migrations/`) with+`define-reversible-migration` (or `define-migration` when there is nothing to reverse), referenced from a changeset:++```yaml+ changes:+ - customChange:+ class: metabase.app_db.custom_migrations.MigrateSomething+ rollback:+```++Custom migrations are append-only, must be idempotent and resumable, and must batch their work: never hold a write+lock on a large table for the whole run. They run inside a transaction and must use table names, never `:model/*` or+other application code: that code changes after the migration ships and breaks it (some older migrations use+`:model/*`; don't copy them). Decrypt encrypted columns such as `metabase_database.details` and `setting.value`+before reading them. The `metabase.app-db.*` module, custom migrations included, is exempt from the `<module>.db`+rule.++## Testing a migration++Migration tests live in `test/metabase/app_db/schema_migrations_test.clj` and use+`metabase.app-db.schema-migrations-test.impl/test-migrations`: it migrates up to the changeset before yours, lets you+seed data, runs your changeset(s) when you call `migrate!`, and you assert on the result:++```clojure+(deftest glossary-entity-id-test+ (impl/test-migrations ["k3x9qa"] [migrate!]+ ;; seed rows, then+ (migrate!)+ (is ...)))+```++See the macro's docstring for asserting on the rollback too. Run them on H2 and Postgres, and add MySQL when the+change touches DDL or JSON columns. On H2, in a fresh JVM:++```sh+MB_DB_TYPE=h2 ./bin/test-agent :only '[metabase.app-db.schema-migrations-test/glossary-entity-id-test]'+```++## Running and rolling back in development++Every `migrate up` run is recorded as its own Liquibase deployment. Use the dev tooling (in the REPL,+`(dev.migrate/migrate!)`, `(dev.migrate/rollback! :last-deployment)` and `(dev.migrate/rollback! :deployment <id>)`;+or the CLI below — it needs the `:dev` alias):++```sh+clojure -M:dev:migrate up # apply unrun changesets+clojure -M:dev:migrate status # latest applied changeset+clojure -M:dev:migrate rollback last-deployment # undo the last migrate up run+clojure -M:dev:migrate rollback deployment <id> # undo everything after that deployment_id+```++`deployment_id`s are in the `databasechangelog_version` table (one row per deployment and Metabase version;+`ran_migrations` tells the version that ran the deployment from versions that merely booted against it). `rollback+count N` and `rollback id <id>` are raw Liquibase rollbacks that bypass the version bookkeeping; only use them for a+deliberate partial rollback.++The release command `migrate down` (`java -jar metabase.jar migrate down`, `mdb/migrate! :down`) rolls back to the+previous recorded **major** Metabase version and refuses a database whose newest deployment was made by a development+build, naming the dev tooling above. A release binary booted against such a database refuses the same way: roll the+development deployments back first, or rebuild the database.++## Related++- `bin/lint-migrations-file/` — the lint and its tests (`cd bin/lint-migrations-file && clojure -M:test`).+- `src/metabase/app_db/liquibase/versions.clj` — how deployments and their Metabase versions are recorded+ (`databasechangelog_version`, the `vNN.legacy-version-tracking` marker for older binaries).+- `src/metabase/app_db/liquibase/rollback.clj` — how `migrate down` and the dev rollback pick and reverse deployments.+- `dev/src/dev/migrate.clj` — the development tooling.diff --git a/.claude/skills/backend-module-conventions/SKILL.md b/.claude/skills/backend-module-conventions/SKILL.mdindex 56ad1a349ae..35c783ccc8f 100644--- a/.claude/skills/backend-module-conventions/SKILL.md+++ b/.claude/skills/backend-module-conventions/SKILL.md@@ -59,19 +59,15 @@ not the lint covers it yet:The docstring of `metabase.app-db.value-guard` explains what the guard refuses without any marker.-## Add app-DB migrations as one file per change--`./bin/lint-migrations-file.sh` enforces these rules. Run it after every change.--- Create the file with `./bin/mage new-migration <NN> <snake_name>` (e.g. `./bin/mage new-migration 65 add_user_table`). It writes `resources/migrations/<NNN>/<yyyyMMdd>_<snake_name>.yaml` with a changeset stub. Fill in the TODOs.-- Starting with v60, each change is its own file. `NNN` is the earliest release the change ships in. New work goes in the newest directory. Don't add to `001_update_migrations.yaml`. Its footer describes the legacy format.-- Changeset IDs are `v<NN>.<suffix>`, where `NN` matches the directory and the suffix is a random 6-char lowercase alphanumeric string (e.g. `v65.m2xncr`). `new-migration` generates one for the first changeset. For extra changesets in the same file, make up a new suffix and grep `resources/migrations` to confirm it is unused. IDs are not ordered within a directory.-- Don't edit a shipped changeset. Add a new one. (Liquibase does recompute checksums and some edits are technically harmless, but the line between "cosmetic" and "ran instances now diverge from fresh installs" is easy to misjudge, so just add a new changeset.) Never delete a shipped changeset either: instances keep the row in `DATABASECHANGELOG` forever; removing the id makes Liquibase log "not rolled back" and any replacement that recreates the same object fails with a duplicate-object error. To retire one when you must, add a NEW changeset that does `DELETE FROM DATABASECHANGELOG WHERE id IN (...)` the retired ids. Put it in the same file as the retired changeset when it ships in the same release (`063/20260714_field_fk_target_field_id_fk.yaml::v63.2026-07-22T12:00:00`), or in the current release's file when it retires an older release's changeset (`059_update_migrations.yaml::v59.2026-01-09T11:30:00` retires `v58.2026-01-09T12:00:00`).-- To backport, copy the changesets into the older directory with `v<older>` IDs. Guard each copy with `preConditions: onFail: MARK_RAN` plus `not: changeSetExecuted` that points at the newer ID, so databases that already ran it skip it. `resources/migrations/064/20260911_glossary_entity_id.yaml` shows the pattern.-- Give every `indexExists`, `primaryKeyExists`, and `uniqueConstraintExists` precondition a `tableName`, and every `foreignKeyConstraintExists` a `foreignKeyTableName`. Without it, Liquibase snapshots the whole schema, which takes seconds per check.-- Prefer SQL changesets. A custom migration (`metabase.app-db.custom-migrations`, `define-migration` / `define-reversible-migration`) must use table names, never `:model/*` or other application code. Application code changes later, and then the migration breaks. Decrypt encrypted columns such as `metabase_database.details` and `setting.value` before reading them.-- The whole `metabase.app-db.*` module, custom migrations included, is exempt from the `<module>.db` rule (the `app-db-namespaces` group in `.clj-kondo/config.edn`).-- Test with the `test-migrations` macro (`metabase.app-db.schema-migrations-test.impl`) on H2 and Postgres. Add MySQL when the change touches DDL or JSON columns.+## Add app-DB migrations++Follow the `app-db-migrations` skill (`.claude/skills/app-db-migrations/SKILL.md`): changesets are version-less, live+in year directories like `resources/migrations/2026/` (`./bin/mage new-migration <name>`), and are backported by+cherry-picking the same file to each release branch. `./bin/lint-migrations-file.sh` enforces the rules. The versioned+layout (`0NN/` directories, `vNN.` ids) only holds changesets that already shipped.++The whole `metabase.app-db.*` module, custom migrations included, is exempt from the `<module>.db` rule (the+`app-db-namespaces` group in `.clj-kondo/config.edn`).## Cross module boundaries through `:api`diff --git a/CLAUDE.md b/CLAUDE.mdindex b8eaf15581a..b2c43db07f7 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -26,6 +26,10 @@ For detailed guidance on writing and reviewing code and documentation, see the s- **clj-nrepl-eval** - This is another good mechanism for running Clojure code on an nrepl server.+### Application database++- **[app-db-migrations](.claude/skills/app-db-migrations/SKILL.md)** - Add, test, run and roll back application-database (Liquibase) migrations: version-less changesets, lint rules, dev rollback by deployment+## TypeScript- **[typescript-write](.claude/skills/typescript-write/SKILL.md)** - TypeScript/JavaScript development and best practicesdiff --git a/bb.edn b/bb.ednindex 3c1f142d9fa..86e77f7d975 100644--- a/bb.edn+++ b/bb.edn@@ -182,12 +182,11 @@(lint-migrations-file/-main))}new-migration- {:doc "Create a new directory-based migration file (v60+)"+ {:doc "Create a new version-less migration file in the current year's directory":requires [[mage.new-migration :as new-migration]]- :examples [["./bin/mage new-migration 60 mq_indexes" "Create resources/migrations/060/20260905_mq_indexes.yaml"]]+ :examples [["./bin/mage new-migration mq_indexes" "Create resources/migrations/2026/20261012_mq_indexes.yaml with a random changeset id"]]:arg-schema [:tuple- [:int {:name "version" :description "Major version number (e.g., 60)"}]- [:string {:name "description" :description "Lowercase alphanumeric/underscore name (e.g., mq_indexes)"}]]+ [:string {:name "description" :description "Lowercase alphanumeric/underscore name for the file (e.g., mq_indexes)"}]]:task (task! (new-migration/new-migration parsed))}jar-downloaddiff --git a/bin/lint-migrations-file/README.md b/bin/lint-migrations-file/README.mdindex 71d445b4673..efe6ee075f0 100644--- a/bin/lint-migrations-file/README.md+++ b/bin/lint-migrations-file/README.md@@ -1,10 +1,18 @@-Linter that validates the Liquibase migrations file against a spec. This lets us check and enforce additional-constraints for the `000_migrations.yaml` file (e.g. make sure you're not using duplicate migration IDs, or including-more than one change in a single `changeSet`).+Linter that validates every Liquibase changelog file under `resources/migrations/` against a spec. This lets us check+and enforce additional constraints (e.g. make sure you're not using duplicate migration IDs, or including more than+one change in a single `changeSet`), plus the file-layout rules:++- `resources/migrations/<year>/<yyyymmdd>_<name>.yaml` -- the current layout: version-less ids (lower-case letters,+ digits and underscores, at least one letter, no leading `v`, no ISO date/timestamp like the old+ `vNN.2024-03-18T16:00:00` ids). The version a changeset ships in is decided by the branch it is merged into.+- `resources/migrations/0NN/…` and `0NN_update_migrations.yaml` -- the older versioned layouts, whose ids carry the+ major (`vNN.…`). Versioned ids are refused from v67 on (`first-versionless-major`). New changesets are version-less+ on every branch, including the release branches of earlier majors, whose patch releases get version-less support+ backported.Older/existing migrations are validated with a less-strict spec; newer ones use a stricter spec that enforces some-additional constraints. Migrations 172 and newer use the stricter spec. The less-strict specs are in `x.unstrict`-namespaces while equivalent stricter ones are in `x.strict` namespaces.+additional constraints. The less-strict specs are in `x.unstrict` namespaces while equivalent stricter ones are in+`x.strict` namespaces. See `.claude/skills/app-db-migrations/SKILL.md` for the rules as a checklist.Run the linter withdiff --git a/bin/lint-migrations-file/src/lint_migrations_file.cljc b/bin/lint-migrations-file/src/lint_migrations_file.cljcindex 51d4ab112b1..017c1a5bd46 100644--- a/bin/lint-migrations-file/src/lint_migrations_file.cljc+++ b/bin/lint-migrations-file/src/lint_migrations_file.cljc@@ -56,9 +56,17 @@(throw (validation-error "Change set IDs are not distinct." {:duplicates duplicates})))))(defn- directory-based-migration-file?- "Returns true if the file is a directory-based migration file (e.g., `060/20260905_mq_indexes.yaml`)."+ "Returns true if the file is a directory-based migration file. This covers both the version-numbered directories+ (e.g., `060/20260905_mq_indexes.yaml`) and the newer year-based directories (e.g., `2026/20260905_workspaces.yaml`)."[file]- (boolean (re-matches #".*\d{3}/\d{8}_[a-z0-9_]+\.yaml$" (str file))))+ (boolean (re-matches #".*(\d{3}|\d{4})/\d{8}_[a-z0-9_]+\.yaml$" (str file))))++(defn- year-dir-migration-file?+ "Returns true if the file lives in a year-based migration directory (e.g., `2026/20260905_workspaces.yaml`). Unlike+ version-numbered directories, year directories are not tied to a Metabase version: their changeset IDs are+ version-less and the version a changeset ships with is decided by which branch it is merged into."+ [file]+ (boolean (re-find #"[/\\]\d{4}[/\\]\d{8}_[a-z0-9_]+\.yaml$" (str file))))(defn- file-version"Extracts the migration version number from a file.@@ -98,18 +106,21 @@{:out-of-order-ids out-of-order-ids}))))))(defn- require-change-set-ids-in-correct-file [change-log file]- (let [fv (file-version file)- ids (change-set-ids change-log)- wrong-file-ids- (->> ids- (filter (fn [id]- (let [id-version (parse-long (re-find #"\d+" id))]- (if (= fv 1)- (> id-version 55)- (not= fv id-version))))))]- (when (seq wrong-file-ids)- (throw (validation-error "Change set IDs are in the wrong file"- {:wrong-file-ids wrong-file-ids})))))+ ;; year-based directories are not tied to a version, so their version-less IDs are not bound to any file version.+ (when-not (year-dir-migration-file? file)+ (let [fv (file-version file)+ ids (change-set-ids change-log)+ wrong-file-ids+ (->> ids+ (filter (fn [id]+ ;; a version-less id has no digits, so no version, and belongs in no versioned file+ (let [id-version (some-> (re-find #"\d+" id) parse-long)]+ (if (= fv 1)+ (some-> id-version (> 55))+ (not= fv id-version))))))]+ (when (seq wrong-file-ids)+ (throw (validation-error "Change set IDs are in the wrong file"+ {:wrong-file-ids wrong-file-ids}))))))(defn- check-change-use-types?"Return `true` if change use any type in `types`."@@ -147,7 +158,11 @@(let [cs-id (get-in change-set [:changeSet :id])](and cs-id(if version- (changeset-at-or-after? file cs-id version id)+ ;; year-dir changesets have version-less ids that+ ;; `changeset-at-or-after?` cannot parse, but they are by+ ;; definition newer than every `vNN.` threshold+ (or (year-dir-migration-file? file)+ (changeset-at-or-after? file cs-id version id))true)))))(filter #(check-change-set-use-types? target-types %))(map #(get-in % [:changeSet :id]))@@ -214,6 +229,16 @@E.g., `v60.my-migration-name` or `v60.1`"#"^v\d{2,}\..+$")+(defn valid-version-less-id?+ "Whether `id` is a valid version-less changeset ID: lowercase letters/digits/underscores, at least one letter, and no+ leading `v`. Binaries from before version-less support read the schema's major from the highest `id LIKE 'v%'` and+ roll back by the number in those ids, so a `view_metrics` id would sort above every `vNN.` id there and break both."+ [id]+ (boolean (and (string? id)+ (re-matches #"[a-z0-9_]+" id)+ (re-find #"[a-z]" id)+ (not (str/starts-with? id "v")))))+(defn- require-change-set-ids-match-file-format"Enforces that changeset IDs use the correct format for the file type:- Directory-based files (v60+): IDs must be prefixed with v{version}. (e.g., v60.anything)@@ -221,9 +246,36 @@- Other per-release files: IDs must match the timestamp format"[change-log file](cond+ ;; year-based directories use version-less IDs (e.g. `aeiagus09e`): lowercase letters/digits/underscores with at+ ;; least one letter, and no leading `v`. This keeps out timestamp-style IDs (`2026-02-09T12:00:00`) and+ ;; version-prefixed IDs (`v60.aeiagus09e`, `v60abc`) -- the version must not be encoded in the ID -- and all-digit+ ;; IDs (`20260703`), which `decide-liquibase-file` would mistake for a pre-4.2 changeset if the year-directory+ ;; filename signal were ever lost. A leading `v` would pollute the `id LIKE 'v%'` version scans that older+ ;; binaries use for downgrade detection and rollback.+ (year-dir-migration-file? file)+ (let [ids (change-set-ids change-log)+ ;; the ISO date/timestamp the versioned ids were built from (`v50.2024-03-18T16:00:00`): checked first so+ ;; the habit gets its own message rather than a generic character-set complaint+ iso-date-id? (fn [id] (and (string? id) (re-find #"\d{4}-\d{2}-\d{2}" id)))+ dated-ids (filter iso-date-id? ids)+ bad-ids (remove valid-version-less-id? ids)]+ (when (seq dated-ids)+ (throw (validation-error+ (format (str "Migration file contains changeset IDs that look like dates or timestamps; a version-less "+ "ID names the change instead (the file name already carries the date): %s")+ (str/join ", " dated-ids))+ {:date-like-ids (vec dated-ids)})))+ (when (seq bad-ids)+ (throw (validation-error+ (format (str "Migration file contains invalid changeset IDs (use a version-less ID: lowercase "+ "letters/digits/underscores with at least one letter, no leading 'v', and no "+ "timestamp or version prefix): %s")+ (str/join ", " bad-ids))+ {:invalid-ids (vec bad-ids)}))))+(directory-based-migration-file? file)(let [ids (change-set-ids change-log)- bad-ids (remove #(re-matches directory-based-id-format-re %) ids)]+ bad-ids (remove #(and (string? %) (re-matches directory-based-id-format-re %)) ids)](when (seq bad-ids)(throw (validation-error(format "Directory-based migration file contains IDs without v{version}. prefix: %s"@@ -277,6 +329,36 @@(:id changeSet) (name precondition) (name required)){:id (:id changeSet)}))))+(defn require-no-run-always+ "Refuses `runAlways: true` changesets. Liquibase lists a runAlways changeset as unrun on every update, which breaks+ how Metabase runs and tracks migrations:++ 1. Every boot of every instance finds an unrun changeset, so it takes the migration lock and runs a migration. Each+ run records a new deployment, adds a `databasechangelog_version` row and moves the legacy-version-tracking+ marker row.+ 2. With `MB_DB_AUTOMIGRATE=false`, Metabase prints the migration SQL and refuses to start on every boot, even+ after the SQL has been applied, because the changeset is still unrun.+ 3. Each run moves the changeset's changelog row into its own deployment. The deployment of the run before is then+ left with no changelog rows, but its `databasechangelog_version` row stays until a rollback removes it: one more+ row per boot.+ 4. An older patch booted against a newer patch's schema runs it too, so the newest deployment looks like it was+ run by the older patch (the `migrate down` guard has to look at every deployment because of this).++ Use `runOnChange` for objects that must follow their definition (views). To allow runAlways, first make+ `metabase.app-db.liquibase/unrun-migrations` and the `migrate print` path ignore runAlways changesets when they+ decide whether there is work to do (run them only with other migrations, or in a separate step that records no+ deployment), and remove the version rows of deployments that a re-run leaves empty when it happens, not only at+ rollback; then run the upgrade, downgrade and `MB_DB_AUTOMIGRATE=false` scenarios again."+ [change-log]+ (doseq [{:keys [changeSet]} change-log+ :when (true? (:runAlways changeSet))]+ (throw (validation-error+ (format (str "Migration '%s' uses runAlways, which Metabase does not support: every boot would run it as a "+ "new migration, and with MB_DB_AUTOMIGRATE=false Metabase would never start. See "+ "require-no-run-always in bin/lint-migrations-file.")+ (:id changeSet))+ {:id (:id changeSet)}))))+(s/def ::changeSet(s/spec :change-set.strict/change-set))@@ -289,12 +371,40 @@(defn- major-version"Returns major version from id string, e.g. 44 from \"v44.00-034\".For directory-based migrations (file path matches `NNN/`), extracts- version from the file path. Otherwise parses from the id string."+ version from the file path. Otherwise parses from the id string.+ Returns nil for year-based directories, whose IDs are version-less and not tied to a version."[id-str file]- (if-let [[_ file-version] (when file (re-find #"(\d{3})[/\\]" (str file)))]- (Integer/parseInt file-version)- (when (string? id-str)- (some-> (re-find #"\d+" id-str) Integer/parseInt))))+ (cond+ (and file (year-dir-migration-file? file))+ nil++ (string? id-str)+ (if-let [[_ file-version] (when file (re-find #"(\d{3})[/\\]" (str file)))]+ (Integer/parseInt file-version)+ (some-> (re-find #"\d+" id-str) Integer/parseInt))++ :else+ (when-let [[_ file-version] (when file (re-find #"(\d{3})[/\\]" (str file)))]+ (Integer/parseInt file-version))))++(def first-versionless-major+ "The first Metabase major whose changesets must all be version-less: they live in a year-based directory with a version-less ID"+ 67)++(defn- require-no-versioned-changesets-from-first-versionless-major+ "Ensures no `vNN.`-prefixed changeset for [[first-versionless-major]] or later exists, whichever kind of versioned+ file it is in. Year-based directories are exempt: their IDs are version-less by construction."+ [change-log file]+ (when-not (year-dir-migration-file? file)+ (let [versioned-ids (filter #(some-> (major-version % file) (>= first-versionless-major))+ (change-set-ids change-log))]+ (when (seq versioned-ids)+ (throw (validation-error+ (str "Versioned changesets are not allowed from v" first-versionless-major " on; add new changesets "+ "to a year-based directory (e.g. migrations/2026/) with version-less IDs: "+ (str/join ", " versioned-ids))+ {:versioned-ids (vec versioned-ids)+ :first-versionless-major first-versionless-major}))))))(def change-types-supporting-rollback"This set was generated with a little grep and awk from the docs here:@@ -345,6 +455,7 @@(require-distinct-change-set-ids change-log)(require-change-set-ids-in-correct-file change-log file)+ (require-no-versioned-changesets-from-first-versionless-major change-log file)(require-change-set-ids-in-order change-log file)(require-change-set-ids-match-file-format change-log file)(require-no-bare-blob-or-text-types change-log)@@ -352,6 +463,7 @@(require-no-datetime-type change-log file)(require-entity-id-type change-log file)(require-table-scoped-existence-preconditions change-log)+ (require-no-run-always change-log)(let [{:keys [changeSet]:as all} (group-by only-key change-log)](when-not (set/subset? (set (keys all)) #{:property :objectQuotingStrategy :changeSet :logicalFilePath})diff --git a/bin/lint-migrations-file/test/lint_migrations_file_test.clj b/bin/lint-migrations-file/test/lint_migrations_file_test.cljindex 72cb459019b..7fbc6c9879b 100644--- a/bin/lint-migrations-file/test/lint_migrations_file_test.clj+++ b/bin/lint-migrations-file/test/lint_migrations_file_test.clj@@ -230,9 +230,9 @@(is (= :ok(validate-id "v56.2024-01-01T10:30:00" "056_update_migrations.yaml")))(is (= :ok- (validate-id "v99.2024-01-01T10:30:00" "099_update_migrations.yaml")))+ (validate-id "v60.2024-01-01T10:30:00" "060_update_migrations.yaml")))(is (= :ok- (validate-id "v500.2024-01-01T10:30:00" "500_update_migrations.yaml")))+ (validate-id "v64.2024-01-01T10:30:00" "064_update_migrations.yaml")))(is(thrown-with-msg?clojure.lang.ExceptionInfo@@ -244,6 +244,155 @@#"Change set IDs are in the wrong file"(validate-id "v57.2024-01-01T10:30:00" "056_update_migrations.yaml"))))))+(deftest ^:parallel no-versioned-changesets-from-first-versionless-major-test+ (testing "v67+ changesets must be version-less in a year-based directory, whatever kind of file they are in"+ (doseq [[file id] [["migrations/067/20270101_something.yaml" "v67.abc123"]+ ["067_update_migrations.yaml" "v67.2027-01-01T12:00:00"]+ ["migrations/068/20270601_something.yaml" "v68.abc123"]]]+ (is-thrown-with-error-info? (str "Versioned changesets are not allowed from v67 on; add new changesets to a "+ "year-based directory (e.g. migrations/2026/) with version-less IDs: " id)+ {:versioned-ids [id] :first-versionless-major 67}+ (validate-file (io/file file) (mock-change-set :id id)))))+ (testing "every offending id is reported"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"v67.abc123, v67.def456$"+ (validate-file (io/file "migrations/067/20270101_something.yaml")+ (mock-change-set :id "v67.abc123")+ (mock-change-set :id "v67.def456")))))+ (testing "v66 and earlier versioned changesets are still allowed"+ (is (= :ok (validate-file (io/file "migrations/066/20261101_something.yaml") (mock-change-set :id "v66.abc123"))))+ (is (= :ok (validate-file (io/file "migrations/065/20260911_glossary_entity_id.yaml")+ (mock-change-set :id "v65.2026-09-11T12:00:00"))))+ (is (= :ok (validate-file (io/file "056_update_migrations.yaml") (mock-change-set :id "v56.2024-01-01T10:30:00")))))+ (testing "year-based directories are not versioned"+ (is (= :ok (validate-file (io/file "migrations/2026/20261101_something.yaml")+ (mock-change-set :id "something_versionless"))))))++(deftest ^:parallel versioned-and-version-less-layouts-do-not-mix-test+ (testing "a version-less id may not sneak into the versioned 065/ file, nor a v65 id into 2026/"+ (is (thrown? clojure.lang.ExceptionInfo+ (validate-file (io/file "migrations/065/20261012_glossary_entity_id.yaml")+ (mock-change-set :id "glossary_entity_id"))))+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"look like dates or timestamps"+ (validate-file (io/file "migrations/2026/20261012_glossary_entity_id.yaml")+ (mock-change-set :id "v65.2026-10-12T12:00:00"))))))++(deftest ^:parallel year-dir-allows-version-less-ids-test+ (testing "year-based directories allow version-less changeset IDs"+ (is (= :ok+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (mock-change-set :id "aeiagus09e"))))))++(deftest ^:parallel versioned-dirs-reject-version-less-ids-test+ (testing "version-numbered directories still reject version-less changeset IDs"+ (is (thrown?+ clojure.lang.ExceptionInfo+ (validate-file (io/file "migrations/060/20260616_workspaces.yaml")+ (mock-change-set :id "aeiagus09e"))))+ (testing "but still accept properly prefixed IDs"+ (is (= :ok+ (validate-file (io/file "migrations/060/20260616_workspaces.yaml")+ (mock-change-set :id "v60.aeiagus09e")))))))++(deftest ^:parallel year-dir-accepts-valid-version-less-ids-test+ (testing "year-based directories accept lowercase alphanumeric/underscore IDs with at least one letter"+ (doseq [good-id ["aeiagus09e" "workspaces_init" "a1234567"]]+ (is (= :ok+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (mock-change-set :id good-id)))+ good-id))))++(deftest ^:parallel year-dir-rejects-invalid-version-less-ids-test+ (testing "year-based directories reject invalid version-less IDs"+ (doseq [bad-id ["foo-bar" ; dash+ "v60.aeiagus09e" ; version-prefixed+ "foo.bar" ; dot+ "20260703" ; all digits: looks like a pre-4.2 changeset id to decide-liquibase-file+ "12345" ; all digits+ "v60abc" ; leading v<digit>: pollutes old binaries' `id LIKE 'v%'` version scans+ "view_metrics" ; any leading v: old binaries' max `id LIKE 'v%'` scan would pick it+ "Foo_bar"]] ; uppercase+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"invalid changeset IDs"+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (mock-change-set :id bad-id)))+ bad-id))))++(deftest ^:parallel year-dir-rejects-non-string-ids-test+ (testing "an unquoted YAML id that parses as a number or boolean gets the validation error, not a ClassCastException"+ (doseq [bad-id [20260703 true]]+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"invalid changeset IDs"+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (mock-change-set :id bad-id)))+ (pr-str bad-id)))))++(deftest ^:parallel year-dir-rejects-date-like-ids-test+ (testing "year-based directories reject the ISO date/timestamp ids the versioned changesets used"+ (doseq [bad-id ["2026-02-09T12:00:00" ; the old `vNN.` id body+ "2026-02-09t12:00:00"+ "2026-02-09"+ "widgets_2026-02-09"]]+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"look like dates or timestamps"+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (mock-change-set :id bad-id)))+ bad-id))+ (testing "digits on their own are fine"+ (doseq [good-id ["widgets_v2" "a1234567" "phase_2026" "table_1_2_3" "add_widgets_20260918"]]+ (is (= :ok+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (mock-change-set :id good-id)))+ good-id)))))++(deftest ^:parallel year-dir-requires-rollback-test+ (testing "year-based directory migrations still require rollback for non-auto-rollback change types"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"Rollback is required but not present\."+ (validate-file (io/file "migrations/2026/20260616_workspaces.yaml")+ (update (mock-change-set :id "aeiagus09e" :changes [{:sql {:sql "select 1"}}])+ :changeSet dissoc :rollback))))))+(deftest ^:parallel year-dir-prevent-bare-types-test+ ;; Regression: the bare-type checks filter changesets through changeset-version+id, which only parses `vNN.` ids --+ ;; version-less year-dir ids returned nil and were silently EXEMPT from the boolean/datetime checks.+ (let [year-file (io/file "migrations/2026/20260616_workspaces.yaml")]+ (testing "year-dir migrations may use the parameterized types"+ (is (= :ok+ (validate-file year-file+ (mock-change-set+ :id "aeiagus09e"+ :changes [(mock-add-column-changes+ :columns [(mock-column :type "${boolean.type}")])]))))+ (is (= :ok+ (validate-file year-file+ (mock-change-set+ :id "aeiagus09e"+ :changes [(mock-add-column-changes+ :columns [(mock-column :type "${timestamp_type}")])])))))+ (testing "year-dir migrations cannot add bare boolean columns"+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"uses invalid types"+ (validate-file year-file+ (mock-change-set+ :id "aeiagus09e"+ :changes [(mock-add-column-changes+ :columns [(mock-column :type "boolean")])])))))+ (testing "year-dir migrations cannot add bare datetime/timestamp columns"+ (doseq [bad-type ["datetime" "timestamp" "timestamp without time zone"]]+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo+ #"uses invalid types"+ (validate-file year-file+ (mock-change-set+ :id "aeiagus09e"+ :changes [(mock-add-column-changes+ :columns [(mock-column :type bad-type)])])))+ bad-type)))))+(deftest ^:parallel prevent-text-types-test(testing "should allow \"${text.type}\" columns from being added"(is (= :ok@@ -431,6 +580,16 @@:type "timestamp with time zone"}}]}}]:rollback nil))))))+(deftest ^:parallel require-no-run-always-test+ (testing "runAlways is rejected"+ (is-thrown-with-error-info?+ "Migration 'v49.2024-01-01T10:30:00' uses runAlways, which Metabase does not support: every boot would run it as a new migration, and with MB_DB_AUTOMIGRATE=false Metabase would never start. See require-no-run-always in bin/lint-migrations-file."+ {:id "v49.2024-01-01T10:30:00"}+ (validate (mock-change-set :runAlways true))))+ (testing "runAlways: false and runOnChange pass"+ (is (= :ok (validate (mock-change-set :runAlways false))))+ (is (= :ok (validate (mock-change-set :runOnChange true))))))+(deftest ^:parallel require-table-scoped-existence-preconditions-test(testing "foreignKeyConstraintExists without foreignKeyTableName is rejected"(is-thrown-with-error-info?diff --git a/deps.edn b/deps.ednindex 6165986e2fa..5a2afa00ac4 100644--- a/deps.edn+++ b/deps.edn@@ -884,13 +884,14 @@:extra-paths ["dev/src"]:main-opts ["-m" "dev.liquibase"]}- ;; Migrate CLI:- ;; clojure -M:migrate <command>+ ;; Migrate CLI (combine with :dev -- dev/src/user.clj needs the :dev classpath):+ ;; clojure -M:dev:migrate <command>;; E.g.- ;; clojure -M:migrate up ;; migrate up to the latest- ;; clojure -M:migrate rollback count 2 ;; rollback 2 migrations- ;; clojure -M:migrate rollback id "v40.00.001" ;; rollback to a specific migration with id- ;; clojure -M:migrate status ;; print the latest migration id+ ;; clojure -M:dev:migrate up ;; migrate up to the latest+ ;; clojure -M:dev:migrate rollback last-deployment ;; roll back the last migrate run+ ;; clojure -M:dev:migrate rollback deployment <id> ;; roll back everything after that deployment_id+ ;; clojure -M:dev:migrate rollback count 2 ;; raw Liquibase rollback of 2 changesets+ ;; clojure -M:dev:migrate status ;; print the latest migration id:migrate{:extra-deps {io.github.camsaul/humane-are {:mvn/version "1.0.2"}}:extra-paths ["dev/src"]diff --git a/dev/src/dev/migrate.clj b/dev/src/dev/migrate.cljindex 0f7ee0450c6..19b481ce8ef 100644--- a/dev/src/dev/migrate.clj+++ b/dev/src/dev/migrate.clj@@ -4,9 +4,10 @@[clojure.string :as str][metabase.app-db.core :as mdb][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.rollback :as rollback]+ [metabase.app-db.liquibase.versions :as versions][metabase.util.malli :as mu]- [toucan2.core :as t2]- [toucan2.honeysql2 :as t2.honeysql])+ [toucan2.core :as t2])(:import(liquibase Contexts LabelExpression Liquibase RuntimeEnvironment)(liquibase.change Change)@@ -36,13 +37,20 @@:limit 1})))(defn migrate!- "Run migrations for the Metabase application database. Possible directions are `:up` (default), `:force`, `:down`, and- `:release-locks`. When migrating `:down` pass along a version to migrate to (44+)."+ "Run migrations for the Metabase application database. Possible directions are `:up` (default), `:force`, `:print`+ and `:release-locks` -- what `java -jar metabase.jar migrate <direction>` does.++ There is deliberately no `:down` here: the release `migrate down` steps back one recorded Metabase *major*, which+ means nothing for development deployments (they all record [[versions/dev-version]]). Roll back by deployment with+ [[rollback!]] instead."([](migrate! :up))- ;; do we really use this in dev?- ([direction & [version]]- (mdb/migrate! (mdb/data-source) direction version)+ ([direction]+ (when (#{:down :down-force "down" "down-force"} direction)+ (throw (ex-info (str "migrate! does not roll back. Use (rollback! :last-deployment) to undo the last migrate! run, "+ "or (rollback! :deployment <id>) to roll back everything after a deployment.")+ {:direction direction})))+ (mdb/migrate! (mdb/data-source) direction)(println (format "Migrated %s. Latest migration: %s" (name direction) (latest-migration)))))(defn- rollback-n-migrations!@@ -73,22 +81,6 @@(string? x)parse-long))-(defn- last-deployment- []- (binding [t2.honeysql/*options* (assoc t2.honeysql/*options*- :quoted false)]- (if (= 1 (:count (t2/query-one {:select [[[:count [:distinct :deployment_id]] :count]]- :from [databasechangelog-name]- :limit 1})))- 0 ;; don't rollback if there was just one deployment of everything- (:count (t2/query-one {:select [[:%count.* :count]]- :from [databasechangelog-name]- :where [:= :deployment_id ^:allow-subquery- {:select [:deployment_id]- :from [databasechangelog-name]- :order-by [[:orderexecuted :desc]]- :limit 1}]})))))-(defn reset-checksums)]@@ -101,30 +93,68 @@(.changeLogSync liquibase (Contexts.) (LabelExpression.)))))(println "Reset checksums"))+(defn- rollback-to-deployment!+ "Roll back everything that ran after `boundary-deployment-id` (nil: nothing to do), in its own transaction, keeping+ `databasechangelog_version` and the legacy-version-tracking marker in step. Returns the boundary deployment id."+ [boundary-deployment-id]+ (when boundary-deployment-id+ (with-open [conn (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (.setAutoCommit conn false)+ (liquibase/with-liquibase [liquibase conn]+ (try+ (versions/ensure-version-tracking! conn (.getDatabase liquibase))+ (rollback/rollback-to-deployment! conn liquibase boundary-deployment-id)+ (.commit conn)+ (catch Throwable e+ (.rollback conn)+ (throw e))))))+ (println (if boundary-deployment-id+ (format "Rolled back to deployment %s. Latest migration: %s" boundary-deployment-id (latest-migration))+ "No earlier deployment to roll back to; nothing to do."))+ boundary-deployment-id)++(defn rollback-last-deployment!+ "Roll back the last `migrate!` run: everything that ran after the second-newest deployment, keeping+ `databasechangelog_version` and the legacy-version-tracking marker in step. Returns the `deployment_id` rolled back+ to, or nil when there was no earlier deployment (nothing is touched then). Same as `(rollback! :last-deployment)`."+ []+ (rollback-to-deployment!+ (with-open [conn (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (liquibase/with-liquibase [liquibase conn]+ (let [database (.getDatabase liquibase)]+ (versions/ensure-version-tracking! conn database)+ (versions/previous-deployment-id conn database))))))+(mu/defn rollback!- "Rollback helper, can take a number of migrations to rollback or a specific migration ID(inclusive) or last-deployment.+ "Rollback helper. Deployment-based rollbacks are the ones to reach for in development -- every `migrate!` run is+ its own Liquibase `deployment_id`, whatever Metabase version (real or [[versions/dev-version]]) it recorded:- ;; Rollback 2 migrations:- (rollback! :count 2)+ ;; Roll back the last migration run (everything after the second-newest deployment); same as+ ;; [[rollback-last-deployment!]]:+ (rollback! :last-deployment)- ;; Rollback last migration run:- ;; (rollback! :last-deployment)+ ;; Roll back everything that ran after a given deployment_id (see `SELECT * FROM databasechangelog_version`):+ (rollback! :deployment \"9673292410\")- ;; rollback to \"v50.2024-03-18T16:00:00\" (inclusive)- (rollback! :id \"v50.2024-03-18T16:00:00\")"- ([k :- [:enum :last-deployment "last-deployment"]]- (let [n (case (keyword k)- :last-deployment (last-deployment))]- (rollback-n-migrations! n)- (println (format "Rollbacked %d migrations. Latest migration: %s" n (latest-migration)))))+ Both keep `databasechangelog_version` and the legacy-version-tracking marker in step, exactly like the release+ `migrate down` does for majors.- ([k :- [:enum :id :count "id" "count"]+ ;; Raw Liquibase rollbacks by changeset position -- they do NOT update `databasechangelog_version` or remove a+ ;; deployment's marker row, so a later `migrate! :down` may misjudge its boundary; partial-rollback use only:+ (rollback! :count 2)+ (rollback! :id \"v50.2024-03-18T16:00:00\") ; inclusive"+ ([_k :- [:enum :last-deployment "last-deployment"]]+ (rollback-last-deployment!))++ ([k :- [:enum :id :count :deployment "id" "count" "deployment"]target :- [:or :int :string]]- (let [n (case (keyword k)- :id (migration-since target)- :count (maybe-parse-long target))]- (rollback-n-migrations! n)- (println (format "Rollbacked %d migrations. Latest migration: %s" n (latest-migration))))))+ (if (= :deployment (keyword k))+ (rollback-to-deployment! (str target))+ (let [n (case (keyword k)+ :id (migration-since target)+ :count (maybe-parse-long target))]+ (rollback-n-migrations! n)+ (println (format "Rollbacked %d migrations. Latest migration: %s" n (latest-migration)))))))(defn migration-status"Print the latest migration ID."@@ -135,12 +165,15 @@"Migrations helpersUsage:- clojure -M:migrate up ;; migrate up to the latest- clojure -M:migrate rollback count 2 ;; rollback 2 migrations- clojure -M:migrate rollback id \"v40.00.001\" ;; rollback to a specific migration with id- clojure -M:migrate rollback last-deployment ;; rollback the last deployment- clojure -M:migrate status ;; print the latest migration id- clojure -M:migrate reset-checksums. ;; sets the checksums to what they would be if migrated from the current changelog"+ clojure -M:dev:migrate up ;; migrate up to the latest+ clojure -M:dev:migrate rollback last-deployment ;; rollback the last deployment (last migrate run)+ clojure -M:dev:migrate rollback deployment <id> ;; rollback everything after that deployment_id+ clojure -M:dev:migrate rollback count 2 ;; raw Liquibase rollback of 2 changesets+ clojure -M:dev:migrate rollback id \"v40.00.001\" ;; raw Liquibase rollback to a specific changeset id+ clojure -M:dev:migrate status ;; print the latest migration id+ clojure -M:dev:migrate reset-checksums ;; sets the checksums to what they would be if migrated from the current changelog++ (`:dev` is needed: `dev/src/user.clj` requires the dev classpath.)"[& args](let [[cmd & migration-args] args]diff --git a/dev/test/dev/migrate_test.clj b/dev/test/dev/migrate_test.cljindex 9291b50c859..d2a6697a7e6 100644--- a/dev/test/dev/migrate_test.clj+++ b/dev/test/dev/migrate_test.clj@@ -1,7 +1,65 @@(ns dev.migrate-test(:require+ [clojure.java.jdbc :as jdbc][clojure.test :refer :all]- [metabase.dev.migrate :as dev.migrate]))+ [dev.migrate :as dev.migrate]+ [metabase.app-db.core :as mdb]+ [metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.versions :as versions]+ [metabase.app-db.test-util :as mdb.test-util]+ [metabase.config.core :as config]+ [metabase.test :as mt]))++(deftest rollback-last-deployment-uses-version-tracking-test+ (testing "rollback! :last-deployment goes through the normal migrate down, so version rows and the marker are kept in step"+ (mt/with-temp-empty-app-db [_conn :h2]+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "vLOCAL_DEV")]+ (let [ct (liquibase/changelog-table-name (mdb/data-source))+ applied? (fn [id] (boolean (seq (jdbc/query {:datasource (mdb/data-source)}+ [(format "SELECT 1 FROM %s WHERE id = ?" ct) id]))))+ versions (fn [] (mapv :metabase_version+ (jdbc/query {:datasource (mdb/data-source)}+ [(format "SELECT metabase_version FROM %s ORDER BY id" (versions/versions-table-name))])))+ markers (fn [] (mapv :id (jdbc/query {:datasource (mdb/data-source)}+ [(format "SELECT id FROM %s WHERE id LIKE '%%legacy-version-tracking'" ct)])))+ deployment-of (fn [id] (:deployment_id (first (jdbc/query {:datasource (mdb/data-source)}+ [(format "SELECT deployment_id FROM %s WHERE id = ?" ct) id]))))]+ (with-redefs [liquibase/changelog-file "versionless-dev-run1.yaml"]+ (dev.migrate/migrate!))+ (mdb.test-util/age-changelog-rows! {:datasource (mdb/data-source)} ct 5)+ (with-redefs [liquibase/changelog-file "versionless-dev-run2.yaml"]+ (dev.migrate/migrate!)+ (is (true? (applied? "dev_run_b")))+ (is (= [versions/dev-version versions/dev-version] (versions)) "two dev deployments")+ (is (= ["v9999.legacy-version-tracking"] (markers)))+ (is (= (deployment-of "dev_run_a") (dev.migrate/rollback-last-deployment!))+ "returns the deployment it rolled back to")+ (is (false? (applied? "dev_run_b")) "the newest deployment's changeset is reversed")+ (is (true? (applied? "dev_run_a")) "the earlier deployment survives")+ (is (= [versions/dev-version] (versions)) "the rolled-back deployment's version row is gone too")+ (is (= ["v9999.legacy-version-tracking"] (markers)) "the surviving deployment's marker is kept")+ (testing "migrate! has no :down -- rollbacks are rollback!'s job"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"rollback! :last-deployment"+ (dev.migrate/migrate! :down)))+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"rollback! :last-deployment"+ (dev.migrate/migrate! :down-force)))+ (is (true? (applied? "dev_run_a")) "and nothing was touched"))+ (testing "with a single deployment left there is nothing earlier: clean no-op"+ (is (nil? (dev.migrate/rollback-last-deployment!)))+ (is (true? (applied? "dev_run_a"))))+ (testing "rollback! :last-deployment is the same thing"+ (with-redefs [liquibase/changelog-file "versionless-dev-run2.yaml"]+ (dev.migrate/migrate!))+ (is (= (deployment-of "dev_run_a") (dev.migrate/rollback! :last-deployment)))+ (is (false? (applied? "dev_run_b"))))+ (testing "rollback! :deployment <id> rolls back everything after that deployment"+ (with-redefs [liquibase/changelog-file "versionless-dev-run2.yaml"]+ (dev.migrate/migrate!))+ (is (true? (applied? "dev_run_b")))+ (let [base-dep (deployment-of "dev_run_a")]+ (dev.migrate/rollback! :deployment base-dep)+ (is (false? (applied? "dev_run_b")))+ (is (true? (applied? "dev_run_a")))))))))))(deftest migration-sql-by-id-test(doseq [[id test] {"v50.2024-05-08T09:00:01"diff --git a/enterprise/backend/src/metabase_enterprise/audit_app/db.clj b/enterprise/backend/src/metabase_enterprise/audit_app/db.cljindex 49c9b2fabcc..44df390d9e1 100644--- a/enterprise/backend/src/metabase_enterprise/audit_app/db.clj+++ b/enterprise/backend/src/metabase_enterprise/audit_app/db.clj@@ -81,6 +81,8 @@;; Exclude DATABASECHANGELOG, DATABASECHANGELOGLOCK, and QRTZ_* tables, they are not metabase managed[:not= :table.name "DATABASECHANGELOG"][:not= :table.name "DATABASECHANGELOGLOCK"] ;; new instances do not get this file, but existing instances may have it+ [:not= :table.name "DATABASECHANGELOG_VERSION"]+ [:not= :table.name "DATABASECHANGELOG_FIRST_RUN"][:not [:like :table.name "QRTZ_%"]][:not [:exists ^:allow-subquery {:select [1]:from [(warehouse-schema-overlay/table-query {:alias :self_table, :user-settings? false})]@@ -109,6 +111,8 @@[:= :table.id :field.table_id]]:where [:and [:= :table.db_id database-id][:not= :table.name "DATABASECHANGELOG"]+ [:not= :table.name "DATABASECHANGELOG_VERSION"]+ [:not= :table.name "DATABASECHANGELOG_FIRST_RUN"][:not [:like :table.name "QRTZ_%"]][:not [:exists ^:allow-subquery {:select [1]:from [(warehouse-schema-overlay/field-query {:alias :self_field})]diff --git a/mage/src/mage/new_migration.clj b/mage/src/mage/new_migration.cljindex 586634c8cfa..575e0b55eba 100644--- a/mage/src/mage/new_migration.clj+++ b/mage/src/mage/new_migration.clj@@ -1,9 +1,7 @@(ns mage.new-migration- "Mage task for creating new directory-based migration files (v60+).-- Each migration file is a YAML file inside a version directory,- e.g., `resources/migrations/060/20260905_mq_indexes.yaml`."+ "Mage task for creating a new migration file."(:require+ [babashka.fs :as fs][clojure.java.io :as io][mage.util :as u])(:import@@ -23,53 +21,74 @@(catch Exception _"author")))-(defn- utc-date []- (.format (DateTimeFormatter/ofPattern "yyyyMMdd")- (ZonedDateTime/now ZoneOffset/UTC)))+(defn- migration-template [id author]+ (str "## TODO: quick description of what feature/bug this migration is for\n"+ "\n"+ "databaseChangeLog:\n"+ " - changeSet:\n"+ " id: " id "\n"+ " author: " author "\n"+ " comment: TODO\n"+ " changes:\n"+ " - TODO\n"))++(def ^:private first-chars+ "A version-less id must have a letter and must not start with `v`, so it starts with one of these."+ "abcdefghijklmnopqrstuwxyz")(def ^:private alphanumeric "abcdefghijklmnopqrstuvwxyz0123456789")(defn- random-id- "Generates a random 6-character lowercase alphanumeric string."- []- (let [rng (SecureRandom.)]- (apply str (repeatedly 6 #(nth alphanumeric (.nextInt rng (count alphanumeric)))))))+ "A random 6-character version-less changeset id not in `existing-ids`."+ [existing-ids]+ (let [rng (SecureRandom.)+ pick (fn [^String chars] (.charAt chars (.nextInt rng (count chars))))]+ (->> (repeatedly #(apply str (pick first-chars) (repeatedly 5 (fn [] (pick alphanumeric)))))+ (remove (set existing-ids))+ first)))++(defn migration-file+ "The `{:path :id :content}` of a new migration file. `:existing-ids` are the changeset ids already in the repository."+ [{:keys [description author ^ZonedDateTime now existing-ids]}]+ (when-not (re-matches valid-description-re description)+ (throw (ex-info (str "description must be lowercase alphanumeric or underscores, got: \"" description "\"")+ {:description description})))+ (let [date (.format (DateTimeFormatter/ofPattern "yyyyMMdd") now)+ id (random-id existing-ids)]+ {:path (str "resources/migrations/" (.getYear now) "/" date "_" description ".yaml")+ :id id+ :content (migration-template id author)}))-(defn- migration-template [version author]- (let [id (random-id)]- (str "## TODO: quick description of what feature/bug this migration is for\n"- "\n"- "databaseChangeLog:\n"- " - changeSet:\n"- " id: v" version "." id "\n"- " author: " author "\n"- " comment: TODO\n"- " changes:\n"- " - TODO\n")))+(defn- existing-changeset-ids []+ (into #{}+ (mapcat (fn [file]+ (map second (re-seq #"(?m)^\s*-?\s*id:\s*(\S+)\s*$" (slurp (str file))))))+ (fs/glob (io/file u/project-root-directory "resources" "migrations") "**.yaml")))(defn new-migration- "Creates a new directory-based migration file for the given version number.+ "Creates a new migration file.- Usage: `./bin/mage new-migration 60 mq_indexes`"+ Usage: `./bin/mage new-migration mq_indexes`"[{:keys [arguments]}]- (let [version (first arguments)- description (second arguments)- _ (when-not (and version description)- (binding [*out* *err*]- (println "Usage: mage new-migration <version> <description>")- (println " e.g., mage new-migration 60 mq_indexes"))- (u/exit 1))- _ (when-not (re-matches valid-description-re description)- (binding [*out* *err*]- (println (str "Error: description must be lowercase alphanumeric or underscores, got: \"" description "\""))- (println " e.g., mq_indexes, add_user_table, fix_fk"))- (u/exit 1))- dir-name (format "%03d" version)- dir (io/file u/project-root-directory "resources" "migrations" dir-name)- date (utc-date)- filename (str date "_" description ".yaml")- file (io/file dir filename)- author (git-user-name)]- (.mkdirs dir)- (spit file (migration-template version author))- (println "Created" (str "resources/migrations/" dir-name "/" filename))))+ (when-not (= 1 (count arguments))+ (throw (ex-info (str "Usage: mage new-migration <description>. Migrations are version-less on every supported "+ "major, so there is no version argument.")+ {:arguments arguments})))+ (let [{:keys [path]+ :as migration} (try+ (migration-file {:description (first arguments)+ :author (git-user-name)+ :now (ZonedDateTime/now ZoneOffset/UTC)+ :existing-ids (existing-changeset-ids)})+ (catch clojure.lang.ExceptionInfo e+ (binding [*out* *err*]+ (println (str "Error: " (ex-message e))))+ (u/exit 1)))+ file (io/file u/project-root-directory path)]+ (when (.exists file)+ (binding [*out* *err*]+ (println "Error:" path "already exists"))+ (u/exit 1))+ (io/make-parents file)+ (spit file (:content migration))+ (println "Created" path)))diff --git a/mage/test/mage/core_test.clj b/mage/test/mage/core_test.cljindex 6fe46f3a712..f03b9438e40 100644--- a/mage/test/mage/core_test.clj+++ b/mage/test/mage/core_test.clj@@ -22,6 +22,7 @@[mage.merge-kondo-ratchets-test][mage.merge-yaml-migrations-test][mage.modules-test]+ [mage.new-migration-test][mage.openapi-diff-test][mage.project-tests-test][mage.quick-test-runner-test]diff --git a/mage/test/mage/new_migration_test.clj b/mage/test/mage/new_migration_test.cljnew file mode 100644index 00000000000..f38fc170476--- /dev/null+++ b/mage/test/mage/new_migration_test.clj@@ -0,0 +1,38 @@+(ns mage.new-migration-test+ (:require+ [clojure.string :as str]+ [clojure.test :refer [deftest is testing]]+ [lint-migrations-file]+ [mage.new-migration :as new-migration])+ (:import+ (java.time ZoneOffset ZonedDateTime)))++(set! *warn-on-reflection* true)++(def ^:private now (ZonedDateTime/of 2026 10 12 9 0 0 0 ZoneOffset/UTC))++(defn- migration-file [opts]+ (new-migration/migration-file (merge {:author "alice" :now now :existing-ids #{}} opts)))++(deftest version-less-migration-test+ (testing "the file goes in the current year's directory, named by the description, with a random version-less id"+ (let [{:keys [path id content]} (migration-file {:description "add_user_table"})]+ (is (= "resources/migrations/2026/20261012_add_user_table.yaml" path))+ (is (re-matches #"[a-uw-z][a-z0-9]{5}" id))+ (is (lint-migrations-file/valid-version-less-id? id))+ (is (str/includes? content (str " id: " id "\n")))+ (is (str/includes? content " author: alice\n"))))+ (testing "ids differ between files"+ (is (< 1 (count (into #{} (map (fn [_] (:id (migration-file {:description "x"})))) (range 20))))))+ (testing "an id already used in the repository is never generated"+ (let [taken (into #{} (map (fn [_] (:id (migration-file {:description "x"})))) (range 50))]+ (is (not (taken (:id (migration-file {:description "x" :existing-ids taken})))))))+ (testing "the description must be a lowercase snake_case name"+ (doseq [description ["Add-Table" "has space" ""]]+ (is (thrown? clojure.lang.ExceptionInfo (migration-file {:description description}))+ description))))++(deftest no-versioned-layout-test+ (testing "there is no versioned mode: every supported major takes version-less changesets"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"version-less"+ (new-migration/new-migration {:arguments [65 "something"]})))))diff --git a/src/metabase/app_db/custom_migrations.clj b/src/metabase/app_db/custom_migrations.cljindex 85ae55f336f..827e957170c 100644--- a/src/metabase/app_db/custom_migrations.clj+++ b/src/metabase/app_db/custom_migrations.clj@@ -950,7 +950,8 @@(->> (t2/query query)(map #(update-vals % (comp keyword lower-case-en)))(remove (fn [{:keys [table_name]}]- (or (#{:databasechangelog :databasechangeloglock} table_name)+ ;; databasechangelog_first_run copies the changelog's dateexecuted, so it keeps the same type+ (or (#{:databasechangelog :databasechangeloglock :databasechangelog_first_run} table_name);; excludes views(str/starts-with? (name table_name) "v_"))))(map #(update % :is_nullable (fn [x] (= :yes x))))diff --git a/src/metabase/app_db/db.clj b/src/metabase/app_db/db.cljindex 86fbcb24bdb..68710272305 100644--- a/src/metabase/app_db/db.clj+++ b/src/metabase/app_db/db.clj@@ -6,7 +6,6 @@first, and while rows are being re-encrypted a setting row can be plaintext under a key, or ciphertext under a keynot yet in effect, which that model's strict read rejects."(:require- [honey.sql :as sql][metabase.util.honey-sql-2 :as h2x][metabase.util.malli :as mu][metabase.util.malli.schema :as ms]@@ -30,16 +29,6 @@:mysql "DATABASECHANGELOG")](t2/query-one [(format "select * from %s where id = ?" table-name) changelog-id])))-(mu/defn changelog-ids- "The ids among `changelog-ids` still present in the Liquibase changelog table `changelog-table-name`, read on- `conn`."- [conn :- (ms/InstanceOfClass java.sql.Connection)- changelog-table-name :- :string- changelog-ids :- [:set :string]]- (map :id (t2/query conn (sql/format {:select [:id]- :from [(keyword changelog-table-name)]- :where [:in :id changelog-ids]}))))-;;; ------------------------------------------------ Settings ------------------------------------------------(def ^:private unmigrated-settings-wherediff --git a/src/metabase/app_db/liquibase.clj b/src/metabase/app_db/liquibase.cljindex 62c1e171c58..c60bf174fdd 100644--- a/src/metabase/app_db/liquibase.clj+++ b/src/metabase/app_db/liquibase.clj@@ -8,6 +8,7 @@[metabase.app-db.db :as app-db.db][metabase.app-db.liquibase.h2 :as liquibase.h2][metabase.app-db.liquibase.mysql :as liquibase.mysql]+ [metabase.app-db.liquibase.versions :as versions][metabase.classloader.core :as classloader][metabase.config.core :as config][metabase.util :as u]@@ -19,14 +20,13 @@(:import(java.io StringWriter)(java.sql Connection)- (java.util ArrayList List Map)+ (java.util List Map)(javax.sql DataSource)(liquibase Contexts LabelExpression Liquibase RuntimeEnvironment Scope Scope$Attr Scope$ScopedRunner UpdateSummaryOutputEnum)(liquibase.change.custom CustomChangeWrapper)- (liquibase.changelog ChangeLogIterator ChangeSet ChangeSet$ExecType)- (liquibase.changelog.filter AlreadyRanChangeSetFilter ChangeSetFilter ChangeSetFilterResult DbmsChangeSetFilter IgnoreChangeSetFilter)+ (liquibase.changelog ChangeLogIterator ChangeSet ChangeSet$ExecType ChangeSet$RunStatus)+ (liquibase.changelog.filter ChangeSetFilter)(liquibase.changelog.visitor AbstractChangeExecListener ChangeExecListener UpdateVisitor)- (liquibase.command.core AbstractRollbackCommandStep)(liquibase.database Database DatabaseFactory ObjectQuotingStrategy)(liquibase.database.jvm JdbcConnection)(liquibase.exception LockException)@@ -67,32 +67,30 @@:doc "Liquibase setting used for upgrading instances running version < 45."}^String changelog-legacy-file "liquibase_legacy.yaml")-(def ^{:private true- :doc "Liquibase setting used for upgrading a fresh instance or instances running version >= 45."}+(def ^{:doc "Liquibase setting used for upgrading a fresh instance or instances running version >= 45."}^String changelog-file "liquibase.yaml")-(defn table-exists?- "Check if a table exists."- [table-name ^Connection conn]- (-> (.getMetaData conn)- (.getTables nil nil table-name (u/varargs String ["TABLE"]))- jdbc/metadata-query- seq- boolean))-(defn- fresh-install?[^Connection conn ^Database database]- (not (table-exists? (.getDatabaseChangeLogTableName database) conn)))+ (not (versions/table-exists? (.getDatabaseChangeLogTableName database) conn)))++(defn- year-directory-migration?+ "Whether `filename` is a version-less migration living in a year-based directory,+ e.g. `migrations/2026/20260703_workspaces.yaml`."+ [filename]+ (boolean (some->> filename (re-find #"(?:^|[/\\])\d{4}[/\\]"))))(defn- decide-liquibase-file[^Connection conn ^Database database](if (fresh-install? conn database)changelog-file- (let [latest-migration (->> (jdbc/query {:connection conn}- [(format "select id from %s order by dateexecuted desc limit 1"- (.getDatabaseChangeLogTableName database))])- first- :id)]+ (let [{latest-migration :id, latest-filename :filename}+ ;; `orderexecuted` breaks ties between rows sharing a `dateexecuted` (e.g. MySQL second precision)+ (first (jdbc/query {:connection conn}+ [(format "select id, filename from %s order by dateexecuted desc, orderexecuted desc limit 1"+ (.getDatabaseChangeLogTableName database))]))+ ;; major version parsed from a legacy "vNN.*" id, or nil for version-less (year-directory) ids+ major (some-> (re-find #"^v(\d+)\." (str latest-migration)) second parse-long)](cond(nil? latest-migration)changelog-file@@ -101,11 +99,15 @@(= latest-migration "v00.00-000")changelog-file- ;; pre 42- (not (str/starts-with? latest-migration "v"))+ (year-directory-migration? latest-filename)+ changelog-file++ ;; Pre-4.2 installs used purely-numeric changeset ids (e.g. "1" .. "316") and need the legacy changelog.+ (re-matches #"\d+" latest-migration)changelog-legacy-file- (< (->> latest-migration (re-find #"v(\d+)\..*") second parse-long) 45)+ ;; versioned ids below 45 need the legacy changelog+ (and major (< major 45))changelog-legacy-file:else@@ -176,18 +178,6 @@[app-db changelog-id](app-db.db/changelog-by-id (:db-type app-db) changelog-id))-(defn migrations-sql- "Return a string of SQL containing the DDL statements needed to perform unrun `liquibase` migrations, custom migrations will be ignored."- ^String [^Liquibase liquibase]- ;; calling update on custom migrations will execute them, so we ignore it and generates- ;; sql for SQL migrations only- (doseq [^ChangeSet change (.listUnrunChangeSets liquibase nil nil)]- (when (instance? CustomChangeWrapper (first (.getChanges change)))- (.setIgnore change true)))- (let [writer (StringWriter.)]- (.update liquibase "" writer)- (.toString writer)))-(defn unrun-migrations"Returns a list of unrun migrations.@@ -316,7 +306,14 @@;; It's possible that the lock was accidentally released by an operation, or force released by;; another process, so it's useful for debugging to know whether we were still within a locked;; scope.- :lock-depth *lock-depth*}))))+ :lock-depth *lock-depth*}))))++(defn- fresh-deployment-id+ "A new Liquibase-style deployment id: the last 10 digits of the current epoch millis (the same format+ `liquibase.Scope/generateDeploymentId` uses, fitting the `varchar(10)` DEPLOYMENT_ID column)."+ ^String []+ (let [s (str (System/currentTimeMillis))]+ (subs s (max 0 (- (count s) 10)))))(defn run-in-scope-locked"Run function `f` in a scope on the Liquibase instance `liquibase`.@@ -333,8 +330,11 @@(throw (LockException. "Attempted to take a Liquibase lock, but we already are holding it."))))(let [database (.getDatabase liquibase)lock-service (lock-service liquibase)- scope-objects {(.name Scope$Attr/database) database- (.name Scope$Attr/resourceAccessor) (.getResourceAccessor liquibase)}]+ scope-objects (cond-> {(.name Scope$Attr/database) database+ (.name Scope$Attr/resourceAccessor) (.getResourceAccessor liquibase)}+ ;; a re-entrant (prod-only) nested scope must not change the deployment id mid-run+ (not (holding-lock? liquibase))+ (assoc (.name Scope$Attr/deploymentId) (fresh-deployment-id)))](Scope/child ^Map scope-objects(reify Scope$ScopedRunner(run [_]@@ -359,8 +359,36 @@[liquibase & body]`(run-in-scope-locked ~liquibase (fn [] ~@body)))+(defn migrations-sql+ "Return a string of SQL containing the DDL statements needed to perform unrun `liquibase` migrations, custom+ migrations will be ignored. Starts with statements recording where the changesets it re-runs first ran (see+ [[versions/first-runs-sql]]), and ends with statements that record the upgrade in `databasechangelog_version` (and+ the legacy version-tracking marker) -- see [[versions/version-tracking-sql]] -- since the listeners that normally+ write those only run when Metabase itself executes the migrations."+ ^String [^Liquibase liquibase]+ ;; calling update on custom migrations will execute them, so we ignore it and generates+ ;; sql for SQL migrations only+ (let [unrun (.listUnrunChangeSets liquibase nil nil)+ writer (StringWriter.)+ database (.getDatabase liquibase)+ conn (.. database getConnection getUnderlyingConnection)+ ;; read before the update: it describes the changelog rows the printed SQL is about to move+ first-runs (versions/first-runs-sql conn database unrun)]+ (doseq [^ChangeSet change unrun]+ (when (instance? CustomChangeWrapper (first (.getChanges change)))+ (.setIgnore change true)))+ (.update liquibase "" writer)+ (str first-runs+ (.toString writer)+ (versions/version-tracking-sql conn+ database+ (->> (.getChangeSets (.getDatabaseChangeLog liquibase))+ (keep (fn [^ChangeSet cs] (some-> (re-find #"^v(\d+)\." (.getId cs)) second parse-long)))+ (reduce max 0))))))+(defn migrate-up-if-needed!- "Run any unrun `liquibase` migrations, if needed."+ "Run any unrun `liquibase` migrations, if needed, recording the Metabase version that ran them (see+ [[metabase.app-db.liquibase.versions]]). Expects [[versions/ensure-version-tracking!]] to have run."[^Liquibase liquibase ^DataSource data-source](log/info "Checking if Database has unrun migrations...")(if (seq (unrun-migrations data-source))@@ -373,14 +401,25 @@unrun-migrations-count (count to-run-migrations)](if (pos? unrun-migrations-count)(let [^Contexts contexts nil- timer (u/start-timer)]+ timer (u/start-timer)+ database (.getDatabase liquibase)](log/infof "Running %s migrations ..." unrun-migrations-count)(doseq [^ChangeSet change to-run-migrations](log/tracef "To run migration %s" (.getId change)))- (.update liquibase contexts)+ (versions/repair-legacy-rollback! (.. database getConnection getUnderlyingConnection)+ database+ to-run-migrations)+ (.setChangeExecListener liquibase (versions/recording-exec-listener database))+ (try+ (.update liquibase contexts)+ (finally+ (.setChangeExecListener liquibase nil)))+ (versions/record-legacy-version-tracking! database)(log/infof "Migration complete in %s" (u/format-milliseconds (u/since-ms timer))))(log/info "Migration lock cleared, but nothing to do here! Migrations were finished by another instance.")))))- (log/info "No unrun migrations found.")))+ (do+ (log/info "No unrun migrations found.")+ (versions/record-boot-version! (.getDatabase liquibase)))))(defn update-with-change-log"Run update with the change log instances in `liquibase`. Must be called within a scope holding the liquibase lock."@@ -388,7 +427,7 @@(update-with-change-log liquibase {}))([^Liquibase liquibase{:keys [^List change-set-filters exec-listener]- :or {change-set-filters []}}]+ :or {change-set-filters []}}](assert-locked liquibase)(let [change-log (.getDatabaseChangeLog liquibase)database (.getDatabase liquibase)@@ -417,38 +456,88 @@(.clearCheckSums liquibase)(with-scope-locked liquibase(when (seq (unrun-migrations data-source))- (let [change-log (.getDatabaseChangeLog liquibase)- fail-on-errors (mapv (fn [^ChangeSet change-set] [change-set (.getFailOnError change-set)])- (.getChangeSets change-log))- exec-listener (proxy [AbstractChangeExecListener] []- (willRun [^ChangeSet change-set _database-change-log _database _run-status]- (when (instance? ChangeSet change-set)- (log/infof "Start executing migration with id %s" (.getId change-set))))-- (runFailed [^ChangeSet _change-set _database-change-log _database ^Exception e]- (log/error (u/format-color 'red "[ERROR] %s" (.getMessage e))))-- (ran [change-set _database-change-log _database ^ChangeSet$ExecType exec-type]- (when (instance? ChangeSet change-set)- (condp = exec-type- ChangeSet$ExecType/EXECUTED- (log/info (u/format-color 'green "[SUCCESS]"))-- ChangeSet$ExecType/FAILED- (log/error (u/format-color 'red "[ERROR]"))-- (log/infof "[%s]" (.name exec-type))))))]+ (let [change-log (.getDatabaseChangeLog liquibase)+ active-deployment-version-recorded? (atom false)+ fail-on-errors (mapv (fn [^ChangeSet change-set] [change-set (.getFailOnError change-set)])+ (.getChangeSets change-log))+ exec-listener (proxy [AbstractChangeExecListener] []+ (willRun [change-set _database-change-log ^Database database run-status]+ (when (instance? ChangeSet change-set)+ (log/infof "Start executing migration with id %s" (.getId ^ChangeSet change-set))+ (when (not= ChangeSet$RunStatus/NOT_RAN run-status)+ (versions/record-first-run! database change-set))))++ (runFailed [^ChangeSet _change-set _database-change-log _database ^Exception e]+ (log/error (u/format-color 'red "[ERROR] %s" (.getMessage e))))++ (ran [change-set _database-change-log ^Database database ^ChangeSet$ExecType exec-type]+ (when (instance? ChangeSet change-set)+ (when (compare-and-set! active-deployment-version-recorded? false true)+ (versions/record-active-deployment-version! database))+ (condp = exec-type+ ChangeSet$ExecType/EXECUTED+ (log/info (u/format-color 'green "[SUCCESS]"))++ ChangeSet$ExecType/FAILED+ (log/error (u/format-color 'red "[ERROR]"))++ (log/infof "[%s]" (.name exec-type))))))](try(doseq [^ChangeSet change-set (.getChangeSets change-log)](.setFailOnError change-set false))(update-with-change-log liquibase {:exec-listener exec-listener})(finally(doseq [[^ChangeSet change-set fail-on-error?] fail-on-errors]- (.setFailOnError change-set fail-on-error?))))))))+ (.setFailOnError change-set fail-on-error?))))+ ;; same old-binary downgrade signal a normal upgrade leaves -- see [[migrate-up-if-needed!]]+ (versions/record-legacy-version-tracking! (.getDatabase liquibase))))))(def ^:private legacy-migrations-file "migrations/000_legacy_migrations.yaml")(def ^:private update001-migrations-file "migrations/001_update_migrations.yaml")+(defn repair-version-less-filenames!+ "Point version-less changelog rows back at the year-directory changelog file that defines them, after an older+ Metabase binary consolidated them into the legacy/001 file.++ Scenario: after upgrading, someone runs `java -jar <older metabase>.jar migrate down` (or any other `migrate`+ command) against the database. The older binary's `migrate!` consolidates changeset filenames *before* its+ newer-database check refuses, and the Liquibase lock release commits that. Its consolidation is an unguarded+ `WHEN ID < 'v45.00-001' THEN <legacy file>`, which catches every version-less id sorting before `v` -- most of them.+ Left in place, the rewritten rows no longer match their changesets, so a later `migrate down` from this version would+ clear their bookkeeping without reversing their DDL and the next upgrade would fail with 'already exists'. (Booting+ the older binary is safe: its downgrade check runs before consolidation.)++ Pre-4.2 numeric ids legitimately live in the legacy file and `vNN.` ids are never mis-consolidated, so only+ other ids are candidates. A candidate is left alone when this changelog has no matching `[author id]`, or has that+ identity in several files and therefore cannot identify the original filename safely."+ [^Connection conn ^Liquibase liquibase changelog-table]+ (let [stray (->> (jdbc/query {:connection conn}+ [(format "SELECT id, author FROM %s WHERE filename IN (?, ?) AND id NOT LIKE 'v%%'" changelog-table)+ legacy-migrations-file update001-migrations-file])+ (remove #(re-matches #"\d+" (:id %))))]+ (when (seq stray)+ (let [paths-of (reduce (fn [paths ^ChangeSet cs]+ (if (year-directory-migration? (.getFilePath cs))+ (update paths [(.getAuthor cs) (.getId cs)] (fnil conj #{}) (.getFilePath cs))+ paths))+ {}+ (.getChangeSets (.getDatabaseChangeLog liquibase)))]+ (doseq [{:keys [id author]} stray+ :let [paths (paths-of [author id])]]+ (cond+ (= 1 (count paths))+ (let [path (first paths)]+ (log/warnf "Restoring the changelog filename of version-less changeset %s to %s (an older Metabase version had consolidated it into the legacy changelog file)"+ id path)+ (jdbc/execute! {:connection conn}+ [(format "UPDATE %s SET filename = ? WHERE id = ? AND author = ?" changelog-table)+ path id author]))++ (< 1 (count paths))+ (log/warnf (str "Cannot safely restore the changelog filename of version-less changeset %s by author %s: "+ "that identity occurs in multiple files (%s)")+ id author (str/join ", " (sort paths)))))))))+(mu/defn consolidate-liquibase-changesets!"Consolidate all previous DB migrations so they come from single file.@@ -467,136 +556,19 @@(let [liquibase-table-name (changelog-table-name liquibase)conn-spec {:connection conn}](when-not (fresh-install? conn (.getDatabase ^Liquibase liquibase))- ;; Skip mutating the table if the filenames are already correct. It assumes we have never moved the boundary- ;; between the two files, i.e. that update-migrations still start from v45.- (when (->> (str "SELECT DISTINCT(FILENAME) AS filename FROM " liquibase-table-name)- (jdbc/query conn-spec)- (into #{} (map :filename))- (filter #(or (= % legacy-migrations-file)- (str/ends-with? % "update_migrations.yaml"))))- (log/info "Updating liquibase table to reflect consolidated changeset filenames")- (with-scope-locked liquibase- (jdbc/execute!- conn-spec- [(format "UPDATE %s SET FILENAME = CASE WHEN ID = ? THEN ? WHEN ID < ? THEN ? WHEN ID < ? THEN ? ELSE FILENAME END" liquibase-table-name)- "v00.00-000" update001-migrations-file- "v45.00-001" legacy-migrations-file- "v56.0000-00-00T00:00:00" update001-migrations-file]))))))--(def ^:private special-case-migrations #{"v56.2025-06-05T16:48:48" "v56.2025-05-19T16:48:48"})--(defn- handle-special-case-migrations- "This handles v56 migrations that were checked into the v55 branch to resolve an issue with- inadventently backported migrations in 55. We check if this or the bad backports are the most recent- available migration and explicitly return 55 as the available major version if so."- [s]- (when (contains? special-case-migrations s)- 55))--(defn- extract-numbers- "Returns contiguous integers parsed from string s"- [s]- (if-let [special-cased (handle-special-case-migrations s)]- [special-cased]- (map #(Integer/parseInt %) (re-seq #"\d+" s))))--(defn latest-available-major-version- "Get the latest version that Liquibase would apply if we ran migrations right now."- [^Liquibase liquibase]- (->> liquibase- (.getDatabaseChangeLog)- (.getChangeSets)- last- (#(.getId ^ChangeSet %))- extract-numbers- first))--(defn latest-applied-major-version- "Gets the latest version applied to the database."- [conn ^Database database]- (when-not (fresh-install? conn database)- (let [changeset-query (format "SELECT id FROM %s WHERE id LIKE 'v%%' ORDER BY id DESC LIMIT 1"- (.getDatabaseChangeLogTableName database))- changeset-id (last (map :id (jdbc/query {:connection conn} [changeset-query])))]- (some-> changeset-id extract-numbers first))))--(defn changesets-from-later-version- "Returns changeset IDs applied from versions later than `latest-available` up to `latest-applied`, ordered by execution date."- [conn ^Database database latest-available latest-applied]- (let [table (.getDatabaseChangeLogTableName database)- versions (range (inc latest-available) (inc latest-applied))- clauses (str/join " OR " (map #(format "id LIKE 'v%d.%%'" %) versions))- query (format "SELECT id FROM %s WHERE %s ORDER BY dateexecuted ASC" table clauses)]- (mapv :id (jdbc/query {:connection conn} [query]))))--(defn rollback-major-version!- "Roll back migrations later than given Metabase major version. If force is true, it will ignore any checks and always- roll back"- ;; default rollback to previous version- ([conn liquibase force]- ;; get current major version of Metabase we are running- (rollback-major-version! conn liquibase force (dec (config/current-major-version))))-- ;; with explicit target version- ([conn ^Liquibase liquibase force target-version]- (when (or (not (integer? target-version)) (< target-version 44))- (throw (IllegalArgumentException.- (format "target version must be a number between 44 and the previous major version (%d), inclusive"- (config/current-major-version)))))- (with-scope-locked liquibase- ;; count and rollback only the applied change set ids which come after the target version (only the "v..." IDs need- ;; to be considered)- (let [changeset-query (format "SELECT id FROM %s WHERE id LIKE 'v%%'" (changelog-table-name liquibase))- changeset-ids (map :id (jdbc/query {:connection conn} [changeset-query]))- ;; IDs in changesets do not include the leading 0/1 digit, so the major version is the first number- ids-to-drop (set (filter #(< target-version (first (extract-numbers %))) changeset-ids))- latest-available (latest-available-major-version liquibase)- latest-applied (latest-applied-major-version conn (.getDatabase liquibase))- lb-db (.getDatabase liquibase)- ran-changesets (.getRanChangeSetList lb-db)- changelog (.getDatabaseChangeLog liquibase)- changeset-filter (proxy [ChangeSetFilter] []- (accepts [^ChangeSet changeSet]- (let [id (.getId changeSet)- result (contains? ids-to-drop id)]- (ChangeSetFilterResult. result (if result- (do- (log/infof "Going to roll back changeset %s" id)- (str "Changeset ID '" id "' is in target list"))- (str "Changeset ID '" id "' is not in target list")) nil))))- changelog-iterator (ChangeLogIterator. ran-changesets changelog- (doto (ArrayList.)- (.addAll- [(AlreadyRanChangeSetFilter. ran-changesets)- (IgnoreChangeSetFilter.)- (DbmsChangeSetFilter. lb-db)- changeset-filter])))- error-ids (atom [])]- (when (and (not force) (> latest-applied latest-available))- (throw (ex-info- (format "Cannot downgrade a database at version %d from Metabase version %d. You must run 'migrate down' from Metabase version >= %d."- latest-applied latest-available latest-applied)- {:latest-available latest-available- :latest-applied latest-applied})))- (log/infof "Rolling back app database schema to version %d" target-version)- (if (empty? ids-to-drop)- (log/info "No changesets to roll back")- (do- (let [change-listener (proxy [liquibase.changelog.visitor.AbstractChangeExecListener] []- (rollbackFailed [^ChangeSet change-set _dbchangelog _db ^Exception e]- (swap! error-ids conj (.getId change-set))- (log/errorf "Error rolling back migration %s: %s" (.getId change-set) (ex-message e))))]- (AbstractRollbackCommandStep/doRollback lb-db- changelog-file- nil- changelog-iterator- (.getChangeLogParameters liquibase)- changelog- change-listener))- (let [remaining-ids (app-db.db/changelog-ids conn (changelog-table-name liquibase) ids-to-drop)]- (when (seq remaining-ids)- (log/warnf "The following changesets were not rolled back. Likely because %s: %s"- (if (seq @error-ids)- (format "there were errors in rollback (%s)" (str/join ", " @error-ids))- "they are not in the changelog file")- (str/join ", " remaining-ids))))))))))+ ;; The UPDATE runs on every non-fresh boot: it is an idempotent no-op once filenames are consolidated, and the+ ;; filenames of pre-consolidation installs are too varied to guard on reliably. It assumes we have never moved+ ;; the boundary between the two files, i.e. that update-migrations still start from v45.+ (log/info "Updating liquibase table to reflect consolidated changeset filenames")+ (with-scope-locked liquibase+ (jdbc/execute!+ conn-spec+ [(format (str "UPDATE %s SET FILENAME = CASE WHEN ID = ? THEN ? WHEN ID < ? THEN ? WHEN ID < ? THEN ? "+ "ELSE FILENAME END WHERE FILENAME NOT LIKE ?")+ liquibase-table-name)+ "v00.00-000" update001-migrations-file+ "v45.00-001" legacy-migrations-file+ "v56.0000-00-00T00:00:00" update001-migrations-file+ ;; version-less migrations live in a year directory: `migrations/2026/...` (see [[year-directory-migration?]])+ "%/____/%"])+ (repair-version-less-filenames! conn liquibase liquibase-table-name)))))diff --git a/src/metabase/app_db/liquibase/rollback.clj b/src/metabase/app_db/liquibase/rollback.cljnew file mode 100644index 00000000000..bc2a777357b--- /dev/null+++ b/src/metabase/app_db/liquibase/rollback.clj@@ -0,0 +1,351 @@+(ns metabase.app-db.liquibase.rollback+ "Rolling the application database back by Liquibase deployment.++ A rollback target is a set of *boundary* deployments: every changeset that ran after the newest changelog row of+ those deployments is reversed (through Liquibase, when this changelog still contains the changeset) and its+ bookkeeping cleared, leaving the schema as those deployments last left it. The release `migrate down`+ ([[rollback-major-version!]]) picks the boundary from a Metabase major version recorded in+ `databasechangelog_version`; the development tooling ([[rollback-to-deployment!]]) names a deployment directly,+ since every dev deployment records the same version. See [[metabase.app-db.liquibase.versions]] for how versions+ are recorded."+ (:require+ [clojure.java.jdbc :as jdbc]+ [clojure.string :as str]+ [metabase.app-db.connection :as mdb.connection]+ [metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.versions :as versions]+ [metabase.util :as u]+ [metabase.util.i18n :refer [trs]]+ [metabase.util.log :as log])+ (:import+ (java.sql Connection)+ (java.util ArrayList)+ (liquibase Liquibase)+ (liquibase.changelog ChangeLogIterator ChangeSet)+ (liquibase.changelog.filter AlreadyRanChangeSetFilter ChangeSetFilter ChangeSetFilterResult DbmsChangeSetFilter IgnoreChangeSetFilter)+ (liquibase.changelog.visitor AbstractChangeExecListener)+ (liquibase.command.core AbstractRollbackCommandStep)+ (liquibase.database Database)))++(set! *warn-on-reflection* true)++;;; ------------------------------------------------- target resolution --------------------------------------------++(defn- resolve-target+ "Resolve a rollback `target` -- an integer major version, or a numeric string like `\"64\"` -- to a major recorded+ (ran or booted) for a deployment in the [[versions/rollback-window]] of `deployments`. nil when the target is not+ a bare major or is not recorded there. Point-release targets (e.g. `\"64.1\"`) are deliberately not supported:+ rollback boundaries are majors, resolved to the latest deployment of that major by [[rollback-plan]]."+ [deployments target force?]+ (let [major (cond+ (integer? target) (long target)+ (string? target) (some-> (re-matches #"\d+" target) parse-long)+ :else nil)]+ (when (contains? (versions/recorded-majors (versions/rollback-window deployments force?)) major)+ major)))++(defn resolve-rollback-target+ "The major version `target` resolves to as a `migrate down` target for the application database on `conn`, or nil+ when it is not a permitted one: it must be a major recorded for a deployment in the recent window (the current+ major plus the previous-major upgrade boundary) by default, or anywhere in the recorded history when `force?`+ widens it. See [[resolve-target]]."+ ([conn database target] (resolve-rollback-target conn database target false))+ ([^Connection conn ^Database database target force?]+ (resolve-target (versions/recorded-deployments conn database) target force?)))++(defn- deployments-with-major+ "The `deployment_id`s of `deployments` with any version of `major` recorded. A major can span several deployments:+ point releases, the same build recording across restarts, boot rows."+ [deployments major]+ (into #{}+ (comp (filter (fn [{:keys [versions]}]+ (some #(= major (versions/version->major %)) versions)))+ (map :deployment-id))+ deployments))++;;; ---------------------------------------------------- planning --------------------------------------------------++(defn- changeset-row-key+ "Unique identity of a `databasechangelog` row as `[filename author id]` -- the id alone is not unique (the same id can+ recur across changelog files or authors)."+ [{:keys [filename author id]}]+ [filename author id])++(defn- changeset-key+ "Unique identity of a Liquibase `ChangeSet` as `[filename author id]` (matches [[changeset-row-key]])."+ [^ChangeSet cs]+ [(.getFilePath cs) (.getAuthor cs) (.getId cs)])++(defn- changelog-keys+ "The [[changeset-key]]s of every changeset in the changelog `liquibase` was built with."+ [^Liquibase liquibase]+ (set (map changeset-key (.getChangeSets (.getDatabaseChangeLog liquibase)))))++(defn- rollback-plan+ "Compute what a rollback to the `boundary-deps` deployments should drop. Returns+ `{:changesets-to-drop <set of [filename author id] keys>, :changesets-to-retain <set of keys of re-run rows that+ must NOT be reversed>, :boundary-deployment <deployment_id the retained rows are reassigned to>,+ :deployments-to-drop <set of deployment_ids whose entire history is rolled back>}`.++ Every changeset that ran after the boundary is dropped, leaving the schema in the state those deployments last left+ it. The boundary is the latest point the `boundary-deps` reached: their latest changelog row, or where one of their+ changesets that re-ran later first ran, if that is later still. Ordering is Liquibase's own [[versions/exec-pos]].++ Rows with exectype `RERAN` in that window are *retained*, not dropped, when the changeset already existed at the+ target: a RERAN row is an older changeset that merely re-executed under the newer deployment (a `runOnChange`+ changeset whose checksum changed, or a `migrate force` re-run) -- Liquibase moves the row to the run's deployment+ with a fresh execution position, but reversing a changeset that predates the boundary would strip schema state the+ rollback target still needs. Retained rows are reassigned to `:boundary-deployment` (the latest deployment of the+ target major) so every surviving row stays attached to a deployment with a recorded version.++ Where the changeset first ran (see [[versions/first-runs]]) says whether it existed at the target: it did when that+ is at or before the boundary. A re-run without that record (one run by a binary from before it was kept) falls back+ to its id: a `vNN.` id existed at the target when its major is at most `target-major` (nil: any major), and a+ version-less id is retained, which keeps a version-less changeset first introduced *after* the target as an orphan+ that a later re-upgrade adopts again."+ [^Connection conn changelog-table boundary-deps target-major]+ (let [all-rows (jdbc/query {:connection conn}+ [(format "SELECT id, author, filename, deployment_id, dateexecuted, orderexecuted, exectype FROM %s" changelog-table)])+ first-runs (versions/first-runs conn)+ first-run (fn [{:keys [exectype] :as row}]+ (when (= "RERAN" (some-> exectype u/upper-case-en))+ (get first-runs (changeset-row-key row))))+ ;; a boundary deployment whose changeset re-ran later reached where it first ran+ [boundary-pos boundary-dep] (->> (concat all-rows (keep first-run all-rows))+ (filter #(boundary-deps (:deployment_id %)))+ (map (juxt versions/exec-pos :deployment_id))+ (sort-by first)+ last)+ window-rows (if boundary-pos+ (filter #(pos? (compare (versions/exec-pos %) boundary-pos)) all-rows)+ [])+ retain? (fn [{:keys [exectype id] :as row}]+ (and (= "RERAN" (some-> exectype u/upper-case-en))+ (if-let [first-ran (first-run row)]+ (not (pos? (compare (versions/exec-pos first-ran) boundary-pos)))+ (let [major (versions/legacy-major id)]+ (or (nil? major) (nil? target-major) (<= major target-major))))))+ rows-by-dep (group-by :deployment_id all-rows)]+ {:changesets-to-drop (set (map changeset-row-key (remove retain? window-rows)))+ :changesets-to-retain (set (map changeset-row-key (filter retain? window-rows)))+ :boundary-deployment boundary-dep+ ;; only clear a deployment's history/version rows when its entire history leaves it (dropped or reassigned)+ :deployments-to-drop (set (for [[dep in-window] (group-by :deployment_id window-rows)+ :when (and dep (= (count in-window) (count (get rows-by-dep dep))))]+ dep))}))++;;; ---------------------------------------------------- execution -------------------------------------------------++(defn- run-liquibase-rollback!+ "Reverse the changesets in `changesets-to-drop` (a set of `[filename author id]` keys) that still exist in the+ changelog file, using Liquibase's rollback machinery. Changesets that were removed from later changelog files cannot+ be reversed here and are instead cleared by [[delete-deployment-rows!]]. Returns a vector of changeset ids that+ errored during rollback (each also logged)."+ [^Liquibase liquibase ^Database lb-db changesets-to-drop]+ (let [ran-changesets (.getRanChangeSetList lb-db)+ changelog (.getDatabaseChangeLog liquibase)+ known (changelog-keys liquibase)+ changeset-filter (proxy [ChangeSetFilter] []+ (accepts [^ChangeSet changeSet]+ (let [k (changeset-key changeSet)+ result (and (contains? changesets-to-drop k) (contains? known k))]+ (ChangeSetFilterResult. result (if result+ (do+ (log/infof "Going to roll back changeset %s" changeSet)+ (str "Changeset '" changeSet "' is in target list"))+ (str "Changeset '" changeSet "' is not in target list")) nil))))+ changelog-iterator (ChangeLogIterator. ran-changesets changelog+ (doto (ArrayList.)+ (.addAll+ [(AlreadyRanChangeSetFilter. ran-changesets)+ (IgnoreChangeSetFilter.)+ (DbmsChangeSetFilter. lb-db)+ changeset-filter])))+ error-ids (atom [])+ change-listener (proxy [AbstractChangeExecListener] []+ (rollbackFailed [^ChangeSet change-set _dbchangelog _db ^Exception e]+ (swap! error-ids conj (.getId change-set))+ (log/errorf "Error rolling back migration %s: %s" (.getId change-set) (ex-message e))))]+ (AbstractRollbackCommandStep/doRollback lb-db+ liquibase/changelog-file+ nil+ changelog-iterator+ (.getChangeLogParameters liquibase)+ changelog+ change-listener)+ @error-ids))++(defn- log-unreversible-rows!+ "Log the rows in `changesets-to-drop` that have no changeset in this changelog to reverse them. Their bookkeeping+ is cleared with their deployment regardless, so any schema change they made stays behind -- worth calling out.+ The legacy-version-tracking marker is bookkeeping only and is expected here."+ [^Liquibase liquibase changesets-to-drop]+ (let [known (changelog-keys liquibase)+ orphan-ids (->> changesets-to-drop+ (remove known)+ (remove (fn [[filename]] (versions/legacy-version-tracking-marker? {:filename filename})))+ (map peek)+ sort)]+ (when (seq orphan-ids)+ (log/infof (str "The following changelog rows were cleared by the rollback but could not be reversed because this "+ "changelog does not contain them; any schema changes they made remain in place: %s")+ (str/join ", " orphan-ids)))))++(defn- reassign-changeset-rows!+ "Move the retained (re-run) `changeset-keys` onto `deployment-id` -- the deployment being rolled back to -- so they+ survive [[delete-deployment-rows!]] and stay attached to a deployment with a recorded version."+ [^Connection conn changelog-table changeset-keys deployment-id]+ (doseq [[filename author id] changeset-keys]+ (jdbc/execute! {:connection conn}+ [(format "UPDATE %s SET deployment_id = ? WHERE filename = ? AND author = ? AND id = ?" changelog-table)+ deployment-id filename author id])))++(defn- delete-deployment-rows!+ "Delete every `databasechangelog` and `databasechangelog_version` row for the fully-rolled-back `deployment-ids`. This+ removes changesets that were dropped from later changelog files (so [[run-liquibase-rollback!]] could not reverse+ them) but still need to be cleared from history, along with those deployments' version rows."+ [^Connection conn changelog-table deployment-ids]+ (when (seq deployment-ids)+ (doseq [table [changelog-table (versions/versions-table-name)]]+ (jdbc/execute! {:connection conn}+ (into [(format "DELETE FROM %s WHERE deployment_id IN (%s)"+ table+ (str/join ", " (repeat (count deployment-ids) "?")))]+ deployment-ids)))))++(defn- rollback-failure-message+ "Error message for a rollback that could not reverse some changesets. Only claims the database was left unchanged+ where that is true: Postgres rolls the transaction's DDL back; H2 and MySQL auto-commit DDL, so rollback steps+ executed before the failure may have persisted."+ [db-type target error-ids]+ (str (trs "Rollback to {0} failed: could not roll back changeset(s) {1}." target (str/join ", " error-ids))+ " "+ (if (= db-type :postgres)+ (trs "The database has been left unchanged.")+ (trs "Rollback steps executed before the failure may already have been committed ({0} cannot roll back DDL transactionally). Verify the schema, or restore from a backup, before retrying." (name db-type)))))++(defn- rollback-to-deployments!+ "Roll back every changeset that ran after the latest changelog row of the `boundary-deps` deployments (see+ [[rollback-plan]]). `target` is only used for logging and error messages; `target-major` is the major rolled back+ to (nil when unknown), which decides which re-run changesets are kept (see [[rollback-plan]]).++ Repairs version-less changelog filenames first (see [[liquibase/repair-version-less-filenames!]]): the rows must+ match their changesets to be reversed, and not every caller has been through+ [[liquibase/consolidate-liquibase-changesets!]] -- the development rollback ([[rollback-to-deployment!]]) has not."+ [conn ^Liquibase liquibase boundary-deps target target-major]+ (liquibase/with-scope-locked liquibase+ (let [lb-db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)+ _ (liquibase/repair-version-less-filenames! conn liquibase changelog-table)+ {:keys [changesets-to-drop changesets-to-retain boundary-deployment deployments-to-drop]}+ (rollback-plan conn changelog-table boundary-deps target-major)]+ (log/infof "Rolling back app database schema to %s" target)+ (if (and (empty? changesets-to-drop) (empty? changesets-to-retain))+ (log/info "No changesets to roll back")+ (let [error-ids (when (seq changesets-to-drop)+ (run-liquibase-rollback! liquibase lb-db changesets-to-drop))]+ ;; If any changeset failed to reverse, do NOT clear the deployments' history. Doing so would leave the+ ;; changelog claiming a rollback that only partly happened, and would drop the `legacy-version-tracking`+ ;; row with it -- so an older binary would read a *lower* major than the schema actually has and happily+ ;; start against it. Fail loudly instead: the caller ([[metabase.app-db.setup/migrate!]]) rolls the+ ;; transaction back -- which restores everything on Postgres, but on H2/MySQL DDL auto-commits, so+ ;; already-executed rollback steps may persist (the message says so; see [[rollback-failure-message]]).+ (when (seq error-ids)+ (throw (ex-info (rollback-failure-message (mdb.connection/db-type) target error-ids)+ {:target target, :failed-changesets (vec error-ids)})))+ (when (seq changesets-to-retain)+ (log/infof "Not reversing %d re-run (runOnChange/force) changeset(s) that predate the rollback target: %s"+ (count changesets-to-retain) (str/join ", " (sort (map peek changesets-to-retain))))+ (reassign-changeset-rows! conn changelog-table changesets-to-retain boundary-deployment))+ (log-unreversible-rows! liquibase changesets-to-drop)+ (delete-deployment-rows! conn changelog-table deployments-to-drop)+ (versions/forget-first-runs! conn changesets-to-drop)+ (versions/record-schema-marker! lb-db)))+ (versions/delete-versions-of-emptied-deployments! conn lb-db))))++;;; ------------------------------------------------------ guards --------------------------------------------------++(defn- refuse-dev-migrated-schema!+ "Throw when the schema's current deployment was made by a development build. Every dev deployment records the same+ [[versions/dev-version]], so there is no release major to step back to; the development tooling rolls back by+ deployment instead (see [[rollback-to-deployment!]])."+ [deployments]+ (when (versions/synthetic-dev-major? (versions/schema-major deployments))+ (throw (ex-info (format (str "This database was last migrated by a development build (%s), so there is no release "+ "version to roll back to. Roll back its deployments from a development checkout with "+ "`clojure -M:dev:migrate rollback last-deployment` (dev.migrate/rollback!), or rebuild "+ "the database.")+ versions/dev-version)+ {:schema-major versions/dev-major}))))++(defn- refuse-to-roll-back-newer-schema!+ "Throw unless `force?` when the schema was migrated by a NEWER Metabase version than this binary -- a later major, or+ a later patch of the same one: this binary's changelog does not contain those changesets, so Liquibase cannot+ reverse their DDL -- it would only delete their bookkeeping rows (including the legacy-version-tracking marker) and+ leave the schema silently corrupted, failing the next upgrade to that version.++ The schema is at the highest version that ran any deployment, not the newest one's: an older patch booted against a+ newer patch's schema runs its re-run changesets (`runAlways`, or a `runOnChange` one the patches differ in) as a+ newer deployment of its own, while the newer patch's changesets stay in place. Development builds are left to+ [[refuse-dev-migrated-schema!]]."+ [deployments force?]+ (when-not force?+ (let [schema-version (->> deployments+ (keep :ran-version)+ (remove (comp versions/synthetic-dev-major? versions/version->major))+ (reduce #(if (and %1 (not (neg? (versions/compare-versions %1 %2)))) %1 %2) nil))+ binary-version (versions/current-recorded-version)]+ (when (and schema-version (pos? (versions/compare-versions schema-version binary-version)))+ (let [display #(str/replace-first % #"^x\." "")]+ (throw (ex-info (format "Cannot downgrade a database at version %s from Metabase version %s. You must run 'migrate down' from Metabase version >= %s."+ (display schema-version) (display binary-version) (display schema-version))+ {:schema-version schema-version, :binary-version binary-version})))))))++;;; --------------------------------------------------- entry points -----------------------------------------------++(defn rollback-to-deployment!+ "Roll back every changeset that ran after the latest changelog row of `boundary-deployment-id`, whatever versions+ are recorded -- the development rollback (`dev.migrate/rollback!`), where each `migrate up` run is its own+ deployment. The id must be a deployment of the changelog (see [[versions/previous-deployment-id]] for the usual+ 'one run back' target)."+ [conn ^Liquibase liquibase boundary-deployment-id]+ (let [deployments (versions/recorded-deployments conn (.getDatabase liquibase))]+ (when-not (some #(= boundary-deployment-id (:deployment-id %)) deployments)+ (throw (IllegalArgumentException.+ (format "%s is not a deployment of this database (see the deployment_id column of %s)."+ (pr-str boundary-deployment-id) (liquibase/changelog-table-name liquibase)))))+ (rollback-to-deployments! conn liquibase #{boundary-deployment-id} (str "deployment " boundary-deployment-id)+ (some #(when (= boundary-deployment-id (:deployment-id %))+ (some-> (:ran-version %) versions/version->major))+ deployments))))++(defn rollback-major-version!+ "Roll back all migrations that ran after the most recent deployment of `target` -- an integer major version (or a+ numeric string like `\"64\"`). The target must be a major recorded in `databasechangelog_version` for a deployment+ in the [[versions/rollback-window]]: by default the current major or the previous recorded major (the last upgrade+ boundary); when `force?` is true, any recorded major in history.++ Without a target, rolls back to the previous *recorded* major -- one recorded major back even when the upgrade+ skipped majors or the current major shipped no migrations.++ Refuses a schema whose newest deployment was made by a development build (see [[refuse-dev-migrated-schema!]]), and,+ unless `force?` is true, one migrated by a NEWER Metabase version than this binary -- see+ [[refuse-to-roll-back-newer-schema!]]; run `migrate down` from the newer binary instead."+ ([conn ^Liquibase liquibase force?]+ (let [deployments (versions/recorded-deployments conn (.getDatabase liquibase))]+ (refuse-dev-migrated-schema! deployments)+ (refuse-to-roll-back-newer-schema! deployments force?)+ (if-let [major (versions/previous-recorded-major deployments force?)]+ (rollback-to-deployments! conn liquibase (deployments-with-major deployments major) major major)+ (log/info "No earlier recorded Metabase version to roll back to; nothing to do."))))++ ([conn ^Liquibase liquibase force? target]+ (let [deployments (versions/recorded-deployments conn (.getDatabase liquibase))]+ (refuse-dev-migrated-schema! deployments)+ (refuse-to-roll-back-newer-schema! deployments force?)+ (let [major (resolve-target deployments target force?)]+ (when (nil? major)+ (throw (IllegalArgumentException.+ (format "%s is not a valid rollback target. Target must be the major version of a recorded deployment (see the %s table)."+ (pr-str target) (versions/versions-table-name)))))+ (rollback-to-deployments! conn liquibase (deployments-with-major deployments major) target major)))))diff --git a/src/metabase/app_db/liquibase/versions.clj b/src/metabase/app_db/liquibase/versions.cljnew file mode 100644index 00000000000..489e32077e2--- /dev/null+++ b/src/metabase/app_db/liquibase/versions.clj@@ -0,0 +1,999 @@+(ns metabase.app-db.liquibase.versions+ "Metabase-version bookkeeping for Liquibase deployments: the `databasechangelog_version` table and the+ `vNN.legacy-version-tracking` marker rows in `databasechangelog`.++ Liquibase records *which* changesets ran, grouped by `deployment_id`, but not which Metabase version ran them. Now+ that changesets are version-less that has to be recorded separately, and it is what `migrate down` and the boot-time+ downgrade check are driven by. `databasechangelog_version` holds one row per `(deployment_id, metabase_version)`:++ * `ran_migrations = true`: the version that executed the deployment's changesets. There is exactly one per+ deployment. It is written by [[recording-exec-listener]] as the run's first changeset lands, or backfilled from the+ `vNN.` changeset ids for installs that predate this table (see [[ensure-version-tracking!]]).+ * `ran_migrations = false`: a version that later booted against the schema at that deployment without running+ anything, i.e. a release that shipped no migrations for this database. These rows are history: they never change+ what version the schema is *at*, but they do make that major a nameable `migrate down` target (see+ [[rollback-window]]).++ The table is managed directly rather than through a changeset, to avoid the bootstrapping loop of a version-tracking+ table that would have to record the very deployment creating it. Callers bring it up to date once, at the entry+ points (boot, `migrate`, the development rollback) with [[ensure-version-tracking!]]; everything else here is a+ side-effect-free read that assumes the table exists.++ Which deployment is *current* is always decided by `databasechangelog`: it is the deployment that owns the newest+ changelog row, in the `[dateexecuted orderexecuted]` order Liquibase itself uses (see [[exec-pos]] and+ [[recorded-deployments]]).+ `databasechangelog_version` never decides ordering.++ Liquibase generates the deployment id once per process (it lives in the root `Scope`), not once per update run;+ [[metabase.app-db.liquibase/run-in-scope-locked]] therefore stamps every locked scope with a fresh deployment id, so+ each migration run records as its own deployment even inside a long-lived process (a dev REPL). Development builds+ all record the same constant [[dev-version]], so for them the deployment is the only thing that tells runs apart."+ (:require+ [clojure.java.jdbc :as jdbc]+ [clojure.string :as str]+ [metabase.app-db.connection :as mdb.connection]+ [metabase.config.core :as config]+ [metabase.util :as u]+ [metabase.util.log :as log])+ (:import+ (java.sql Connection SQLException)+ (liquibase Scope)+ (liquibase.changelog ChangeSet ChangeSet$RunStatus)+ (liquibase.changelog.visitor AbstractChangeExecListener ChangeExecListener)+ (liquibase.database Database)+ (liquibase.util LiquibaseUtil)))++(set! *warn-on-reflection* true)++(defn- postgres?+ "Whether the application database is Postgres."+ []+ (= :postgres (mdb.connection/db-type)))++(defn versions-table-name+ "Name of the table that records the Metabase version associated with each Liquibase `deployment_id`, cased like+ Liquibase's own `DATABASECHANGELOG` on the application database (upper case on H2 and MySQL, where MySQL table names+ can be case-sensitive; lower case on Postgres)."+ ^String []+ (if (postgres?)+ "databasechangelog_version"+ "DATABASECHANGELOG_VERSION"))++;;; ------------------------------------------------- JDBC helpers --------------------------------------------------++(defn table-exists?+ "Check if a table exists."+ [table-name ^Connection conn]+ (-> (.getMetaData conn)+ (.getTables nil nil table-name (u/varargs String ["TABLE"]))+ jdbc/metadata-query+ seq+ boolean))++(defn versions-table-exists?+ "Whether the [[versions-table-name]] table exists, checked without creating it."+ [^Connection conn]+ (table-exists? (versions-table-name) conn))++(defn first-runs-table-name+ "Name of the table that records where each re-run changeset first ran (see [[record-first-run!]]), cased like+ [[versions-table-name]]."+ ^String []+ (if (postgres?)+ "databasechangelog_first_run"+ "DATABASECHANGELOG_FIRST_RUN"))++(defn- underlying-connection ^Connection [^Database database]+ (.. database getConnection getUnderlyingConnection))++(defn- changelog-table ^String [^Database database]+ (.getDatabaseChangeLogTableName database))++(defn- query [^Connection conn sql-params]+ (jdbc/query {:connection conn} sql-params))++(defn- execute-ddl!+ "Run `ddl` on `conn` as it is, without the transaction `clojure.java.jdbc` wraps statements in: that transaction's+ rollback on failure would also discard the caller's savepoint (see [[create-table-if-missing!]])."+ [^Connection conn ^String ddl]+ (jdbc/execute! {:connection conn} [ddl] {:transaction? false}))++(defn- execute! [^Connection conn sql-params]+ (jdbc/execute! {:connection conn} sql-params))++;;; -------------------------------------------- the version being recorded ----------------------------------------++(defn version->major+ "Parse the Metabase major version out of a recorded version string, e.g. 55 from `x.55.2.1`."+ [version]+ (some-> (re-find #"^x\.(\d+)" (str version)) second parse-long))++(defn major->version+ "The version string recorded for major `major` when no exact version is known, e.g. `x.55.0.0` for 55: what the+ backfill and the repairs record, and what development builds record. The inverse of [[version->major]]."+ [major]+ (format "x.%d.0.0" major))++(defn- version-numbers+ "The leading dotted numbers of a recorded version string, e.g. `[65 5 1]` for `x.65.5.1` and `[66 0]` for+ `x.66.0-RC2`; nil when it has none."+ [version]+ (some->> (re-find #"^x\.(\d+(?:\.\d+)*)" (str version)) second (re-seq #"\d+") (mapv parse-long)))++(defn compare-versions+ "Compare two recorded version strings by their numbers, a missing trailing number counting as 0 (`x.65.0` is+ `x.65.0.0`). A pre-release suffix is ignored, so `x.66.0-RC2` compares equal to `x.66.0`."+ [a b]+ (let [a (version-numbers a)+ b (version-numbers b)+ padded #(vec (take (max (count a) (count b)) (concat % (repeat 0))))]+ (compare (padded a) (padded b))))++(def dev-major+ "The major of [[dev-version]]. Far above any real Metabase major, so a schema last written by a development build+ always reads as *newer* than every release binary, which then refuses to boot against it (see+ [[metabase.app-db.setup/error-if-downgrade-required!]])."+ 9999)++(def dev-version+ "The constant version recorded by builds that have no real release version (local development, where+ `version.properties` is absent). Every dev deployment records this same version, so the release paths -- the+ boot-time downgrade check and `migrate down` -- refuse a schema whose newest deployment is a dev one and point at the+ development tooling, which tells deployments apart by `deployment_id` instead (see+ [[metabase.app-db.liquibase.rollback/rollback-to-deployment!]] and `dev.migrate/rollback!`)."+ (major->version dev-major))++(defn synthetic-dev-major?+ "Whether `major` is the development major ([[dev-major]]) rather than a real released version."+ [major]+ (= dev-major major))++(defonce ^{:private true+ :doc "Whether this process has already logged that a prod build fell back to [[dev-version]]: the version+ is consulted several times per boot, and the error only needs to be logged once."}+ dev-version-fallback-logged?+ (atom false))++(defn- synthetic-dev-version+ "The [[dev-version]] constant. A production build should never get here -- it means the build's version tag could+ not be parsed -- so log (once) that version tracking is degraded."+ []+ (when (and config/is-prod? (compare-and-set! dev-version-fallback-logged? false true))+ (log/errorf (str "This build's version tag %s is not a release version (vX.Y.Z), so this instance will run "+ "migrations as a development build and record the development version %s for them. An official "+ "Metabase release pointed at this application database later will refuse to start "+ "(\"Development build detected\") until those deployments are rolled back with the development "+ "tooling or the database is rebuilt. This usually means version.properties is missing or "+ "malformed, e.g. a build from a git checkout without release tags (vUNKNOWN).")+ (pr-str (:tag config/mb-version-info)) dev-version))+ dev-version)++(defn- real-recorded-version+ "The edition-agnostic real version string from the build tag, or nil in dev where there is no real version."+ []+ (let [tag (:tag config/mb-version-info)]+ (when (re-find #"^v\d+\.\d+" tag)+ (str/replace tag #"^v\d+" "x"))))++(defn current-recorded-version+ "The edition-agnostic Metabase version string to record for this process (`x.56.2` for both `v0.56.2` and+ `v1.56.2`), or the constant [[dev-version]] when the build has no real version."+ []+ (or (real-recorded-version)+ (synthetic-dev-version)))++(defn current-recorded-major+ "Major version of [[current-recorded-version]]."+ []+ (version->major (current-recorded-version)))++;;; ------------------------------------------- the legacy `vNN.` id scan -------------------------------------------++(def ^:private special-case-migrations+ "Changeset ids of a later major that an earlier release line also ships, mapped to that earlier major, so a database+ of the earlier major that ran them is not taken for one of the later major:++ * the v56 changesets checked into the v55 branch to resolve inadvertently backported migrations in 55;+ * the v65 view changesets that 1.64.2.1 ran and 64 ships again (#84415)."+ {"v56.2025-06-05T16:48:48" 55+ "v56.2025-05-19T16:48:48" 55+ "v65.2026-10-07T12:00:00" 64+ "v65.2026-10-07T12:00:01" 64})++(defn- extract-numbers+ "Returns contiguous integers parsed from string s"+ [s]+ (if-let [special-cased (special-case-migrations s)]+ [special-cased]+ (map #(Integer/parseInt %) (re-seq #"\d+" s))))++(defn latest-applied-major-version+ "The major of the highest `vNN.` changeset id in the changelog, or nil when there is none.++ This is how Metabase binaries that predate version tracking decide what version a schema is at, and it is what the+ `vNN.legacy-version-tracking` marker rows keep accurate for them (see [[record-legacy-version-tracking!]]). Nothing+ in this binary decides anything by it any more -- it is kept as the oracle for what those older binaries will see,+ which is what the tests of the marker assert on, and it feeds the one-time backfill of pre-tracking installs."+ [^Connection conn ^Database database]+ (let [changelog (changelog-table database)]+ (when (table-exists? changelog conn)+ (let [changeset-id (-> (query conn [(format "SELECT id FROM %s WHERE id LIKE 'v%%' ORDER BY id DESC LIMIT 1" changelog)])+ first+ :id)]+ (some-> changeset-id extract-numbers first)))))++(def ^:private legacy-version-tracking-suffix "legacy-version-tracking")++(def ^:private legacy-version-tracking-author "version-tracking")++(def ^:private legacy-version-tracking-comment+ "Not a real migration, tracking version for instances prior to the databasechangelog_version tracking.")++(defn- legacy-version-tracking-description+ "The `description` of the marker row: what a `SELECT * FROM databasechangelog` reader sees next to the changeset+ descriptions Liquibase writes (`addColumn tableName=...`)."+ [major]+ (format "Metabase v%d version marker (not a changeset)" major))++(defn legacy-version-tracking-marker?+ "Whether a `databasechangelog` row (`{:filename ..}`) is a `vNN.legacy-version-tracking` marker rather than a changeset."+ [{:keys [filename]}]+ (= filename legacy-version-tracking-suffix))++(defn- legacy-marker-id [major]+ (format "v%d.%s" major legacy-version-tracking-suffix))++(defn legacy-major+ "The major of a `vNN.` changeset id (see [[extract-numbers]]), or nil for a version-less id."+ [id]+ (when (re-find #"^v\d+\." id)+ (first (extract-numbers id))))++(defn- version-less-id?+ "Whether `id` is a version-less changeset's: neither a `vNN.` id nor a pre-4.2 numeric one."+ [id]+ (not (or (legacy-major id) (re-matches #"\d+" id))))++(defn- highest-legacy-major+ "The highest `vNN.` major in the changelog, or 0 when there are no versioned changesets.++ The highest major, not the major of the most-recently-*executed* changeset: back-ported patches can land a+ lower-version changeset (e.g. a `v44.` id) with a later `dateexecuted` than the highest-version changeset already+ applied, and the schema's real version is the highest one."+ [^Connection conn ^Database database]+ (->> (query conn [(format "SELECT id FROM %s WHERE id LIKE 'v%%'" (changelog-table database))])+ (keep (comp legacy-major :id))+ (reduce max 0)))++(defn- highest-real-legacy-major+ "The highest legacy changeset major, ignoring synthetic version-tracking markers. Used only to detect work that an+ older binary really rolled back: a marker is unknown to that binary and therefore survives its rollback."+ [^Connection conn ^Database database]+ (->> (query conn [(format "SELECT id, filename FROM %s WHERE id LIKE 'v%%'" (changelog-table database))])+ (remove legacy-version-tracking-marker?)+ (keep (comp legacy-major :id))+ (reduce max 0)))++;;; -------------------------------------------------- changelog reads ---------------------------------------------++(defn exec-pos+ "The execution position of a `databasechangelog` row as a comparable `[dateexecuted orderexecuted]` vector -- the+ order Liquibase itself lists ran changesets in (`ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC`). `orderexecuted`+ only increases within an update run and can restart, so it only orders rows that share a timestamp (e.g. one run's+ changesets on MySQL's second-precision column). `dateexecuted` is the database's `NOW()` in the writing session's+ time zone (the JVM's on Postgres and H2), so binaries run in different zones, or a clock set back, misorder+ deployments; the marker for older binaries does not depend on this (see [[legacy-marker-statement]])."+ [{:keys [dateexecuted orderexecuted]}]+ [dateexecuted orderexecuted])++(defn last-deployment-id+ "The `deployment_id` of the most-recently-applied changeset (in [[exec-pos]] order), or nil for an empty changelog."+ [^Connection conn ^Database database]+ (-> (query conn [(format "SELECT deployment_id FROM %s ORDER BY dateexecuted DESC, orderexecuted DESC LIMIT 1"+ (changelog-table database))])+ first+ :deployment_id))++(defn- ran-row? [{:keys [ran_migrations]}]+ ;; a MySQL table created before the column became bit(1) stores tinyint(1), which reads back as a number+ (or (true? ran_migrations) (= 1 ran_migrations)))++(defn- ran-version+ "The version recorded as having run `deployment-id`, or nil when it has no such row."+ [^Connection conn deployment-id]+ (-> (query conn [(format "SELECT metabase_version FROM %s WHERE deployment_id = ? AND ran_migrations = TRUE ORDER BY id LIMIT 1"+ (versions-table-name))+ deployment-id])+ first+ :metabase_version))++;;; -------------------------------------------------- deployment reads ---------------------------------------------++(defn first-runs+ "Where each re-run changeset first ran, as `{[filename author id] {:deployment_id :dateexecuted :orderexecuted}}`+ -- keyed by the changeset's full identity, as Liquibase identifies it -- the values copied from its changelog row+ before its first re-run. Empty when the table does not exist. A record is only meaningful while the changelog row is+ RERAN: one left behind by a binary that removed the row without it is replaced the next time the changeset re-runs+ (see [[first-run-statements]]).++ The filename is the one this binary keeps on the changelog row: older binaries may rewrite the row's, but+ [[metabase.app-db.liquibase/repair-version-less-filenames!]] restores it before a migration or rollback reads it."+ [^Connection conn]+ (if (table-exists? (first-runs-table-name) conn)+ (into {}+ (map (juxt (juxt :filename :author :changeset_id) #(select-keys % [:deployment_id :dateexecuted :orderexecuted])))+ (query conn [(format "SELECT changeset_id, author, filename, deployment_id, dateexecuted, orderexecuted FROM %s"+ (first-runs-table-name))]))+ {}))++(defn recorded-deployments+ "The application database's deployments, newest first: every `deployment_id` in `databasechangelog`, or in the+ first run of a changeset that re-ran later (see [[first-runs]]), ordered by the newest point it reached -- the+ [[exec-pos]] of its newest changelog row, or of where such a changeset first ran, if that is later still -- each with+ what `databasechangelog_version` records for it:++ {:deployment-id \"9673292410\"+ :position [#inst \"...\" 1234] ; the newest exec-pos it reached+ :ran-version \"x.65.2\" ; the version that ran it (nil for an unrecorded legacy deployment)+ :versions [\"x.65.2\" \"x.66.0\"]} ; every version recorded for it, ran and booted alike++ A deployment whose only changesets re-ran later (say an older binary that re-ran a `runOnChange` view on its own)+ keeps its place this way. The first entry is the deployment the schema is currently at. Rows with no+ `deployment_id` (written by Liquibase versions from before it existed) are ignored."+ [^Connection conn ^Database database]+ (let [rows (query conn [(format "SELECT id, author, filename, deployment_id, dateexecuted, orderexecuted, exectype FROM %s"+ (changelog-table database))])+ first-run (first-runs conn)+ positions (reduce (fn [positions {:keys [deployment_id] :as row}]+ (let [pos (exec-pos row)]+ (cond-> positions+ (and deployment_id+ (or (not (contains? positions deployment_id))+ (pos? (compare pos (positions deployment_id)))))+ (assoc deployment_id pos))))+ {}+ (concat rows+ (keep (fn [{:keys [filename author id exectype]}]+ (when (= "RERAN" (some-> exectype u/upper-case-en))+ (first-run [filename author id])))+ rows)))+ ;; `migrate print` reads this before the table exists: nothing is recorded yet+ by-dep (if (versions-table-exists? conn)+ (group-by :deployment_id+ (query conn [(format "SELECT deployment_id, metabase_version, ran_migrations FROM %s ORDER BY id"+ (versions-table-name))]))+ {})]+ (->> positions+ (sort-by val #(compare %2 %1))+ (mapv (fn [[deployment-id position]]+ (let [rows (by-dep deployment-id)]+ {:deployment-id deployment-id+ :position position+ :ran-version (some-> (first (filter ran-row? rows)) :metabase_version)+ :versions (mapv :metabase_version rows)}))))))++(defn schema-major+ "Major version the schema described by `deployments` (see [[recorded-deployments]]) is at: the ran-version major of+ its current deployment. Booted-only versions do not move this. nil when the current deployment has no recorded+ version (or there is no deployment at all)."+ [deployments]+ (some-> (first deployments) :ran-version version->major))++(defn last-deployment-version+ "The Metabase version string that *ran* the deployment the schema is currently at, or nil if there is none (empty+ changelog, or a deployment with no recorded version)."+ [^Connection conn ^Database database]+ (:ran-version (first (recorded-deployments conn database))))++(defn current-schema-major+ "[[schema-major]] of the application database on `conn`."+ [^Connection conn ^Database database]+ (schema-major (recorded-deployments conn database)))++(defn previous-deployment-id+ "The `deployment_id` of the second-newest deployment -- the boundary for rolling back just the newest one -- or nil+ when there is no earlier deployment. Used by the development rollback tooling (`dev.migrate/rollback!`)."+ [^Connection conn ^Database database]+ (:deployment-id (second (recorded-deployments conn database))))++(defn rollback-window+ "The deployments a `migrate down` may target, as a prefix of `deployments` (see [[recorded-deployments]]): by+ default every deployment that ran on the current schema major plus the single most recent deployment from an+ earlier major (the boundary of the last major upgrade). With `all?` the full recorded history is the window instead+ (what a forced rollback may target).++ The current schema major is that of the newest deployment, not this binary's version: a binary that shipped no+ migrations for this database is still windowed on the major the schema is actually at."+ [deployments all?]+ (if all?+ deployments+ (let [current (schema-major deployments)+ [current-major older] (split-with #(= current (some-> (:ran-version %) version->major)) deployments)]+ (concat current-major (take 1 older)))))++(defn recorded-majors+ "Every major recorded, ran or booted, for `deployments`, as a sorted set."+ [deployments]+ (into (sorted-set) (keep version->major) (mapcat :versions deployments)))++(defn previous-recorded-major+ "The recorded major immediately before the newest recorded release within the [[rollback-window]] -- the default+ target of `migrate down`. This is deliberately based on ran and boot rows alike: if v64 booted without migrations+ on a schema last changed by v63, the previous release is v63 (a no-op boundary), not v62. nil when there is no+ earlier recorded major to roll back to."+ [deployments all?]+ (let [majors (recorded-majors (rollback-window deployments all?))]+ (when (< 1 (count majors))+ (nth (vec majors) (- (count majors) 2)))))++(defn changesets-from-later-version+ "Returns changeset IDs applied by versions later than `latest-available` up to `latest-applied`, ordered by execution+ position. Version-prefixed ids are matched by the major in the id; version-less changesets carry no version in their+ id, so they are matched by the recorded ran-version major of their deployment (see [[recorded-deployments]])."+ [^Connection conn ^Database database latest-available latest-applied]+ (let [changelog (changelog-table database)+ versions (range (inc latest-available) (inc latest-applied))+ later-deps (->> (recorded-deployments conn database)+ (filter #(when-let [major (some-> (:ran-version %) version->major)]+ (and (> major latest-available) (<= major latest-applied))))+ (mapv :deployment-id))+ ;; RERAN rows are older changesets that merely re-executed under the later deployment (edited runOnChange+ ;; changesets); they are not "from" the later version and would only confuse the listing+ clauses (concat (map #(format "id LIKE 'v%d.%%'" %) versions)+ (when (seq later-deps)+ [(format "(deployment_id IN (%s) AND exectype <> 'RERAN')"+ (str/join ", " (repeat (count later-deps) "?")))]))+ sql (format "SELECT id FROM %s WHERE %s ORDER BY dateexecuted ASC, orderexecuted ASC"+ changelog (str/join " OR " clauses))]+ (into [] (comp (map :id) (remove special-case-migrations)) (query conn (into [sql] later-deps)))))++;;; ------------------------------------------------ writing version rows -------------------------------------------++(defn- sql-literal+ "Render a statement parameter as a SQL literal, for the SQL that `migrate print` hands to a human."+ [x]+ (cond+ (string? x) (str "'" (str/replace x "'" "''") "'")+ (boolean? x) (if x "TRUE" "FALSE")+ :else (str x)))++(defn- render-sql+ "Render a `[sql & params]` statement as a single SQL string with the parameters inlined."+ ^String [[sql & params]]+ (reduce (fn [s param] (str/replace-first s "?" (sql-literal param))) sql params))++(defn- ignores-duplicate-rows?+ "Whether [[version-row-statement]] can tell the application database to skip an already-present row itself. Postgres+ *must* (a failed INSERT aborts the surrounding transaction) and MySQL can; H2 has no portable form, so its callers+ check first."+ []+ (contains? #{:postgres :mysql} (mdb.connection/db-type)))++(defn- version-row-statement+ "`[sql & params]` inserting a `databasechangelog_version` row, skipping an already-present+ `(deployment_id, metabase_version)` pair where the dialect can express that (see [[ignores-duplicate-rows?]])."+ [deployment-id version ran?]+ (let [insert (format "INSERT INTO %s (deployment_id, metabase_version, ran_migrations, deployed_at) VALUES (?, ?, ?, CURRENT_TIMESTAMP)"+ (versions-table-name))+ sql (case (mdb.connection/db-type)+ :postgres (str insert " ON CONFLICT DO NOTHING")+ :mysql (str/replace-first insert "INSERT INTO" "INSERT IGNORE INTO")+ insert)]+ [sql deployment-id version ran?]))++(defn record-deployment-version!+ "Record that `version` ran deployment `deployment-id` (`ran?` true) or booted against the schema at it (`ran?`+ false). Idempotent: an already-present `(deployment_id, metabase_version)` pair is left as it is, so a boot row never+ overwrites the ran row of the same version. No-op when `deployment-id` is nil (an empty changelog has no deployment+ to record). Safe to call concurrently from several instances: every path tolerates the pair appearing between check+ and insert."+ [^Connection conn deployment-id version ran?]+ (when (and deployment-id version)+ (let [statement (version-row-statement deployment-id version ran?)+ pair-exists? #(seq (query conn [(format "SELECT 1 FROM %s WHERE deployment_id = ? AND metabase_version = ?"+ (versions-table-name))+ deployment-id version]))]+ (if (ignores-duplicate-rows?)+ (execute! conn statement)+ (when-not (pair-exists?)+ (try+ (execute! conn statement)+ (catch SQLException e+ ;; lost the check-then-insert race with another instance; the unique constraint makes the loss benign+ (when-not (pair-exists?)+ (throw e)))))))))++(defn record-active-deployment-version!+ "Record the running Metabase version as the one that ran the deployment of the current Liquibase scope. The exec+ listeners call this as the first changeset of a run lands, so the version is recorded even if a later changeset+ fails."+ [^Database database]+ (record-deployment-version! (underlying-connection database)+ (.getDeploymentId (Scope/getCurrentScope))+ (current-recorded-version)+ true))++;;; ------------------------------------------ first runs of re-run changesets -------------------------------------+;;; When a changeset runs again (a `runOnChange` changeset whose checksum changed, or any `runAlways` changeset),+;;; Liquibase moves its changelog row to the new run: new deployment_id, dateexecuted and orderexecuted, exectype RERAN.+;;; A rollback must know where it first ran to decide whether the rollback target has it, so that is copied to+;;; `databasechangelog_first_run` first.++(defn- first-run-statements+ "`[sql & params]` statements that record where the changeset `[filename author id]` first ran, for a run that is+ about to re-run it. Its changelog row still holds that run unless it is already RERAN, in which case a recorded first run is+ kept. A record left from an earlier life of the changeset (see [[first-runs]]) is replaced. Nothing to do when the+ changeset has no changelog row, or when its first run is recorded already."+ [^Connection conn ^Database database [filename author id]]+ (when-let [{:keys [exectype]} (first (query conn [(format "SELECT exectype FROM %s WHERE id = ? AND author = ? AND filename = ?"+ (changelog-table database))+ id author filename]))]+ (let [first-runs (first-runs-table-name)+ recorded? (and (table-exists? first-runs conn)+ (seq (query conn [(format "SELECT 1 FROM %s WHERE changeset_id = ? AND author = ? AND filename = ?"+ first-runs)+ id author filename])))+ reran? (= "RERAN" (some-> exectype u/upper-case-en))]+ (when-not (and reran? recorded?)+ (cond->> [[(format (str "INSERT INTO %s (changeset_id, author, filename, deployment_id, dateexecuted, orderexecuted) "+ "SELECT id, author, filename, deployment_id, dateexecuted, orderexecuted FROM %s "+ "WHERE id = ? AND author = ? AND filename = ?")+ first-runs (changelog-table database))+ id author filename]]+ recorded? (cons [(format "DELETE FROM %s WHERE changeset_id = ? AND author = ? AND filename = ?" first-runs)+ id author filename]))))))++(defn- change-set-key+ "`[filename author id]` of `change-set`, with the filename Liquibase writes to its changelog row."+ [^ChangeSet change-set]+ [(.getStoredFilePath change-set) (.getAuthor change-set) (.getId change-set)])++(defn record-first-run!+ "Record where `change-set` first ran, before Liquibase re-runs it (see [[first-run-statements]])."+ [^Database database ^ChangeSet change-set]+ (let [conn (underlying-connection database)]+ (doseq [statement (first-run-statements conn database (change-set-key change-set))]+ (execute! conn statement))))++(defn forget-first-runs!+ "Delete the first-run records of the changesets `changeset-keys` (`[filename author id]`), for changesets a+ rollback reversed."+ [^Connection conn changeset-keys]+ (when (and (seq changeset-keys) (table-exists? (first-runs-table-name) conn))+ (doseq [[filename author id] changeset-keys]+ (execute! conn [(format "DELETE FROM %s WHERE changeset_id = ? AND author = ? AND filename = ?" (first-runs-table-name))+ id author filename]))))++(defn delete-versions-of-emptied-deployments!+ "Delete the `databasechangelog_version` rows of deployments that have no changelog row left and no first run (see+ [[first-runs]]): a run that re-runs changesets moves their rows out of the deployment that ran them, so a deployment+ that only re-ran changesets is emptied by the next one that re-runs them too, which leaves its version row behind."+ [^Connection conn ^Database database]+ (execute! conn [(format (str "DELETE FROM %s WHERE deployment_id NOT IN "+ "(SELECT deployment_id FROM %s WHERE deployment_id IS NOT NULL)%s")+ (versions-table-name)+ (changelog-table database)+ (if (table-exists? (first-runs-table-name) conn)+ (format " AND deployment_id NOT IN (SELECT deployment_id FROM %s WHERE deployment_id IS NOT NULL)"+ (first-runs-table-name))+ ""))]))++(defn recording-exec-listener+ "A Liquibase `ChangeExecListener` that records the running Metabase version against the current `deployment_id` as+ soon as the first changeset of a run is applied (see [[record-active-deployment-version!]]), and where each+ changeset it re-runs first ran (see [[record-first-run!]])."+ ^ChangeExecListener [^Database database]+ (let [active-deployment-version-recorded? (atom false)]+ (proxy [AbstractChangeExecListener] []+ (willRun [change-set _database-change-log _database run-status]+ ;; RUN_AGAIN for a runOnChange changeset whose checksum changed, ALREADY_RAN for a runAlways one+ (when (and (instance? ChangeSet change-set) (not= ChangeSet$RunStatus/NOT_RAN run-status))+ (record-first-run! database change-set)))+ (ran [change-set _database-change-log _database _exec-type]+ (when (and (instance? ChangeSet change-set) (compare-and-set! active-deployment-version-recorded? false true))+ (record-active-deployment-version! database))))))++(defn record-boot-version!+ "Record that this Metabase version booted against the schema at its current deployment without running any+ migrations (`ran_migrations = false`). History only: a boot that ran nothing did not change the schema, so the+ deployment's ran version still decides what version it is at; but a release that shipped no migrations for this+ database becomes a nameable `migrate down` target this way.++ Nothing is recorded for a development build (its constant [[dev-version]] says nothing), for a binary older than the+ version that ran the deployment (an unsupported downgrade -- the boot check refuses it, but `migrate up` skips that+ check), or when the deployment has no ran version at all (a pre-tracking install with no `vNN.` ids to backfill+ from)."+ [^Database database]+ (let [conn (underlying-connection database)+ major (current-recorded-major)]+ (when-let [deployment-id (last-deployment-id conn database)]+ (let [ran-major (some-> (ran-version conn deployment-id) version->major)]+ (when (and ran-major+ (not (synthetic-dev-major? major))+ (>= major ran-major))+ (record-deployment-version! conn deployment-id (current-recorded-version) false))))))++;;; ---------------------------------------- the legacy-version-tracking marker -------------------------------------++(defn- liquibase-build-version+ "What Liquibase writes to the `liquibase` column of the rows it records: its build version, `SNAPSHOT` shortened as+ Liquibase's own `MarkChangeSetRanGenerator` does."+ ^String []+ (str/replace (LiquibaseUtil/getBuildVersion) "SNAPSHOT" "SNP"))++(def ^:private position-marker-major+ "The major of the marker recorded for a schema at a versioned major, one whose own `vNN.` changesets are in the+ changelog. Those ids already give older binaries its major, so the marker only has to be a row they read safely as+ the newest one: a major low enough never to be the highest `id LIKE 'v%'` (or be dropped by their `migrate down`), at+ least 45 (their pre-v45 changelog cutoff), and an id sorting after `v56.0000-00-00T00:00:00`, which their changelog+ filename consolidation would otherwise rewrite."+ 56)++(defn- marker-major+ "The major of the marker recorded for a schema at `major`: [[position-marker-major]] when `major` is versioned --+ `versioned-through` (the highest real `vNN.` major the schema has) reaches it -- else the major itself, which older+ binaries learn from nothing else."+ [major versioned-through]+ (if (>= versioned-through major) position-marker-major major))++(defn- marker-delete-statement [changelog major]+ [(format "DELETE FROM %s WHERE id = ? AND author = ?" changelog)+ (legacy-marker-id major) legacy-version-tracking-author])++(defn- legacy-marker-statement+ "`[sql & params]` inserting the `vNN.legacy-version-tracking` row for `major` as a row of deployment `deployment-id`,+ as the newest row of the changelog: at Liquibase's own next execution position (`MAX(orderexecuted) + 1`), and+ executed strictly after every other row. Binaries from before version-less support pick their changelog from the+ row with the newest `dateexecuted` alone, so a tie would be as bad as losing: MySQL stores Liquibase's sub-second+ timestamps rounded, up to a second in the future, hence the fallback to one second past the newest row. Computed on+ the server, in the same clock Liquibase's rows use. Stamped with the Liquibase version like the rows Liquibase+ records itself."+ [changelog major deployment-id]+ [(format (str "INSERT INTO %s (id, author, filename, dateexecuted, orderexecuted, exectype, deployment_id, "+ "description, comments, liquibase) "+ "SELECT ?, ?, ?, "+ "CASE WHEN MAX(dateexecuted) IS NULL OR CURRENT_TIMESTAMP > MAX(dateexecuted) THEN CURRENT_TIMESTAMP "+ "ELSE MAX(dateexecuted) + INTERVAL '1' SECOND END, "+ "COALESCE(MAX(orderexecuted), 0) + 1, 'EXECUTED', ?, ?, ?, ? FROM %s")+ changelog changelog)+ (legacy-marker-id major) legacy-version-tracking-author legacy-version-tracking-suffix+ deployment-id (legacy-version-tracking-description major) legacy-version-tracking-comment (liquibase-build-version)])++(defn- newest-row-is?+ "Whether the single row with the newest `dateexecuted` is marker `id` of `deployment-id`."+ [^Connection conn changelog id deployment-id]+ (= [[id deployment-id]]+ (mapv (juxt :id :deployment_id)+ (query conn [(format "SELECT id, deployment_id FROM %s WHERE dateexecuted = (SELECT MAX(dateexecuted) FROM %s)"+ changelog changelog)]))))++(defn record-legacy-version-tracking!+ "Make a synthetic `vNN.legacy-version-tracking` row the newest row of the changelog, as an ordinary row of+ `deployment-id` (by default the deployment that just ran migrations) for a schema at `major` (by default the+ running binary's).++ Binaries from before version-less support -- the earlier patches of every major it was backported to, and every+ earlier major -- know nothing about `databasechangelog_version`, and read the changelog in two ways this row keeps+ safe:++ * they pick their changelog from the id of the newest row by `dateexecuted`, and take a version-less id for a+ pre-4.2 install: they load the legacy changelog and fail re-running changeset 1. So the marker is moved to the end+ of every run, rollback included -- once per major is not enough when a later patch runs version-less changesets+ only.+ * they read the schema's major from the highest `vNN.` id ([[latest-applied-major-version]]). A versioned major's own+ ids already say it, and its marker is the low [[position-marker-major]], which their `migrate down` never treats+ as newer than the major it rolls back to. A version-less major has no other `vNN.` id to say so, and the marker+ carries the major; earlier majors' markers are deliberately left in place, so rolling back the later deployments+ leaves the earlier major's row the highest again.++ It deliberately carries the **same `deployment_id`** as the deployment it ends, so it needs no special handling+ anywhere: rolling that deployment back deletes it like any other row of the deployment.++ Dev deployments record a `v9999.` marker too: a pre-version-less binary pointed at a dev-written DB then refuses+ cleanly ('migrations from v9999')."+ ([^Database database]+ (record-legacy-version-tracking! database+ (current-recorded-major)+ (last-deployment-id (underlying-connection database) database)))+ ([^Database database major deployment-id]+ (when (and major deployment-id)+ (let [conn (underlying-connection database)+ changelog (changelog-table database)+ mmajor (marker-major major (highest-real-legacy-major conn database))+ id (legacy-marker-id mmajor)]+ (when-not (newest-row-is? conn changelog id deployment-id)+ (try+ (execute! conn (marker-delete-statement changelog mmajor))+ (execute! conn (legacy-marker-statement changelog mmajor deployment-id))+ (catch Throwable e+ ;; This row only improves downgrade detection for *older* binaries -- it must never block a migration.+ (log/warnf e "Could not record %s in %s; older Metabase versions may not detect a downgrade from this schema"+ id changelog))))))))++(defn record-schema-marker!+ "[[record-legacy-version-tracking!]] for the schema as it now is, on its current deployment -- after a rollback,+ which can delete the marker with the deployments it removes, and keeps re-run rows that are newer than the boundary's+ marker (see [[metabase.app-db.liquibase.rollback/rollback-plan]]). The major is the current deployment's recorded+ one, or for an unrecorded legacy deployment its highest `vNN.` id."+ [^Database database]+ (let [conn (underlying-connection database)+ major (or (current-schema-major conn database)+ (let [highest (highest-real-legacy-major conn database)]+ (when (pos? highest) highest)))]+ (record-legacy-version-tracking! database major (last-deployment-id conn database))))++;;; --------------------------------------------- table creation and backfill ---------------------------------------++(defn- table-ddl+ ^String []+ (let [db-type (mdb.connection/db-type)+ id-column (case db-type+ :mysql "id bigint NOT NULL AUTO_INCREMENT PRIMARY KEY"+ "id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY")+ ;; MySQL's `boolean` is tinyint(1); Metabase booleans there are bit(1) -- see general-schema-tests+ bool-type (case db-type+ :mysql "bit(1)"+ "boolean")+ ;; Match the type produced by the old UnifyTimeColumnsType custom migration. The version table is created+ ;; before Liquibase runs: a fresh install still runs that migration over it, while an upgrade on which the+ ;; migration already ran does not. Creating the final type directly keeps both paths identical.+ timestamp-type (case db-type+ (:postgres :h2) "timestamp with time zone"+ :mysql "timestamp(6)")+ ;; match the storage options every other Metabase table gets on MySQL -- see [[metabase.app-db.liquibase.mysql]]+ suffix (case db-type+ :mysql " ENGINE InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"+ "")]+ (format (str "CREATE TABLE IF NOT EXISTS %s ("+ "%s, "+ "deployment_id varchar(10) NOT NULL, "+ "metabase_version varchar(255) NOT NULL, "+ "ran_migrations %s NOT NULL, "+ "deployed_at %s NOT NULL, "+ "CONSTRAINT uq_databasechangelog_version UNIQUE (deployment_id, metabase_version))%s")+ (versions-table-name) id-column bool-type timestamp-type suffix)))++(defn- first-runs-table-ddl+ ^String []+ (let [db-type (mdb.connection/db-type)+ ;; Liquibase's own DATEEXECUTED type: the copied values must compare with the changelog's. UnifyTimeColumnsType+ ;; leaves this table alone, as it does the changelog.+ datetime-type (case db-type+ :postgres "timestamp without time zone"+ :mysql "datetime"+ "timestamp")+ suffix (case db-type+ :mysql " ENGINE InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci"+ "")]+ (format (str "CREATE TABLE IF NOT EXISTS %s ("+ "changeset_id varchar(255) NOT NULL, "+ "author varchar(255) NOT NULL, "+ "filename varchar(255) NOT NULL, "+ "deployment_id varchar(10), "+ "dateexecuted %s NOT NULL, "+ "orderexecuted int NOT NULL, "+ "CONSTRAINT pk_databasechangelog_first_run PRIMARY KEY (changeset_id, author, filename))%s")+ (first-runs-table-name) datetime-type suffix)))++(defn- change-set-id [change-set]+ (if (instance? ChangeSet change-set)+ (.getId ^ChangeSet change-set)+ (str change-set)))++(defn- backfill-version+ "The version to record as having run the current deployment of an install that predates version tracking: the+ [[highest-legacy-major]] as `x.<major>.0.0`, or nil when there are no versioned changesets. Older deployments are+ recorded by [[backfill-untracked-deployments!]] at the major their own position puts the schema at."+ [^Connection conn ^Database database]+ (let [major (highest-legacy-major conn database)]+ (when (pos? major)+ (major->version major))))++(defn- legacy-major-at+ "Fn of a [[recorded-deployments]] entry's `:position`: the highest real (non-marker) `vNN.` major among the rows+ executed up to it -- what a binary that predates version tracking took the schema to be when it ran that+ deployment."+ [^Connection conn ^Database database]+ (let [rows (->> (query conn [(format "SELECT id, filename, dateexecuted, orderexecuted FROM %s WHERE id LIKE 'v%%'"+ (changelog-table database))])+ (remove legacy-version-tracking-marker?)+ (keep (fn [row] (when-let [major (legacy-major (:id row))] [(exec-pos row) major]))))]+ (fn [position]+ (transduce (comp (filter #(not (pos? (compare (first %) position)))) (map second)) max 0 rows))))++(defn- backfill-versions+ "`[deployment-id version]` for each deployment with no recorded version: on an install that predates this table,+ every deployment; otherwise every deployment newer than the newest recorded one. Those were run by a binary from+ before version-less support -- an earlier major, or an earlier patch of a major it was backported to -- and are+ recorded at the major their `vNN.` ids put the schema at, so they stay boundaries the downgrade check and `migrate+ down` know: an install that went 63 -> 64 before this table existed can still roll back to 63. The current deployment+ of a pre-tracking install is recorded at its [[backfill-version]]."+ [^Connection conn ^Database database]+ (let [[untracked tracked] (split-with (comp nil? :ran-version) (recorded-deployments conn database))+ major-at (legacy-major-at conn database)+ current (when (empty? tracked)+ (when-let [deployment-id (:deployment-id (first untracked))]+ (when-let [version (backfill-version conn database)]+ [[deployment-id version]])))]+ (concat current+ (for [{:keys [deployment-id position]} (cond-> untracked (empty? tracked) rest)+ :let [major (major-at position)]+ :when (pos? major)]+ [deployment-id (major->version major)]))))++(defn- backfill-untracked-deployments!+ "Record the [[backfill-versions]] as ran rows."+ [^Connection conn ^Database database]+ (doseq [[deployment-id version] (backfill-versions conn database)]+ (record-deployment-version! conn deployment-id version true)))++(defn- repair-old-binary-rollbacks!+ "Correct the version rows of deployments a binary from before version-less support rolled back with its own+ `migrate down`. It reverses the `vNN.` changesets above its target and nothing else: version rows, version-less+ changesets and the marker stay. So a deployment recorded above the highest real `vNN.` major left is a candidate,+ unless its major has a major marker (it is version-less, see [[record-legacy-version-tracking!]]). Its major was+ versioned -- and those changesets were removed -- when a candidate of that major carries the position marker that+ ended its run; such deployments are recorded instead at the highest major left, with their rows for the majors above+ dropped, otherwise every binary of those majors would take the schema for newer than it is and refuse to boot. A+ failed first run of a version-less major has no marker at all and is left alone, as is a development build's+ [[dev-version]]."+ [^Connection conn ^Database database]+ (let [highest (highest-real-legacy-major conn database)]+ (when (pos? highest)+ (let [markers (query conn [(format "SELECT id, deployment_id FROM %s WHERE filename = ?" (changelog-table database))+ legacy-version-tracking-suffix])+ position-id (legacy-marker-id position-marker-major)+ position-deps (into #{} (comp (filter #(= position-id (:id %))) (map :deployment_id)) markers)+ major-markers (into #{} (comp (remove #(= position-id (:id %))) (keep (comp legacy-major :id))) markers)+ candidates (for [{:keys [ran-version] :as dep} (recorded-deployments conn database)+ :let [ran-major (some-> ran-version version->major)]+ :when (and ran-major+ (> ran-major highest)+ (not (synthetic-dev-major? ran-major))+ (not (major-markers ran-major)))]+ (assoc dep :ran-major ran-major))+ versioned (into #{} (comp (filter (comp position-deps :deployment-id)) (map :ran-major)) candidates)]+ (doseq [{:keys [deployment-id ran-version ran-major]} candidates+ :when (versioned ran-major)]+ (log/warnf "Deployment %s was recorded as run by version %s, but an older Metabase version rolled back its changesets above %d; recording it as %s"+ deployment-id ran-version highest (major->version highest))+ (doseq [{:keys [id metabase_version]} (query conn [(format "SELECT id, metabase_version FROM %s WHERE deployment_id = ?"+ (versions-table-name))+ deployment-id])+ :when (> (version->major metabase_version) highest)]+ (execute! conn [(format "DELETE FROM %s WHERE id = ?" (versions-table-name)) id]))+ (record-deployment-version! conn deployment-id (major->version highest) true))))))++(defn- newest-row-safe-for-old-binaries?+ "Whether a binary from before version-less support picks the right changelog from this database: none of the rows+ with the newest `dateexecuted` (their only ordering, so with a tie the database may return either) is a+ version-less changeset's, which they would take for a pre-4.2 install. A `vNN.` or numeric id is read correctly --+ including below v45, which really does need the legacy changelog."+ [^Connection conn ^Database database]+ (let [changelog (changelog-table database)]+ (not-any? (comp version-less-id? :id)+ (query conn [(format "SELECT id FROM %s WHERE dateexecuted = (SELECT MAX(dateexecuted) FROM %s)"+ changelog changelog)]))))++(defn- create-table-if-missing!+ "Create `table-name` with `ddl` unless it exists. Checked first so an already-upgraded database runs no DDL at boot+ (a Postgres user without CREATE on the schema fails even `CREATE TABLE IF NOT EXISTS`). The table is committed at+ once, so other connections see it. A CREATE that fails because a concurrent boot created the table first is ignored;+ on Postgres a savepoint keeps that failure from aborting the caller's transaction."+ [^Connection conn table-name ddl]+ (when-not (table-exists? table-name conn)+ (let [savepoint (when (and (postgres?) (not (.getAutoCommit conn)))+ (.setSavepoint conn))]+ (try+ (execute-ddl! conn ddl)+ (when-not (.getAutoCommit conn)+ (.commit conn))+ (catch SQLException e+ (when savepoint+ (.rollback conn savepoint))+ (when-not (table-exists? table-name conn)+ (throw e)))))))++(defn ensure-version-tracking!+ "Bring version tracking up to date for the application database on `conn`: create `databasechangelog_version` and+ `databasechangelog_first_run` if they are missing, record versions for deployments that have none (see+ [[backfill-untracked-deployments!]]), correct the ones a binary from before version-less support rolled back (see+ [[repair-old-binary-rollbacks!]]), and make sure those binaries read the newest changelog row safely (see+ [[record-legacy-version-tracking!]]).++ Called once at each entry point that goes on to read or write version rows: the boot-time downgrade check, the+ `migrate` commands, and the development rollback. The CREATEs commit at once (see [[create-table-if-missing!]]), so+ running them first keeps that commit ahead of any migration work."+ [^Connection conn ^Database database]+ (create-table-if-missing! conn (versions-table-name) (table-ddl))+ (create-table-if-missing! conn (first-runs-table-name) (first-runs-table-ddl))+ (backfill-untracked-deployments! conn database)+ (repair-old-binary-rollbacks! conn database)+ ;; every run and rollback ends with a marker; this heals a database whatever else left it otherwise+ (when-not (newest-row-safe-for-old-binaries? conn database)+ (record-schema-marker! database)))++(defn repair-legacy-rollback!+ "Correct the version rows of the current deployment after a binary that predates version tracking rolled it back with+ its own `migrate down`: that removes the deployment's newest `vNN.` changesets but not its version rows, so its ran+ version still names the major it was rolled back from. When the deployment ran only `vNN.` changesets and its ran+ major is above the highest real legacy changeset left in the changelog, the ran version becomes that major and the+ deployment's version rows above it are dropped. `unrun-change-sets` is the changelog work the caller found while+ holding the migration lock. It distinguishes a stale synthetic marker left by an old rollback from a valid marker+ on a deployment that simply ran a lower-major backport.++ Deployments this binary's own versions made are left alone when they carry version-less changesets, or when their+ marker has no matching legacy changeset in the locked unrun set; a development build's [[dev-version]] is also left+ alone. Only safe while holding the migration lock: until a migrating binary writes its marker, its deployment looks+ just like a trimmed one."+ [^Connection conn ^Database database unrun-change-sets]+ (when-let [deployment-id (last-deployment-id conn database)]+ (let [ran (ran-version conn deployment-id)+ ran-major (some-> ran version->major)+ major (highest-real-legacy-major conn database)+ deployment-rows (query conn [(format "SELECT id, filename FROM %s WHERE deployment_id = ?"+ (changelog-table database))+ deployment-id])+ marker-majors (into #{} (comp (filter legacy-version-tracking-marker?)+ (keep (comp legacy-major :id)))+ deployment-rows)+ unrun-legacy-majors (into #{} (keep (comp legacy-major change-set-id)) unrun-change-sets)+ stale-marker? (some #(and (> % major)+ (some (fn [unrun-major]+ (<= (inc major) unrun-major %))+ unrun-legacy-majors))+ marker-majors)]+ (when (and ran-major+ (not (synthetic-dev-major? ran-major))+ (pos? major)+ (> ran-major major)+ (or (empty? marker-majors) stale-marker?)+ (not-any? (comp version-less-id? :id)+ deployment-rows))+ (let [version (major->version major)]+ (log/warnf "Deployment %s was recorded as run by version %s, but an older Metabase version rolled back its changesets above %d; recording it as %s"+ deployment-id ran major version)+ (doseq [{:keys [id metabase_version]} (query conn [(format "SELECT id, metabase_version FROM %s WHERE deployment_id = ?"+ (versions-table-name))+ deployment-id])+ :when (> (version->major metabase_version) major)]+ (execute! conn [(format "DELETE FROM %s WHERE id = ?" (versions-table-name)) id]))+ (record-deployment-version! conn deployment-id version true))))))++(defn first-runs-sql+ "SQL statements recording where the changesets in `unrun-change-sets` that `migrate print` re-runs first ran (see+ [[record-first-run!]]), for the start of its output: they must run before Liquibase's UPDATE moves those changelog+ rows. Nil when it re-runs nothing. Read-only, so the table's CREATE is part of the returned SQL."+ ^String [^Connection conn ^Database database unrun-change-sets]+ (when-let [statements (seq (mapcat #(first-run-statements conn database (change-set-key %)) unrun-change-sets))]+ (str "-- Record where re-run changesets first ran\n"+ (first-runs-table-ddl) ";\n"+ (str/join (for [statement statements] (str (render-sql statement) ";\n")))+ "\n")))++(defn version-tracking-sql+ "SQL statements recording an upgrade performed by hand from `migrate print` output: the same rows the exec listener,+ [[record-legacy-version-tracking!]] and [[ensure-version-tracking!]] write when Metabase itself runs the+ migrations -- the version table, the one-time backfill of the pre-upgrade versions (see [[backfill-versions]]),+ the upgrading version, and the `vNN.legacy-version-tracking` marker for older binaries. Without them a manual+ upgrade would silently disable downgrade detection for both older binaries (which read the marker) and newer ones+ (which read `databasechangelog_version`).++ Read-only: `migrate print` must not mutate the database, so the CREATE TABLE is part of the returned SQL.+ `changelog-major` is the highest `vNN.` major in the changelog being printed: once the SQL is applied, the schema+ has those ids too (see [[record-legacy-version-tracking!]] for what the marker depends on)."+ ^String [^Connection conn ^Database database changelog-major]+ (let [changelog (changelog-table database)+ deployment-id (.getDeploymentId (Scope/getCurrentScope))+ version (current-recorded-version)+ major (current-recorded-major)+ backfill (backfill-versions conn database)]+ (str "\n-- Record the Metabase version performing this upgrade\n"+ (table-ddl) ";\n"+ (str/join (for [[deployment-id version] backfill]+ (str (render-sql (version-row-statement deployment-id version true)) ";\n")))+ (render-sql (version-row-statement deployment-id version true)) ";\n"+ (when major+ (let [mmajor (marker-major major (max changelog-major (highest-real-legacy-major conn database)))]+ (str (render-sql (marker-delete-statement changelog mmajor)) ";\n"+ (render-sql (legacy-marker-statement changelog mmajor deployment-id)) ";\n"))))))diff --git a/src/metabase/app_db/setup.clj b/src/metabase/app_db/setup.cljindex e04a76327f7..46a83558e74 100644--- a/src/metabase/app_db/setup.clj+++ b/src/metabase/app_db/setup.clj@@ -18,6 +18,8 @@[metabase.app-db.encryption :as mdb.encryption][metabase.app-db.jdbc-protocols :as mdb.jdbc-protocols][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.rollback :as rollback]+ [metabase.app-db.liquibase.versions :as versions][metabase.app-db.setting :as mdb.setting][metabase.config.core :as config][metabase.funnysql.core :as funnysql]@@ -89,12 +91,15 @@;; Releasing the locks does not depend on the changesets, so we skip this step as it might require locking.(when-not (= :release-locks direction)(liquibase/consolidate-liquibase-changesets! conn liquibase))+ ;; The version bookkeeping every other direction reads and writes; `print` must not touch the database.+ (when-not (#{:print :release-locks} direction)+ (versions/ensure-version-tracking! conn (.getDatabase liquibase)))(log/info "Liquibase is ready.")(case direction:up (liquibase/migrate-up-if-needed! liquibase data-source):force (liquibase/force-migrate-up-if-needed! liquibase data-source)- :down (apply liquibase/rollback-major-version! conn liquibase false args)- :down-force (apply liquibase/rollback-major-version! conn liquibase true args)+ :down (apply rollback/rollback-major-version! conn liquibase false args)+ :down-force (apply rollback/rollback-major-version! conn liquibase true args):print (print-migrations-and-quit-if-needed! liquibase data-source):release-locks (liquibase/force-release-locks! liquibase));; Migrations were successful; commit everything and re-enable auto-commit@@ -217,26 +222,49 @@(log/info (u/format-color 'cyan "Checking if a database downgrade is required..."))(with-open [conn (.getConnection ^javax.sql.DataSource data-source)](liquibase/with-liquibase [liquibase conn]- (let [latest-available (liquibase/latest-available-major-version liquibase)- latest-applied (liquibase/latest-applied-major-version conn (.getDatabase liquibase))]- ;; `latest-applied` will be `nil` for fresh installs- (when (and latest-applied (< latest-available latest-applied))- (let [later-changesets (liquibase/changesets-from-later-version conn (.getDatabase liquibase) latest-available latest-applied)]- (log/warn (u/format-color 'red "Database has migrations from v%d but this binary only knows up to v%d:"- latest-applied latest-available))- (doseq [cs later-changesets]- (log/warn (u/format-color 'red " - %s" cs))))+ (let [database (.getDatabase liquibase)+ _ (versions/ensure-version-tracking! conn database)+ ;; this binary's own major (for dev builds this is the synthetic version, so a dev DB doesn't look+ ;; 'downgraded' against itself).+ current-major (versions/current-recorded-major)+ ;; The version that ran the deployment the schema is currently at+ recorded-version (versions/last-deployment-version conn database)+ recorded-major (some-> recorded-version versions/version->major)]+ (cond+ ;; nothing recorded yet (fresh install), or the DB is at/behind us -> fine+ (or (nil? recorded-major) (>= current-major recorded-major))+ nil++ ;; The DB was last migrated by a *development* build: it may carry un-released migrations this binary knows+ ;; nothing about. Refuse, pointing at the development tooling that can roll those deployments back.+ (versions/synthetic-dev-major? recorded-major)(throw (ex-info- (str (u/format-color 'red (trs "ERROR: Downgrade detected."))+ (str (u/format-color 'red (trs "ERROR: Development build detected."))"\n\n"- (trs "Your metabase instance appears to have been downgraded without a corresponding database downgrade.")+ (trs "This database was last migrated by a development build ({0}), which this version (v{1}) cannot use." recorded-version current-major)"\n\n"- (trs "You must run `java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar migrate down` from version {0}." latest-applied)- "\n\n"- (trs "Once your database has been downgraded, try running the application again.")- "\n\n"- (trs "See: https://www.metabase.com/docs/latest/installation-and-operation/upgrading-metabase#rolling-back-an-upgrade-or-to-an-older-version"))- {})))))))+ (trs "Roll back its development deployments from a development checkout with `clojure -M:dev:migrate rollback last-deployment` (dev.migrate/rollback!), or rebuild the database."))+ {}))++ ;; A genuine downgrade: a newer *released* version migrated this database.+ :else+ (do+ (let [later-changesets (versions/changesets-from-later-version conn database current-major recorded-major)]+ (log/warn (u/format-color 'red "Database has migrations from v%d but this binary only knows up to v%d:"+ recorded-major current-major))+ (doseq [cs later-changesets]+ (log/warn (u/format-color 'red " - %s" cs))))+ (throw (ex-info+ (str (u/format-color 'red (trs "ERROR: Downgrade detected."))+ "\n\n"+ (trs "Your metabase instance appears to have been downgraded without a corresponding database downgrade.")+ "\n\n"+ (trs "You must run `java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar migrate down` from version {0}." recorded-major)+ "\n\n"+ (trs "Once your database has been downgraded, try running the application again.")+ "\n\n"+ (trs "See: https://www.metabase.com/docs/latest/installation-and-operation/upgrading-metabase#rolling-back-an-upgrade-or-to-an-older-version"))+ {}))))))))(mu/defn- run-schema-migrations!"Run through our DB migration process and make sure DB is fully prepared"diff --git a/src/metabase/sync/sync_metadata/tables.clj b/src/metabase/sync/sync_metadata/tables.cljindex 1f269f9b06c..3049c84d5db 100644--- a/src/metabase/sync/sync_metadata/tables.clj+++ b/src/metabase/sync/sync_metadata/tables.clj@@ -79,6 +79,8 @@;; Liquibase#"^databasechangelog$"#"^databasechangeloglock$"+ #"^databasechangelog_version$"+ #"^databasechangelog_first_run$";; Lobos#"^lobos_migrations$";; MSSQLdiff --git a/test/metabase/app_db/force_migration_test.clj b/test/metabase/app_db/force_migration_test.cljindex 7ab18d77222..6d135dfe3c2 100644--- a/test/metabase/app_db/force_migration_test.clj+++ b/test/metabase/app_db/force_migration_test.clj@@ -1,5 +1,6 @@(ns ^:mb/app-db-migrations-test metabase.app-db.force-migration-test(:require+ [clojure.string :as str][clojure.test :refer :all][metabase.app-db.core :as mdb][metabase.app-db.custom-migrations :as custom-migrations]@@ -48,8 +49,10 @@(.acquireLock lock-service))(db.setup/migrate! data-source :force)(testing "Make sure the migrations that intended to succeed are succeed"+ ;; ignoring the `vNN.legacy-version-tracking` marker row the migration run records for older binaries(is (= ["1" "2" "5"]- (t2/select-pks-vec (@#'liquibase/changelog-table-name conn) {:order-by [:dateexecuted :id]}))))+ (->> (t2/select-pks-vec (@#'liquibase/changelog-table-name conn) {:order-by [:dateexecuted :id]})+ (remove #(str/ends-with? % ".legacy-version-tracking"))))))(testing "the custom migration that fails doesn't commit its operation"(is (nil? (t2/select-one :ancient_civilization :name "Greek"))))(testing "the custom migration that success will persists it result successfully"diff --git a/test/metabase/app_db/liquibase/rollback_test.clj b/test/metabase/app_db/liquibase/rollback_test.cljnew file mode 100644index 00000000000..fa227cb7cb7--- /dev/null+++ b/test/metabase/app_db/liquibase/rollback_test.clj@@ -0,0 +1,908 @@+(ns ^:mb/driver-tests metabase.app-db.liquibase.rollback-test+ (:require+ [clojure.java.jdbc :as jdbc]+ [clojure.test :refer :all]+ [metabase.app-db.core :as mdb]+ [metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.rollback :as rollback]+ [metabase.app-db.liquibase.versions :as versions]+ [metabase.app-db.schema-migrations-test.impl :as schema-migrations.impl]+ [metabase.app-db.test-util :as mdb.test-util]+ [metabase.config.core :as config]+ [metabase.driver :as driver]+ [metabase.test :as mt]+ [metabase.util :as u])+ (:import+ (liquibase.changelog ChangeSet)))++(set! *warn-on-reflection* true)++(defn- tag [s] (assoc config/mb-version-info :tag s))++(defn- changelog-ids+ "The changeset ids in the changelog, in execution order, without the `vNN.legacy-version-tracking` markers (every run+ and rollback re-records one; see [[versions/record-legacy-version-tracking!]])."+ [conn changelog-table]+ (into []+ (comp (remove versions/legacy-version-tracking-marker?) (map :id))+ (jdbc/query {:connection conn}+ [(format "SELECT id, filename FROM %s ORDER BY dateexecuted, orderexecuted" changelog-table)])))++(defn- marker-ids+ "The `vNN.legacy-version-tracking` marker ids in the changelog, in id order."+ [conn changelog-table]+ (mapv :id (jdbc/query {:connection conn}+ [(format "SELECT id FROM %s WHERE filename = 'legacy-version-tracking' ORDER BY id" changelog-table)])))++(defn- has-changeset? [conn changelog-table id]+ (boolean (seq (jdbc/query {:connection conn} [(format "SELECT 1 FROM %s WHERE id = ?" changelog-table) id]))))++(defn- deployment-of [spec changelog-table id]+ (:deployment_id (first (jdbc/query spec [(format "SELECT deployment_id FROM %s WHERE id = ?" changelog-table) id]))))++(defn- changelog-deployments+ "The set of deployment_ids with any changelog row."+ [conn changelog-table]+ (set (map :deployment_id (jdbc/query {:connection conn} [(format "SELECT DISTINCT deployment_id FROM %s" changelog-table)]))))++(defn- version-deployments+ "The set of deployment_ids with any version row."+ [conn]+ (set (map :deployment_id (jdbc/query {:connection conn}+ [(format "SELECT DISTINCT deployment_id FROM %s" (versions/versions-table-name))]))))++(defn- versions-recorded+ "Every recorded version string, in insertion order, read on a fresh connection."+ []+ (mapv :metabase_version (jdbc/query {:datasource (mdb/data-source)}+ [(format "SELECT metabase_version FROM %s ORDER BY id" (versions/versions-table-name))])))++(defn- rollback-last-deployment!+ "What `dev.migrate/rollback! :last-deployment` does: roll back everything after the second-newest deployment, in+ its own transaction. Returns the boundary deployment id, or nil when there was nothing earlier to roll back to."+ []+ (with-open [conn (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (.setAutoCommit conn false)+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ _ (versions/ensure-version-tracking! conn db)+ boundary (versions/previous-deployment-id conn db)]+ (when boundary+ (rollback/rollback-to-deployment! conn liquibase boundary))+ (.commit conn)+ boundary))))++;;; ---------------------------------------------- target resolution ----------------------------------------------++(deftest rollback-target-resolution-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; a prior-major deployment, then two current-major ones+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d99" :ran "x.99.0.0" :changesets ["z99"]}+ {:deployment "d100" :ran "x.100.5.0" :changesets ["aaa111" "bbb222"]}+ {:deployment "d101a" :ran "x.101.1.0" :changesets ["ccc333" "ddd444"]}+ {:deployment "d101b" :ran "x.101.2.3" :changesets ["eee555" "fff666"]}])+ (testing "recorded majors in the window resolve"+ (is (= 101 (rollback/resolve-rollback-target conn db "101")) "the current major")+ (is (= 100 (rollback/resolve-rollback-target conn db 100)) "the previous major, as an integer too"))+ (testing "majors outside the window do not, unless forced"+ (is (nil? (rollback/resolve-rollback-target conn db "99")))+ (is (= 99 (rollback/resolve-rollback-target conn db "99" true))))+ (testing "never-recorded majors and point-release targets do not"+ (is (nil? (rollback/resolve-rollback-target conn db "102")))+ (is (nil? (rollback/resolve-rollback-target conn db "101.1.0")))+ (is (nil? (rollback/resolve-rollback-target conn db :nope))))+ (with-redefs [config/mb-version-info (tag "v0.101.2.3")]+ (testing "rolling back to an unsupported target is rejected"+ (is (thrown-with-msg? IllegalArgumentException #"not a valid rollback target"+ (rollback/rollback-major-version! conn liquibase false "101.1.0")))+ (is (thrown-with-msg? IllegalArgumentException #"not a valid rollback target"+ (rollback/rollback-major-version! conn liquibase false "99")))+ (is (= #{"d99" "d100" "d101a" "d101b"} (changelog-deployments conn ct)) "and nothing is dropped"))+ (testing "rolling back to the current major is a no-op: its latest deployment is the boundary"+ (rollback/rollback-major-version! conn liquibase false "101")+ (is (= #{"d99" "d100" "d101a" "d101b"} (changelog-deployments conn ct))))+ (testing "rolling back to the previous major drops every current-major deployment (their changesets are not+ in the changelog file, so this exercises the supplemental delete path)"+ (rollback/rollback-major-version! conn liquibase false "100")+ (is (= #{"d99" "d100"} (changelog-deployments conn ct)))+ (is (= #{"d99" "d100"} (version-deployments conn)) "the dropped deployments' version rows go with them"))))))))++(deftest bare-major-rollback-targets-latest-deployment-of-major-test+ (testing "a bare-major target rolls back to the LATEST deployment of that major, whatever the recorded version format"+ ;; Regression: version strings of different lengths (a backfilled `x.64.0.0` vs a real-tag `x.64.9`) used to be+ ;; compared with Clojure's count-first vector sort, which resolved "64" to 64.0.0 and rolled back the 64.9 hotfix.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; a pre-tracking install backfilled as x.64.0.0, then a 64.9 hotfix deployment, then the 65 upgrade+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64a" :ran "x.64.0.0" :changesets ["c64base"]}+ {:deployment "d64b" :ran "x.64.9" :changesets ["c64hotfix"]}+ {:deployment "d65" :ran "x.65.0" :changesets ["c65"]}])+ (with-redefs [config/mb-version-info (tag "v0.65.0")]+ (rollback/rollback-major-version! conn liquibase false "64"))+ (is (= ["c64base" "c64hotfix"] (changelog-ids conn ct))+ "the 65 changeset was rolled back; the 64.9 hotfix (the latest 64 deployment) is retained")+ (is (= #{"d64a" "d64b"} (version-deployments conn)) "only the 65 version row was removed")))))))++(deftest booted-major-is-a-rollback-target-test+ (testing "a major recorded only as a boot row (it shipped no migrations) is still a valid rollback boundary"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; d56 ran under x.56.0 and was later booted by 57 without migrations; d58 is the current major+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d56" :ran "x.56.0" :booted ["x.57.0"] :changesets ["cs56"]}+ {:deployment "d58" :ran "x.58.0" :changesets ["cs58"]}])+ (with-redefs [config/mb-version-info (tag "v0.58.0")]+ (is (= 57 (rollback/resolve-rollback-target conn db "57"))+ "57 never ran migrations here but is still a recorded boundary")+ (is (= 56 (rollback/resolve-rollback-target conn db "56")))+ (rollback/rollback-major-version! conn liquibase false "57")+ (is (= ["cs56"] (changelog-ids conn ct))+ "the current-major deployment was rolled back and the booted deployment retained"))))))))++(deftest default-rollback-target-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "only" :ran "x.65.0.0" :changesets ["c65"]}])+ (with-redefs [config/mb-version-info (tag "v0.65.0")]+ (testing "with no earlier recorded major the default `migrate down` no-ops rather than throwing"+ (is (nil? (rollback/rollback-major-version! conn liquibase false)))+ (is (= ["c65"] (changelog-ids conn ct))))+ (testing "otherwise it steps back to the previous recorded major, even across a skipped one (63 -> 65)"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d66" :ran "x.66.0" :changesets ["c66"]}])+ (with-redefs [config/mb-version-info (tag "v0.66.0")]+ (rollback/rollback-major-version! conn liquibase false))+ (is (= ["c65"] (changelog-ids conn ct)))+ (is (= ["x.65.0.0"] (versions-recorded))))))))))++(deftest default-rollback-does-not-skip-a-release-without-migrations-test+ (testing "v64 migrate down is a no-op when v64 booted on the schema v63 produced"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d62" :ran "x.62.0" :changesets ["c62"]}+ {:deployment "d63" :ran "x.63.0" :booted ["x.64.0"] :changesets ["c63"]}])+ (with-redefs [config/mb-version-info (tag "v0.64.0")]+ (rollback/rollback-major-version! conn liquibase false))+ (is (= ["c62" "c63"] (changelog-ids conn ct))+ "the default target is v63 at the current deployment, so no schema changes are reversed")))))))++(deftest rollback-force-widens-to-full-history-test+ (testing "force widens the rollback window from the recent history to the full recorded history"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; three majors deployed; 43 is two majors back from current, so it is outside the default window+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d43" :ran "x.43.0.0" :changesets ["c43"]}+ {:deployment "d44" :ran "x.44.0.0" :changesets ["c44"]}+ {:deployment "d45" :ran "x.45.0.0" :changesets ["c45"]}])+ (with-redefs [config/mb-version-info (tag "v0.45.0.0")]+ (testing "without force, a target outside the window is rejected and nothing is dropped"+ (is (thrown-with-msg? IllegalArgumentException #"not a valid rollback target"+ (rollback/rollback-major-version! conn liquibase false "43")))+ (is (= #{"d43" "d44" "d45"} (changelog-deployments conn ct))))+ (testing "with force, the further-back target is reachable and drops the later majors"+ (rollback/rollback-major-version! conn liquibase true "43")+ (is (= #{"d43"} (changelog-deployments conn ct)) "only the major-43 deployment remains")+ (is (= #{"d43"} (version-deployments conn)) "the rolled-back deployments' version rows were also removed")))))))))++(deftest rollback-from-older-binary-guard-test+ (testing "a binary older than the schema refuses to roll back: its changelog cannot reverse the newer changesets,+ so proceeding would delete their bookkeeping and leave the schema silently corrupted"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64" :ran "x.64.0" :changesets ["c64"]}+ ;; a NEWER v65 binary ran a changeset that is NOT in this binary's changelog+ {:deployment "d65" :ran "x.65.0" :changesets ["future65cs"]}])+ (with-redefs [config/mb-version-info (tag "v0.64.0")]+ (testing "default `migrate down` throws instead of silently deleting the newer deployment's history"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo+ #"Cannot downgrade a database at version 65\.0 from Metabase version 64\.0"+ (rollback/rollback-major-version! conn liquibase false))))+ (testing "an explicit target throws too"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo+ #"Cannot downgrade a database at version 65\.0"+ (rollback/rollback-major-version! conn liquibase false "64"))))+ (is (has-changeset? conn ct "future65cs") "nothing was deleted")+ (testing "force still allows it (the operator explicitly accepts the risk)"+ (rollback/rollback-major-version! conn liquibase true "64")+ (is (not (has-changeset? conn ct "future65cs")))))))))))++(deftest rollback-from-older-patch-guard-test+ (testing "an older patch of the schema's major refuses to roll back too: the newer patch's changesets are not in its+ changelog either, so it could only clear their bookkeeping and leave their DDL behind"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64" :ran "x.64.0.0" :changesets ["c64"]}+ {:deployment "d654" :ran "x.65.4" :changesets ["c654"]}+ {:deployment "d655" :ran "x.65.5" :changesets ["future655cs"]}])+ (with-redefs [config/mb-version-info (tag "v1.65.4")]+ (is (thrown-with-msg? clojure.lang.ExceptionInfo+ #"Cannot downgrade a database at version 65\.5 from Metabase version 65\.4"+ (rollback/rollback-major-version! conn liquibase false)))+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"from Metabase version >= 65\.5"+ (rollback/rollback-major-version! conn liquibase false "64")))+ (is (has-changeset? conn ct "future655cs") "nothing was deleted"))+ (testing "the same patch, however its version was written, may roll back"+ (with-redefs [config/mb-version-info (tag "v0.65.5.0")]+ (is (= 64 (rollback/resolve-rollback-target conn db "64")))+ (rollback/rollback-major-version! conn liquibase false "65")+ (is (has-changeset? conn ct "future655cs") "rolling back to the current major is a no-op")))+ (testing "force still allows it"+ (with-redefs [config/mb-version-info (tag "v1.65.4")]+ (rollback/rollback-major-version! conn liquibase true "64")+ (is (not (has-changeset? conn ct "future655cs")))))))))))++(deftest rollback-from-older-patch-after-it-booted-guard-test+ (testing "an older patch that booted the newer patch's schema since still refuses: a re-run (a runAlways changeset,+ or a runOnChange one the patches differ in) recorded the newest deployment as its own, but the newer+ patch's changesets are still in the schema"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64" :ran "x.64.0.0" :changesets ["c64"]}+ {:deployment "d655" :ran "x.65.5" :changesets ["future655cs"]}+ {:deployment "d654b" :ran "x.65.4" :changesets [{:id "always" :exectype "RERAN"}]}])+ (with-redefs [config/mb-version-info (tag "v1.65.4")]+ (is (thrown-with-msg? clojure.lang.ExceptionInfo+ #"Cannot downgrade a database at version 65\.5 from Metabase version 65\.4"+ (rollback/rollback-major-version! conn liquibase false)))+ (is (has-changeset? conn ct "future655cs") "nothing was deleted"))))))))++;;; ------------------------------------------ rolling back real deployments ----------------------------------------++(deftest rollback-to-the-major-before-the-first-backfill-test+ (testing "an install that went 63 -> 64 before version tracking can still `migrate down` to 63 from a 64 patch"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d63" :changesets ["v63.2026-01-01T00:00:00"]}+ {:deployment "d64" :changesets ["v64.2026-05-01T00:00:00"]}])+ (versions/ensure-version-tracking! conn db)+ (with-redefs [config/mb-version-info (tag "v1.64.3")]+ (rollback/rollback-major-version! conn liquibase false))+ (is (= ["v63.2026-01-01T00:00:00"] (changelog-ids conn ct)))+ (is (= #{"d63"} (version-deployments conn)))))))))++(deftest rollback-after-upgrade-backfill-test+ (testing "an install upgraded from 53 to 55 backfills the old major, records the new one, and downgrades back to 53"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ ;; a "v53 install" upgraded over time: an older deployment (the 51->52 upgrade) plus the most recent one+ ;; whose last changeset is a v53.* id -- all from before version tracking, so nothing is recorded yet+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d52older" :changesets ["v51.2023-06-01T00:00:00" "v52.2023-12-01T00:00:00"]}+ {:deployment "d53install" :changesets ["v53.2024-06-01T00:00:00"]}])+ (testing "on 55 startup the backfill records every deployment, the current one from its highest changeset id"+ (versions/ensure-version-tracking! conn db)+ (is (= #{"d52older" "d53install"} (version-deployments conn)))+ (is (= "x.53.0.0" (versions/last-deployment-version conn db))))+ ;; then the 55 upgrade runs as its own deployment and records its version like a normal run+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d55" :ran "x.55.2.0" :changesets ["v55.2024-10-01T00:00:00"]}])+ (with-redefs [config/mb-version-info (tag "v0.55.2.0")]+ (testing "53 is a valid rollback target (the previous major)"+ (is (= 53 (rollback/resolve-rollback-target conn db "53"))))+ (testing "downgrading to 53 rolls back the 55 deployment and keeps the rest of history"+ (rollback/rollback-major-version! conn liquibase false "53")+ (is (= #{"d52older" "d53install"} (changelog-deployments conn ct))+ "the 53 install's changesets (including older unrecorded history) are retained")+ (is (= #{"d52older" "d53install"} (version-deployments conn)) "the 55 version row was deleted")))))))))++(deftest migrate-up-backfills-pre-tracking-install-test+ (testing "the first `migrate up` on an install that predates version tracking backfills the pre-upgrade version,+ so `migrate down` still has a boundary to roll back to (CLI upgrades skip the boot-time checks)"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)+ all-ids (mapv (fn [^ChangeSet cs] (.getId cs))+ (.getChangeSets (.getDatabaseChangeLog liquibase)))+ last-id (peek all-ids)+ stop-id (nth all-ids (- (count all-ids) 2))]+ ;; install everything except the last changeset, as a pre-version-tracking binary would have left it+ (schema-migrations.impl/run-migrations-in-range! conn ["v00.00-000" stop-id])+ (mdb.test-util/age-changelog-rows! {:datasource (mdb/data-source)} ct)+ ;; ... which had no version rows at all. The DELETE and all later reads use fresh autocommit connections:+ ;; on MySQL the shared non-autocommit `conn` would leave the DELETE invisible to the migration's own+ ;; connection and serve later reads from a stale REPEATABLE READ snapshot.+ (jdbc/execute! {:datasource (mdb/data-source)} [(format "DELETE FROM %s" (versions/versions-table-name))])+ (let [pre-major (versions/latest-applied-major-version conn db)]+ (with-redefs [config/mb-version-info (tag (format "v0.%d.0" (inc pre-major)))]+ (testing "migrate up records both the backfilled pre-upgrade version and the upgrade's own version"+ (mdb/migrate! (mdb/data-source) :up)+ (is (= (versions/major->version pre-major) (first (versions-recorded))))+ (is (= (format "x.%d.0" (inc pre-major)) (last (versions-recorded)))))+ (testing "migrate down (default) rolls the upgrade back"+ (mdb/migrate! (mdb/data-source) :down)+ (is (empty? (jdbc/query {:datasource (mdb/data-source)} [(format "SELECT 1 FROM %s WHERE id = ?" ct) last-id]))+ "the upgrade's changeset was rolled back")+ (is (= (versions/major->version pre-major) (first (versions-recorded))))+ (is (not-any? #{(format "x.%d.0" (inc pre-major))} (versions-recorded))+ "the upgrade's version row was removed"))))))))))++(deftest release-migrate-down-reverses-version-less-ddl-test+ (testing "the release `migrate down` reverses a version-less changeset's DDL, not just its bookkeeping"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [liquibase/changelog-file "versionless-dev-run1.yaml"]+ ;; reads use fresh autocommit connections (see migrate-up-backfills-pre-tracking-install-test)+ (let [spec {:datasource (mdb/data-source)}+ table? (fn [table]+ (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ ;; Liquibase quotes identifiers, so the stored case varies by database+ (or (versions/table-exists? table c)+ (versions/table-exists? (u/upper-case-en table) c))))]+ (with-redefs [config/mb-version-info (tag "v0.65.0")]+ (mdb/migrate! (mdb/data-source) :up))+ (is (true? (table? "dev_run_a_table")))+ (mdb.test-util/age-changelog-rows! spec (liquibase/changelog-table-name conn))+ (with-redefs [liquibase/changelog-file "versionless-dev-run2.yaml"+ config/mb-version-info (tag "v0.66.0")]+ (mdb/migrate! (mdb/data-source) :up)+ (is (true? (table? "dev_run_b_table")))+ (is (= ["x.65.0" "x.66.0"] (versions-recorded)))+ (mdb/migrate! (mdb/data-source) :down)+ (is (false? (table? "dev_run_b_table")) "66's table was dropped")+ (is (true? (table? "dev_run_a_table")) "65's table survives")+ (is (= ["x.65.0"] (versions-recorded))))))))))++(deftest legacy-version-tracking-removed-by-rollback-test+ (testing "rolling back several version-less majors (68 -> 66) removes the v67 and v68 markers with their deployments, leaving v66"+ ;; No special handling: each marker belongs to the deployment that created it, so delete-deployment-rows! sweeps it up.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)+ markers #(marker-ids conn ct)]+ (versions/ensure-version-tracking! conn db)+ (doseq [major [66 67 68]+ :let [dep (str "d" major)]]+ (mdb.test-util/fabricate-history! conn ct [{:deployment dep :ran (versions/major->version major) :changesets [(str "c" major)]}])+ (versions/record-legacy-version-tracking! db major dep))+ (is (= ["v66.legacy-version-tracking" "v67.legacy-version-tracking" "v68.legacy-version-tracking"] (markers)))+ (is (= 68 (versions/latest-applied-major-version conn db)) "older binaries see 68 before the downgrade")+ (with-redefs [config/mb-version-info (tag "v0.68.0")]+ ;; 66 is two majors back, so it is outside the default window -- force widens it to the full history+ (rollback/rollback-major-version! conn liquibase true "66"))+ (is (= ["v66.legacy-version-tracking"] (markers)))+ (is (= 66 (versions/latest-applied-major-version conn db)) "older binaries now correctly see 66")+ (is (= #{"d66"} (changelog-deployments conn ct)) "only the v66 deployment remains")))))))++(deftest rollback-failure-preserves-history-test+ (testing "a rollback that cannot reverse a changeset throws and leaves the deployment's history intact"+ ;; Otherwise we would clear the deployment's rows -- including its legacy-version-tracking row -- while the schema+ ;; is still (partly) at the higher major, so an older binary would read the lower major and start against it.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d66" :ran "x.66.0.0" :changesets ["c66"]}+ {:deployment "d67" :ran "x.67.0.0" :changesets ["c67"]}])+ (versions/record-legacy-version-tracking! db 67 "d67")+ (with-redefs [config/mb-version-info (tag "v0.67.0")+ ;; simulate Liquibase failing to reverse one of the changesets+ rollback/run-liquibase-rollback! (fn [& _] ["c67"])]+ (let [e (is (thrown-with-msg? clojure.lang.ExceptionInfo #"could not roll back changeset"+ (rollback/rollback-major-version! conn liquibase false "66")))]+ (if (= driver/*driver* :postgres)+ (is (re-find #"left unchanged" (ex-message e))+ "transactional DDL: the transaction rollback really does leave the database unchanged")+ (is (re-find #"may (already )?have been committed" (ex-message e))+ "auto-committing DDL: the error must not claim the database is unchanged"))))+ (testing "the deployment's history is left alone, so the changelog does not claim a rollback that failed"+ (is (has-changeset? conn ct "c67"))+ (is (has-changeset? conn ct "v67.legacy-version-tracking"))+ (is (= 67 (versions/latest-applied-major-version conn db))+ "older binaries still see 67 -- the schema really is still at 67"))))))))++(deftest ^:parallel rollback-failure-message-test+ (testing "the rollback-failure message only claims 'left unchanged' where transactional DDL makes that true"+ (let [msg (fn [db-type] (#'rollback/rollback-failure-message db-type "64" ["c1" "c2"]))]+ (testing "every variant names the failed changesets"+ (doseq [db-type [:postgres :h2 :mysql]]+ (is (re-find #"could not roll back changeset\(s\) c1, c2" (msg db-type)) db-type)))+ (testing "Postgres rolls DDL back transactionally, so the database really is unchanged"+ (is (re-find #"left unchanged" (msg :postgres))))+ (testing "H2 and MySQL auto-commit DDL, so earlier steps may have persisted"+ (doseq [db-type [:h2 :mysql]]+ (is (not (re-find #"left unchanged" (msg db-type))) db-type)+ (is (re-find #"may (already )?have been committed" (msg db-type)) db-type))))))++(deftest rollback-logs-rows-it-cannot-reverse-test+ (testing "rows cleared by a rollback without a matching changeset to reverse are called out, not silently dropped"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64" :ran "x.64.0.0" :changesets ["base"]}+ {:deployment "d65" :ran "x.65.0.0" :changesets ["unknown_to_this_changelog"]}])+ (with-redefs [config/mb-version-info (tag "v0.65.0")]+ (mt/with-log-messages-for-level [messages [metabase.app-db.liquibase.rollback :info]]+ (rollback/rollback-major-version! conn liquibase false "64")+ (is (some #(re-find #"could not be reversed.*unknown_to_this_changelog" (:message %)) (messages))+ "names the row whose DDL (if any) is now orphaned")))+ (is (not (has-changeset? conn ct "unknown_to_this_changelog"))+ "its bookkeeping row is still cleared, as before")))))))++;;; ---------------------------------------------- the development rollback ----------------------------------------++(deftest rollback-to-deployment-test+ (testing "rollback-to-deployment! rolls back everything after the given deployment, whatever versions are recorded"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; three dev "restarts", each its own deployment at the same constant dev version+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "dev1" :ran versions/dev-version :changesets ["base01"]}+ {:deployment "dev2" :ran versions/dev-version :changesets ["mig02"]}+ {:deployment "dev3" :ran versions/dev-version :changesets ["newmig03"]}])+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")]+ (testing "the release rollback refuses a dev-migrated schema and points at the dev tooling"+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"development build.*dev.migrate"+ (rollback/rollback-major-version! conn liquibase false)))+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"development build.*dev.migrate"+ (rollback/rollback-major-version! conn liquibase true "9999"))+ "even with force and an explicit target")+ (is (= ["base01" "mig02" "newmig03"] (changelog-ids conn ct)) "and changed nothing"))+ (testing "previous-deployment-id is the second-newest deployment"+ (is (= "dev2" (versions/previous-deployment-id conn db))))+ (rollback/rollback-to-deployment! conn liquibase "dev2")+ (is (= ["base01" "mig02"] (changelog-ids conn ct)) "only the newest deployment was rolled back")+ (is (= #{"dev1" "dev2"} (version-deployments conn)) "the rolled-back deployment's version row is gone")+ (testing "a second step back"+ (rollback/rollback-to-deployment! conn liquibase (versions/previous-deployment-id conn db))+ (is (= ["base01"] (changelog-ids conn ct)))+ (is (= #{"dev1"} (version-deployments conn))))+ (testing "with only one deployment left there is nothing earlier"+ (is (nil? (versions/previous-deployment-id conn db))))+ (testing "an unknown deployment id is refused"+ (is (thrown-with-msg? IllegalArgumentException #"nope.*not a deployment"+ (rollback/rollback-to-deployment! conn liquibase "nope")))))))))))++(deftest rollback-to-deployment-onto-a-release-install-test+ (testing "rolling a dev deployment back onto a release install leaves a schema the release rollback accepts again"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; a real v65 install, then a developer points a dev build at it and runs a new migration+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d65" :ran "x.65.0.0" :changesets ["real65"]}+ {:deployment "dev1" :ran versions/dev-version :changesets ["devmig"]}])+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")]+ (is (= "d65" (versions/previous-deployment-id conn db)))+ (rollback/rollback-to-deployment! conn liquibase "d65")+ (is (= ["real65"] (changelog-ids conn ct)) "the dev deployment was rolled back, the real install kept")+ (is (= 65 (versions/current-schema-major conn db)))+ (testing "the schema is at a real major again, so the release rollback works (nothing earlier here: no-op)"+ (is (nil? (rollback/rollback-major-version! conn liquibase false)))))))))))++(deftest rollback-to-deployment-repairs-rewritten-filenames-test+ (testing "the deployment rollback repairs version-less filenames an older binary rewrote, without relying on the+ release migrate! path having consolidated first (dev.migrate/rollback! does not go through migrate!)"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [liquibase/changelog-file "versionless-dev-run1.yaml"]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; an earlier dev deployment to roll back to, then the dev changeset itself+ (mdb.test-util/fabricate-history! conn ct [{:deployment "dev1" :ran versions/dev-version :changesets ["boundary_row"]}])+ (liquibase/with-scope-locked liquibase (.update liquibase ""))+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET deployment_id = 'dev2' WHERE id = 'dev_run_a'" ct)])+ (versions/record-deployment-version! conn "dev2" versions/dev-version true)+ ;; what an old binary's consolidation leaves behind+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET filename = 'migrations/000_legacy_migrations.yaml' WHERE id = 'dev_run_a'" ct)])+ (mt/with-log-messages-for-level [messages [metabase.app-db.liquibase :info]]+ (rollback/rollback-to-deployment! conn liquibase "dev1")+ (is (some #(re-find #"Restoring the changelog filename of version-less changeset dev_run_a" (:message %)) (messages)))+ (is (not-any? #(re-find #"could not be reversed" (:message %)) (messages))))+ (is (false? (versions/table-exists? "DEV_RUN_A_TABLE" conn)) "the DDL was reversed, not orphaned"))))))))++(deftest dev-consecutive-migration-runs-are-separate-deployments-test+ (testing "two `migrate up` runs in ONE process (a long-lived dev REPL) create separate deployments, so `migrate down`+ reverts exactly the last run"+ ;; Regression: the deployment id was generated once per process (Liquibase root Scope), so a fresh install plus a+ ;; later dev migration in the same JVM merged into a single deployment -- `migrate down` then had no earlier+ ;; boundary and could not undo the added migration.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")]+ ;; reads use fresh autocommit connections: on MySQL the shared non-autocommit `conn` would serve reads+ ;; after the second migration from a stale REPEATABLE READ snapshot+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ applied? (fn [id] (boolean (seq (jdbc/query spec [(format "SELECT 1 FROM %s WHERE id = ?" ct) id]))))]+ (with-redefs [liquibase/changelog-file "versionless-dev-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (is (true? (applied? "dev_run_a")))+ (is (= [versions/dev-version] (versions-recorded)))+ (mdb.test-util/age-changelog-rows! spec ct)+ ;; the developer adds a migration and runs `migrate up` again in the same process+ (with-redefs [liquibase/changelog-file "versionless-dev-run2.yaml"]+ (mdb/migrate! (mdb/data-source) :up)+ (is (true? (applied? "dev_run_b")))+ (is (= [versions/dev-version versions/dev-version] (versions-recorded))+ "the second run records the same constant dev version against its own deployment")+ (is (not= (deployment-of spec ct "dev_run_a") (deployment-of spec ct "dev_run_b"))+ "each run is its own deployment, even within one process")+ (is (thrown-with-msg? clojure.lang.ExceptionInfo #"development build.*dev.migrate"+ (mdb/migrate! (mdb/data-source) :down))+ "the release `migrate down` refuses a dev-migrated schema and points at the dev tooling")+ (is (true? (applied? "dev_run_b")) "...without touching anything")+ (is (= (deployment-of spec ct "dev_run_a") (rollback-last-deployment!)))+ (is (false? (applied? "dev_run_b")) "the dev rollback reverts exactly the second run")+ (is (true? (applied? "dev_run_a")) "the first run's migration survives")+ (is (= [versions/dev-version] (versions-recorded)) "the second run's version row is gone"))))))))++;;; ------------------------------------------------- re-run changesets --------------------------------------------++(deftest rollback-does-not-reverse-reran-changesets-test+ (testing "migrate down must not reverse changesets that merely re-ran (RERAN) in the newer deployment"+ ;; Liquibase moves a re-executed changeset (runOnChange whose checksum changed, or anything re-run by `migrate+ ;; force`) into the CURRENT run's deployment: new deployment_id, new dateexecuted, exectype RERAN. Those rows are+ ;; positioned after the rollback boundary, but reversing them would strip schema objects that the rollback target+ ;; still needs. They must instead be retained and reassigned to the boundary deployment, so the deployment+ ;; bookkeeping stays consistent.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ ;; all reads go through a fresh autocommit connection: a read on the shared non-autocommit `conn` leaves an+ ;; open transaction whose lock on v_roc deadlocks the next migration's CREATE OR REPLACE VIEW on Postgres+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, exectype FROM %s WHERE id = ?" ct) id])))+ view-x (fn [] (:x (first (jdbc/query spec ["SELECT x FROM v_roc"]))))]+ (with-redefs [config/mb-version-info (tag "v0.65.0")+ liquibase/changelog-file "versionless-roc-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (is (= 1 (view-x)))+ (mdb.test-util/age-changelog-rows! spec ct)+ (with-redefs [config/mb-version-info (tag "v0.66.0")+ liquibase/changelog-file "versionless-roc-run2.yaml"]+ (mdb/migrate! (mdb/data-source) :up)+ (is (= 2 (view-x)))+ (is (= "RERAN" (:exectype (row-of "roc_view"))) "sanity: the edited runOnChange changeset re-ran")+ (is (= (:deployment_id (row-of "roc_new")) (:deployment_id (row-of "roc_view")))+ "sanity: Liquibase moved the re-run changeset into the newer deployment")+ (mdb/migrate! (mdb/data-source) :down)+ (testing "the newer release's own changeset is reversed"+ (is (nil? (row-of "roc_new")))+ (is (thrown? Exception (jdbc/query spec ["SELECT 1 FROM roc_new_table"]))))+ (testing "the re-run changeset is NOT reversed"+ (is (= 2 (view-x)) "the view survives (the rollback-target binary's own runOnChange update will restore its content)")+ (is (some? (row-of "roc_view")) "its changelog row is retained"))+ (testing "bookkeeping is consistent after the rollback"+ (is (= (:deployment_id (row-of "roc_base")) (:deployment_id (row-of "roc_view")))+ "the retained row was reassigned to the boundary deployment")+ (is (= ["x.65.0"] (versions-recorded)) "the newer deployment's version row is gone")))+ (testing "booting the rollback-target binary heals the view content via normal runOnChange semantics"+ (with-redefs [config/mb-version-info (tag "v0.65.0")+ liquibase/changelog-file "versionless-roc-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (is (= 1 (view-x)))))))))++(deftest rollback-reverses-reran-changesets-newer-than-the-target-test+ (testing "a re-run changeset whose vNN. major is above the rollback target did not exist there, so it is reversed:+ keeping it would leave 65's view over the 65 column the rollback drops"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ ;; reads use fresh autocommit connections -- see rollback-does-not-reverse-reran-changesets-test+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, exectype FROM %s WHERE id = ?" ct) id])))+ view? (fn [] (try (jdbc/query spec ["SELECT x FROM v_vroc"]) true (catch Exception _ false)))]+ ;; the 64 install that 65.0 upgrades+ (with-redefs [liquibase/changelog-file "versioned-roc-run1.yaml"]+ (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (liquibase/with-liquibase [liquibase c]+ (versions/ensure-version-tracking! c (.getDatabase liquibase))+ (mdb.test-util/fabricate-history! c ct [{:deployment "d64" :ran "x.64.0" :changesets ["base64"]}]))))+ (with-redefs [config/mb-version-info (tag "v0.65.0")+ liquibase/changelog-file "versioned-roc-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (mdb.test-util/age-changelog-rows! spec ct)+ (with-redefs [config/mb-version-info (tag "v0.65.1")+ liquibase/changelog-file "versioned-roc-run2.yaml"]+ (mdb/migrate! (mdb/data-source) :up)+ (is (= "RERAN" (:exectype (row-of "v65.roc-view"))) "sanity: the edited view re-ran in 65.1")+ (mdb/migrate! (mdb/data-source) :down)+ (is (false? (view?)) "the view is reversed with the rest of 65")+ (is (nil? (row-of "v65.roc-view")))+ (is (nil? (row-of "v65.roc-column")))+ (is (= ["x.64.0"] (versions-recorded)))))))))++(deftest rollback-keeps-reran-changesets-of-the-target-major-test+ (testing "a re-run changeset whose vNN. major is at or below the target existed there, so it is kept"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64" :ran "x.64.0" :changesets ["v64.base"]}+ {:deployment "d65" :ran "x.65.0" :changesets ["v65.new"+ {:id "v64.view" :exectype "RERAN"}+ {:id "v65.view" :exectype "RERAN"}]}])+ (with-redefs [config/mb-version-info (tag "v0.65.0")]+ (rollback/rollback-major-version! conn liquibase false "64"))+ (is (= "d64" (deployment-of {:connection conn} ct "v64.view")) "64's re-run view is kept on 64's deployment")+ (is (not (has-changeset? conn ct "v65.view")) "65's re-run view goes with the rest of 65")+ (is (not (has-changeset? conn ct "v65.new")))+ (is (= #{"d64"} (version-deployments conn)))))))))++(defn- migrate-late-roc-history!+ "Fabricate a deployment `d0` run by `before`, then upgrade it with `versionless-late-roc-run1.yaml` as version+ `first-tag` and re-run its view with `versionless-late-roc-run2.yaml` as `second-tag`. Returns fns reading the view's+ changelog row and whether the view exists."+ [ct before first-tag second-tag]+ (let [spec {:datasource (mdb/data-source)}]+ (with-redefs [liquibase/changelog-file "versionless-late-roc-run1.yaml"]+ (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (liquibase/with-liquibase [liquibase c]+ (versions/ensure-version-tracking! c (.getDatabase liquibase))+ (mdb.test-util/fabricate-history! c ct [{:deployment "d0" :ran before :changesets ["base"]}]))))+ (with-redefs [config/mb-version-info (tag first-tag)+ liquibase/changelog-file "versionless-late-roc-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (mdb.test-util/age-changelog-rows! spec ct)+ (with-redefs [config/mb-version-info (tag second-tag)+ liquibase/changelog-file "versionless-late-roc-run2.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ {:row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, exectype FROM %s WHERE id = ?" ct) id])))+ :view? (fn [] (try (jdbc/query spec ["SELECT x FROM v_lroc"]) true (catch Exception _ false)))}))++(deftest rollback-reverses-version-less-reran-changesets-newer-than-the-target-test+ (testing "a version-less re-run changeset that first ran after the rollback target is reversed: where it first ran is+ recorded when it re-runs"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (let [ct (liquibase/changelog-table-name conn)+ {:keys [row-of view?]} (migrate-late-roc-history! ct "x.64.0" "v0.65.0" "v0.65.1")]+ (is (= "RERAN" (:exectype (row-of "late_roc_view"))) "sanity: the edited view re-ran in 65.1")+ (with-redefs [config/mb-version-info (tag "v0.65.1")+ liquibase/changelog-file "versionless-late-roc-run2.yaml"]+ (mdb/migrate! (mdb/data-source) :down))+ (is (false? (view?)) "the view is reversed with the rest of 65")+ (is (nil? (row-of "late_roc_view")))+ (is (nil? (row-of "late_roc_column")))+ (is (= ["x.64.0"] (versions-recorded)))+ (is (empty? (versions/first-runs (.getConnection ^javax.sql.DataSource (mdb/data-source))))+ "the reversed changeset's first run is forgotten with it"))))))++(deftest rollback-to-deployment-reverses-reran-changesets-after-the-boundary-test+ (testing "the development rollback, where every deployment records the same version, also reverses a re-run+ changeset that first ran after the boundary deployment"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (let [ct (liquibase/changelog-table-name conn)+ {:keys [row-of view?]} (migrate-late-roc-history! ct versions/dev-version "vLOCAL_DEV" "vLOCAL_DEV")]+ (is (= "RERAN" (:exectype (row-of "late_roc_view"))) "sanity: the edited view re-ran")+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")+ liquibase/changelog-file "versionless-late-roc-run2.yaml"]+ (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (.setAutoCommit c false)+ (liquibase/with-liquibase [liquibase c]+ (versions/ensure-version-tracking! c (.getDatabase liquibase))+ (rollback/rollback-to-deployment! c liquibase "d0"))+ (.commit c)))+ (is (false? (view?)))+ (is (nil? (row-of "late_roc_column")))+ (is (= #{"d0"} (changelog-deployments (.getConnection ^javax.sql.DataSource (mdb/data-source)) ct))))))))++(deftest rollback-reverses-run-always-changesets-newer-than-the-target-test+ (testing "a runAlways changeset that first ran after the rollback target is reversed, though every run re-ran it"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, exectype FROM %s WHERE id = ?" ct) id])))+ view? (fn [] (try (jdbc/query spec ["SELECT x FROM v_always"]) true (catch Exception _ false)))]+ (with-redefs [liquibase/changelog-file "versionless-always.yaml"]+ (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (liquibase/with-liquibase [liquibase c]+ (versions/ensure-version-tracking! c (.getDatabase liquibase))+ (mdb.test-util/fabricate-history! c ct [{:deployment "d65" :ran "x.65.0" :changesets ["base65"]}]))))+ (with-redefs [config/mb-version-info (tag "v0.66.0")+ liquibase/changelog-file "versionless-always.yaml"]+ (mdb/migrate! (mdb/data-source) :up)+ (mdb.test-util/age-changelog-rows! spec ct)+ (mdb/migrate! (mdb/data-source) :up)+ (is (= "RERAN" (:exectype (row-of "always_view"))) "sanity: the second run re-ran the view")+ (mdb/migrate! (mdb/data-source) :down))+ (is (false? (view?)) "the view is reversed with the rest of 66")+ (is (nil? (row-of "always_view")))+ (is (nil? (row-of "always_base")))+ (is (= ["x.65.0"] (versions-recorded))))))))++(defn- re-run-into!+ "Move changeset `id`'s changelog row into `deployment-id` as Liquibase does when it re-runs it: exectype RERAN, at the+ newest position. The newest `dateexecuted` is read through a derived table: MySQL refuses a subquery on the table an+ UPDATE changes."+ [conn changelog-table id deployment-id]+ (jdbc/execute! {:connection conn}+ [(format (str "UPDATE %s SET exectype = 'RERAN', deployment_id = ?, orderexecuted = orderexecuted + 100, "+ "dateexecuted = (SELECT m FROM (SELECT MAX(dateexecuted) AS m FROM %s) newest) WHERE id = ?")+ changelog-table changelog-table)+ deployment-id id]))++(deftest rollback-keeps-re-runs-first-run-by-an-emptied-deployment-test+ (testing "a deployment whose only changeset re-ran later still holds where it first ran: an old 64 binary that re-ran+ a 64 view on its own, then 67 re-ran it, then `migrate down` to 64 keeps the view"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d64" :ran "x.64.0.0" :changesets ["base64"]}+ {:deployment "d64b" :ran "x.64.0.0" :changesets ["view64"]}])+ (doseq [statement (#'versions/first-run-statements conn db ["synthetic.yaml" "test" "view64"])]+ (jdbc/execute! {:connection conn} statement))+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d67" :ran "x.67.1" :changesets ["c67"]}])+ (re-run-into! conn ct "view64" "d67")+ (is (= #{"d64" "d67"} (changelog-deployments conn ct)) "sanity: d64b has no changelog rows left")+ (with-redefs [config/mb-version-info (tag "v0.67.1")]+ (rollback/rollback-major-version! conn liquibase false "64"))+ (is (has-changeset? conn ct "view64") "the view 64 has is kept")+ (is (not (has-changeset? conn ct "c67")))+ (is (contains? (version-deployments conn) "d64b")+ "the emptied deployment keeps its version row: the view's first run still places it")))))))++(deftest rollback-removes-version-rows-of-emptied-deployments-test+ (testing "a rollback removes the version rows of deployments with no changelog rows left -- every runAlways run, and+ any run that only re-ran changesets, empties the deployment of the run before it"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d65" :ran "x.65.0" :changesets ["c65"]}])+ (versions/record-deployment-version! conn "d65ghost" "x.65.0" true)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d66" :ran "x.66.0" :changesets ["c66"]}])+ (versions/record-deployment-version! conn "d66ghost" "x.66.0" true)+ (with-redefs [config/mb-version-info (tag "v0.66.0")]+ (rollback/rollback-major-version! conn liquibase false))+ (is (= #{"d65"} (version-deployments conn)))))))))++(deftest rollback-tells-re-run-changesets-apart-by-file-test+ (testing "first runs are kept per [filename author id]: the same id and author in two files are two changesets"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)+ view-in (fn [filename] {:id "same_view" :filename filename})+ row-in (fn [filename]+ (first (jdbc/query {:connection conn}+ [(format "SELECT deployment_id FROM %s WHERE id = 'same_view' AND filename = ?" ct)+ filename])))]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d64" :ran "x.64.0" :changesets ["base64" (view-in "a.yaml")]}+ {:deployment "d65" :ran "x.65.0" :changesets ["new65" (view-in "b.yaml")]}])+ ;; 65.1 re-runs both: record where each first ran, then move them as Liquibase does+ (doseq [filename ["a.yaml" "b.yaml"]+ statement (#'versions/first-run-statements conn db [filename "test" "same_view"])]+ (jdbc/execute! {:connection conn} statement))+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d651" :ran "x.65.1" :changesets ["new651"]}])+ (re-run-into! conn ct "same_view" "d651")+ (with-redefs [config/mb-version-info (tag "v0.65.1")]+ (rollback/rollback-major-version! conn liquibase false "64"))+ (is (= "d64" (:deployment_id (row-in "a.yaml"))) "the view that first ran in 64 is kept")+ (is (nil? (row-in "b.yaml")) "the view that first ran in 65 goes with 65")+ (is (= #{["a.yaml" "test" "same_view"]} (set (keys (versions/first-runs conn)))))))))))++(deftest rollback-of-reran-only-deployment-test+ (testing "a deployment consisting ONLY of a re-run changeset can still be rolled back: the boundary steps back and+ the deployment's version row is removed, even though there is nothing to reverse"+ ;; The dev flow that produces this: edit just a runOnChange changeset (no new changesets) and re-run migrate up.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ ;; reads use fresh autocommit connections -- see rollback-does-not-reverse-reran-changesets-test+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, exectype FROM %s WHERE id = ?" ct) id])))+ view-x (fn [] (:x (first (jdbc/query spec ["SELECT x FROM v_roc"]))))]+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")]+ (with-redefs [liquibase/changelog-file "versionless-roc-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (is (= [versions/dev-version] (versions-recorded)))+ (mdb.test-util/age-changelog-rows! spec ct)+ (with-redefs [liquibase/changelog-file "versionless-roc-run3.yaml"]+ (mdb/migrate! (mdb/data-source) :up)+ (is (= [versions/dev-version versions/dev-version] (versions-recorded)))+ (is (= "RERAN" (:exectype (row-of "roc_view"))) "sanity: the second run consists of exactly the re-run changeset")+ (rollback-last-deployment!)+ (is (= [versions/dev-version] (versions-recorded)) "the re-run-only deployment dissolves; the boundary steps back")+ (is (= (:deployment_id (row-of "roc_base")) (:deployment_id (row-of "roc_view")))+ "the retained re-run row joins the boundary deployment")+ (is (= 3 (view-x)) "the view is not reversed")+ (is (nil? (rollback-last-deployment!)) "a further rollback with no earlier deployment is a clean no-op")+ (is (= [versions/dev-version] (versions-recorded))))))))))++(deftest rollback-leaves-a-marker-as-the-newest-row-test+ (testing "rows a rollback keeps can be newer than the boundary's marker; old binaries must still read a marker last"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)+ newest #(mapv :id (jdbc/query {:connection conn}+ [(format "SELECT id FROM %s WHERE dateexecuted = (SELECT MAX(dateexecuted) FROM %s)" ct ct)]))]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58" :ran "x.58.9" :changesets ["v58.2025-01-01T00:00:00" "thing_one"]}])+ (versions/record-legacy-version-tracking! db 58 "d58")+ ;; 59.9 re-ran a version-less runOnChange view of 58's, which the rollback keeps on the 58 deployment+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d59" :ran "x.59.9"+ :changesets ["v59.2025-02-01T00:00:00"+ {:id "thing_view" :exectype "RERAN"}]}])+ (versions/record-legacy-version-tracking! db 59 "d59")+ (with-redefs [config/mb-version-info (tag "v0.59.9")]+ (rollback/rollback-major-version! conn liquibase false))+ (is (= "d58" (deployment-of {:connection conn} ct "thing_view")) "the re-run row is kept")+ (is (= ["v56.legacy-version-tracking"] (newest)))+ (is (= "d58" (deployment-of {:connection conn} ct "v56.legacy-version-tracking")))+ (is (= 58 (versions/current-schema-major conn db)))))))))diff --git a/test/metabase/app_db/liquibase/versions_test.clj b/test/metabase/app_db/liquibase/versions_test.cljnew file mode 100644index 00000000000..5f993546ccc--- /dev/null+++ b/test/metabase/app_db/liquibase/versions_test.clj@@ -0,0 +1,873 @@+(ns ^:mb/driver-tests metabase.app-db.liquibase.versions-test+ (:require+ [clojure.java.jdbc :as jdbc]+ [clojure.test :refer :all]+ [metabase.app-db.core :as mdb]+ [metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.versions :as versions]+ [metabase.app-db.test-util :as mdb.test-util]+ [metabase.config.core :as config]+ [metabase.driver :as driver]+ [metabase.test :as mt]+ [metabase.test.util.dynamic-redefs :as dynamic-redefs]+ [metabase.util :as u])+ (:import+ (liquibase Liquibase)+ (liquibase.changelog ChangeSet)+ (liquibase.lockservice LockServiceFactory)+ (liquibase.util LiquibaseUtil)))++(set! *warn-on-reflection* true)++(defn- tag [s] (assoc config/mb-version-info :tag s))++(defn- version-rows+ "`[deployment_id metabase_version ran?]` for every version row, in insertion order."+ [conn]+ (mapv (fn [{:keys [deployment_id metabase_version ran_migrations]}]+ [deployment_id metabase_version (or (true? ran_migrations) (= 1 ran_migrations))])+ (jdbc/query {:connection conn}+ [(format "SELECT deployment_id, metabase_version, ran_migrations FROM %s ORDER BY id"+ (versions/versions-table-name))])))++(defn- highest-versioned-major+ "The highest `vNN.` major in the changelog file: the major the one-time backfill records once these migrations have+ run. Stays meaningful after migrations become version-less -- it freezes at the last versioned major, exactly like+ the applied `vNN.` ids the backfill reads."+ [^Liquibase liquibase]+ (->> (.getChangeSets (.getDatabaseChangeLog liquibase))+ (keep (fn [^ChangeSet cs] (some-> (re-find #"^v(\d+)\." (.getId cs)) second parse-long)))+ (reduce max 0)))++(defn- migrate-without-recording!+ "Run every migration WITHOUT the recording listener, as a binary that predates version tracking would have."+ [^Liquibase liquibase]+ (liquibase/with-scope-locked liquibase (.update liquibase "")))++;;; ------------------------------------------- the version being recorded ----------------------------------------++(deftest current-recorded-version-test+ (testing "real version tags are normalized to the edition-agnostic x.{major}... form"+ (with-redefs [config/mb-version-info (tag "v0.56.2")]+ (is (= "x.56.2" (versions/current-recorded-version)) "OSS")+ (is (= 56 (versions/current-recorded-major))))+ (with-redefs [config/mb-version-info (tag "v1.56.2")]+ (is (= "x.56.2" (versions/current-recorded-version)) "EE records the same as OSS"))+ (with-redefs [config/mb-version-info (tag "v0.56.0-SNAPSHOT")]+ (is (= "x.56.0-SNAPSHOT" (versions/current-recorded-version)) "off-head snapshot")+ (is (= 56 (versions/current-recorded-major)))))+ (testing "version->major parses the stored x. form"+ (is (= 55 (versions/version->major "x.55.2.1")))+ (is (= 9999 (versions/version->major versions/dev-version)))+ (is (nil? (versions/version->major nil)))))++(deftest dev-version-test+ (testing "a build with no real release version records the constant dev version"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")]+ (is (= "x.9999.0.0" versions/dev-version (versions/current-recorded-version)))+ (is (= 9999 versions/dev-major (versions/current-recorded-major)))+ (is (false? (versions/versions-table-exists? conn))+ "computing it touches no state, so read-only paths like `migrate print` never create the version table")))))+ (testing "synthetic-dev-major? recognises exactly the dev major"+ (is (true? (versions/synthetic-dev-major? versions/dev-major)))+ (is (false? (versions/synthetic-dev-major? 65)))+ (is (false? (versions/synthetic-dev-major? nil)))))++(deftest prod-synthetic-version-fallback-warns-test+ (testing "a prod-mode binary that cannot parse its own version tag logs loudly before recording synthetic versions"+ (with-redefs [config/mb-version-info (tag "vUNKNOWN")+ config/is-prod? true]+ (reset! @#'versions/dev-version-fallback-logged? false)+ (mt/with-log-messages-for-level [messages :error]+ (is (= versions/dev-version (versions/current-recorded-version))+ "still falls back to the dev version so the instance can run")+ (is (= 1 (count (filter #(re-find #"development version" (:message %)) (messages))))+ "and logs an error explaining the degraded version tracking")+ (let [message (:message (first (filter #(re-find #"development version" (:message %)) (messages))))]+ (testing "the message names the bad tag, the version recorded, and what a release binary will do later"+ (is (re-find #"\"vUNKNOWN\"" message))+ (is (re-find #"x\.9999\.0\.0" message))+ (is (re-find #"Development build detected" message))+ (is (re-find #"version\.properties" message))))+ (versions/current-recorded-version)+ (is (= 1 (count (filter #(re-find #"development version" (:message %)) (messages))))+ "only once per process -- it is consulted several times per boot")))))++(deftest ^:parallel extract-numbers-special-case-test+ (testing "the v56 changesets that shipped in v55 report 55"+ (is (= 55 (first (#'versions/extract-numbers "v56.2025-06-05T16:48:48"))))+ (is (= 55 (first (#'versions/extract-numbers "v56.2025-05-19T16:48:48")))))+ (testing "the v65 view changesets that 1.64.2.1 ran, and 64 ships again, report 64"+ (is (= 64 (first (#'versions/extract-numbers "v65.2026-10-07T12:00:00"))))+ (is (= 64 (first (#'versions/extract-numbers "v65.2026-10-07T12:00:01")))))+ (is (= 60 (first (#'versions/extract-numbers "v60.ghdf99efd")))))++;;; ------------------------------------------------- recording runs ----------------------------------------------++(deftest record-deployment-version-nil-guard-test+ (testing "recording with a nil deployment id (empty changelog) is a no-op instead of a NOT NULL violation"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase))+ (versions/record-deployment-version! conn nil "x.1.0" true)+ (is (= [] (version-rows conn))))))))++(deftest record-deployment-version-is-idempotent-test+ (testing "an already-present (deployment, version) pair is left alone, so a boot row never overwrites a ran row"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase))+ (versions/record-deployment-version! conn "dep" "x.65.0" true)+ (versions/record-deployment-version! conn "dep" "x.65.0" false)+ (versions/record-deployment-version! conn "dep" "x.65.0" true)+ (is (= [["dep" "x.65.0" true]] (version-rows conn))))))))++(deftest recording-listener-records-the-run-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)]+ (versions/ensure-version-tracking! conn db)+ (testing "no version is recorded before any migrations have run"+ (is (nil? (versions/last-deployment-version conn db)))+ (is (= [] (versions/recorded-deployments conn db))))+ (let [run-version (versions/current-recorded-version)]+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source))+ (testing "exactly one ran row is recorded for the single deployment, with the running version"+ (is (= [run-version] (map second (version-rows conn))))+ (is (= [true] (map peek (version-rows conn)))))+ (testing "last-deployment-version returns the recorded version"+ (is (= run-version (versions/last-deployment-version conn db))))+ (testing "a second (no-op) run of the same version records nothing new"+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source))+ (is (= 1 (count (version-rows conn)))))))))))++(deftest record-boot-version-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)+ latest (highest-versioned-major liquibase)+ backfilled (versions/major->version latest)]+ ;; migrate as a pre-tracking binary would have, attributing the changesets to a *previous* process so the+ ;; deployment differs from this JVM's Liquibase deployment id+ (migrate-without-recording! liquibase)+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET deployment_id = 'priorrun'" changelog-table)])+ (versions/ensure-version-tracking! conn db)+ (is (= [["priorrun" backfilled true]] (version-rows conn))+ "ensure-version-tracking! backfills the version the vNN. ids imply as the deployment's ran version")+ (testing "a no-op boot of a real version records a boot row against the current deployment"+ (with-redefs [config/mb-version-info (tag (format "v0.%d.1" latest))]+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source)))+ (is (= [["priorrun" backfilled true]+ ["priorrun" (format "x.%d.1" latest) false]]+ (version-rows conn))))+ (testing "re-booting the same version does not add a duplicate row"+ (with-redefs [config/mb-version-info (tag (format "v0.%d.1" latest))]+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source)))+ (is (= 2 (count (version-rows conn)))))+ (testing "a NEWER major's no-op boot is recorded (that major shipped no migrations for this database)..."+ (with-redefs [config/mb-version-info (tag (format "v0.%d.0" (inc latest)))]+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source)))+ (is (= ["priorrun" (format "x.%d.0" (inc latest)) false] (last (version-rows conn)))))+ (testing "...but does not move the schema's version: the ran version still decides"+ (is (= latest (versions/current-schema-major conn db)))+ (is (= backfilled (versions/last-deployment-version conn db))))+ (testing "an OLDER major's no-op boot records nothing (an unsupported downgrade is not history worth a row)"+ (with-redefs [config/mb-version-info (tag (format "v0.%d.0" (- latest 5)))]+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source)))+ (is (= 3 (count (version-rows conn)))))+ (testing "a development build's no-op boot records nothing: its constant version carries no information"+ (with-redefs [config/mb-version-info (tag "vLOCAL_DEV")]+ (liquibase/migrate-up-if-needed! liquibase (mdb/data-source)))+ (is (= 3 (count (version-rows conn))))+ (is (= latest (versions/current-schema-major conn db)))))))))++;;; --------------------------------------------- table creation and backfill ---------------------------------------++(deftest ensure-backfills-the-current-deployment-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)+ latest (versions/major->version (highest-versioned-major liquibase))]+ (migrate-without-recording! liquibase)+ ;; an existing instance with no recorded versions, split across two deployments by execution position (not+ ;; by id: the most recent changesets may be version-less, which carry no major in their id)+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET deployment_id = 'older'" changelog-table)])+ ;; MySQL can't UPDATE a table that a subquery of the same statement reads, so find the boundary first+ (let [boundary (-> (jdbc/query {:connection conn} [(format "SELECT MAX(orderexecuted) - 10 AS boundary FROM %s" changelog-table)])+ first+ :boundary)]+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET deployment_id = 'newer' WHERE orderexecuted > ?" changelog-table)+ boundary]))+ (versions/ensure-version-tracking! conn db)+ (testing "the current deployment gets a ran row with the highest vNN. major, and the older one the major its+ own rows put the schema at"+ (is (= [["newer" latest true] ["older" latest true]] (version-rows conn))))+ (testing "deployed_at is populated for the backfilled row"+ (is (every? :deployed_at (jdbc/query {:connection conn}+ [(format "SELECT deployed_at FROM %s" (versions/versions-table-name))]))))+ (testing "ensure is idempotent"+ (versions/ensure-version-tracking! conn db)+ (is (= 2 (count (version-rows conn)))))+ (testing "the deployments keep their changelog order"+ (is (= ["newer" "older"] (map :deployment-id (versions/recorded-deployments conn db))))))))))++(deftest backfill-uses-highest-version-not-latest-executed-test+ (testing "the backfill records the highest major present, even when a lower-version changeset ran last (a back-ported patch)"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)+ latest (versions/major->version (highest-versioned-major liquibase))]+ (migrate-without-recording! liquibase)+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET deployment_id = 'main'" changelog-table)])+ ;; a back-ported patch: a much lower-version changeset that EXECUTED LAST, in its own deployment+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "patch" :changesets ["v44.patch-backport"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= [["patch" latest true] ["main" latest true]] (version-rows conn))+ "the current deployment is the patch's, but its version is the highest major in the changelog")+ (is (= latest (versions/last-deployment-version conn db)))))))))++(deftest backfill-counts-the-v65-changesets-64-ships-as-64-test+ (testing "a 64 database that ran the two v65 view changesets 64 ships again is backfilled as 64, not 65 -- otherwise+ every later 64 patch would refuse to boot against it"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d64" :changesets ["v64.2026-07-22T00:00:00"+ "v65.2026-10-07T12:00:00"+ "v65.2026-10-07T12:00:01"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= [["d64" "x.64.0.0" true]] (version-rows conn)))+ (is (= 64 (versions/current-schema-major conn db)))))))))++(deftest first-backfill-records-every-legacy-deployment-test+ (testing "the first backfill records each legacy deployment at the major its vNN. ids put the schema at, so the+ major before the current one stays a `migrate down` target"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d63" :changesets ["v63.2026-01-01T00:00:00"]}+ {:deployment "d64a" :changesets ["v64.2026-05-01T00:00:00"]}+ {:deployment "d64b" :changesets ["v64.2026-06-01T00:00:00"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= [["d64b" "x.64.0.0" true]+ ["d64a" "x.64.0.0" true]+ ["d63" "x.63.0.0" true]]+ (version-rows conn)))+ (testing "ensure stays idempotent"+ (versions/ensure-version-tracking! conn db)+ (is (= 3 (count (version-rows conn)))))+ (is (= 63 (versions/previous-recorded-major (versions/recorded-deployments conn db) false)))))))))++(deftest version-tracking-sql-backfills-every-legacy-deployment-test+ (testing "the manual-upgrade SQL backfills every legacy deployment, like the first boot does: once it is applied the+ versions are tracked, and no later backfill records the older deployments"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "d63" :changesets ["v63.2026-01-01T00:00:00"]}+ {:deployment "d64" :changesets ["v64.2026-05-01T00:00:00"]}])+ (is (false? (versions/versions-table-exists? conn)) "sanity: a pre-tracking install")+ (with-redefs [config/mb-version-info (tag "v0.65.0")]+ (let [sql (versions/version-tracking-sql conn db 65)]+ (is (re-find #"'d64', 'x\.64\.0\.0'" sql))+ (is (re-find #"'d63', 'x\.63\.0\.0'" sql))))))))))++(deftest migrate-up-repairs-legacy-rollbacks-under-the-lock-test+ (testing "migrate up repairs a legacy rollback once it holds the migration lock, and not when there is nothing to run"+ (mt/with-temp-empty-app-db [_conn :h2]+ (let [lock-held-at-repair (atom [])+ repair (dynamic-redefs/original-fn #'versions/repair-legacy-rollback!)]+ (mt/with-dynamic-fn-redefs [versions/repair-legacy-rollback!+ (fn [conn database unrun-change-sets]+ (swap! lock-held-at-repair conj+ (.hasChangeLogLock (.getLockService (LockServiceFactory/getInstance) database)))+ (repair conn database unrun-change-sets))]+ (mdb/migrate! (mdb/data-source) :up)+ (is (= [true] @lock-held-at-repair))+ (mdb/migrate! (mdb/data-source) :up)+ (is (= [true] @lock-held-at-repair)))))))++(deftest migrate-recreates-version-table-after-failure-test+ (testing "the version table is (re)created at the start of every migrate!, so a rolled-back CREATE (transactional DDL) is simply redone"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase)))+ (is (true? (versions/versions-table-exists? conn)))+ (jdbc/execute! {:connection conn} [(format "DROP TABLE %s" (versions/versions-table-name))])+ (mt/with-dynamic-fn-redefs [liquibase/migrate-up-if-needed! (fn [& _] (throw (ex-info "boom" {})))]+ (is (thrown-with-msg? Exception #"boom" (mdb/migrate! (mdb/data-source) :up))))+ (mdb/migrate! (mdb/data-source) :up)+ (is (true? (versions/versions-table-exists? conn)))))))++(defn- ddl-statements!+ "Run `f` with [[versions/ensure-version-tracking!]]'s statements spied on; returns the CREATE TABLE statements run."+ [f]+ (let [orig (dynamic-redefs/original-fn #'versions/execute-ddl!)+ statements (atom [])]+ (mt/with-dynamic-fn-redefs [versions/execute-ddl! (fn [conn ddl]+ (swap! statements conj ddl)+ (orig conn ddl))]+ (f))+ @statements))++(deftest ensure-version-tracking-skips-ddl-when-table-exists-test+ (testing "once the version tables exist, boots run no DDL: a Postgres user without CREATE on the schema can still boot"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)]+ (is (= 2 (count (ddl-statements! #(versions/ensure-version-tracking! conn db))))+ "the first call creates the version and first-run tables")+ (is (= [] (ddl-statements! #(versions/ensure-version-tracking! conn db)))+ "later calls see it and skip the CREATE")))))))++(deftest ensure-version-tracking-tolerates-losing-create-race-test+ (testing "a CREATE that fails because a concurrent boot created the table first is not an error"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ orig (dynamic-redefs/original-fn #'versions/execute!)+ checks (atom {})]+ ;; the other instance's CREATEs land between our existence checks and our own CREATEs+ (orig conn [(#'versions/table-ddl)])+ (orig conn [(#'versions/first-runs-table-ddl)])+ (mt/with-dynamic-fn-redefs [versions/table-exists? (fn [table-name _conn] ; stale on each table's first check only+ (> (get (swap! checks update table-name (fnil inc 0)) table-name) 1))+ versions/execute-ddl! (fn [_conn _ddl]+ (throw (java.sql.SQLException. "duplicate key value violates unique constraint \"pg_type_typname_nsp_index\"")))]+ (is (nil? (versions/ensure-version-tracking! conn db))))+ (testing "but a CREATE failure with no table behind it still propagates"+ (mt/with-dynamic-fn-redefs [versions/table-exists? (constantly false)+ versions/execute-ddl! (fn [_conn _ddl]+ (throw (java.sql.SQLException. "permission denied for schema public")))]+ (is (thrown-with-msg? java.sql.SQLException #"permission denied"+ (versions/ensure-version-tracking! conn db)))))))))))++(deftest ensure-version-tracking-tolerates-losing-create-race-in-a-transaction-test+ (testing "on Postgres, inside the caller's transaction, a CREATE that loses the race is rolled back to its savepoint+ and the transaction goes on (two instances upgrading the same install at once)"+ (mt/test-drivers #{:postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ checks (atom {})]+ (.setAutoCommit ^java.sql.Connection conn false)+ ;; the other instance's CREATE commits between our existence check and our own CREATE, which then fails+ (mt/with-dynamic-fn-redefs [versions/table-exists? (fn [table-name _conn]+ (> (get (swap! checks update table-name (fnil inc 0)) table-name) 1))+ versions/table-ddl (constantly "CREATE TABLE databasechangelog_version (this is not valid")]+ (with-open [other (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (jdbc/execute! {:connection other} [((dynamic-redefs/original-fn #'versions/table-ddl))]))+ (is (nil? (versions/ensure-version-tracking! conn db))))+ (is (= [{:one 1}] (jdbc/query {:connection conn} ["SELECT 1 AS one"]))+ "the caller's transaction is still usable")))))))++(deftest versions-table-name-matches-changelog-casing-test+ (testing "the version table is cased like Liquibase's DATABASECHANGELOG: upper case on H2 and MySQL, lower on Postgres"+ ;; MySQL table names are case-sensitive on Linux, and other code (the audit app, h2 dump comparisons) looks it up+ ;; by the same spelling as the changelog table+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase))+ (let [expected (str (liquibase/changelog-table-name liquibase)+ (if (= :postgres driver/*driver*) "_version" "_VERSION"))+ actual (->> (jdbc/metadata-result (.getTables (.getMetaData conn) (.getCatalog conn) nil "%"+ (u/varargs String ["TABLE"])))+ (map :table_name)+ (filter #(= "databasechangelog_version" (u/lower-case-en %))))]+ (is (= expected (versions/versions-table-name)))+ (is (= [expected] actual))))))))++(deftest mysql-ran-migrations-is-bit-test+ (testing "on MySQL/MariaDB ran_migrations is bit(1), like every other boolean column (see general-schema-tests)"+ (mt/test-driver :mysql+ (mt/with-temp-empty-app-db [conn :mysql]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase)))+ (is (= "bit"+ (-> (jdbc/query {:connection conn}+ ["SELECT data_type FROM information_schema.columns WHERE table_schema = DATABASE() AND table_name = ? AND column_name = 'ran_migrations'"+ (versions/versions-table-name)])+ first+ :data_type)))+ (testing "and still reads back as a ran row"+ (versions/record-deployment-version! conn "dep1" "x.64.0" true)+ (is (= [["dep1" "x.64.0" true]] (version-rows conn))))))))++(deftest deployed-at-uses-unified-timestamp-type-test+ (testing "the version table starts with the post-UnifyTimeColumnsType type on both fresh installs and upgrades"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase)))+ (let [{:keys [data_type datetime_precision]}+ (first+ (case driver/*driver*+ :mysql+ (jdbc/query {:connection conn}+ [(str "SELECT data_type, datetime_precision FROM information_schema.columns "+ "WHERE table_schema = DATABASE() AND table_name = ? AND column_name = 'deployed_at'")+ (versions/versions-table-name)])++ (:h2 :postgres)+ (jdbc/query {:connection conn}+ [(str "SELECT data_type, datetime_precision FROM information_schema.columns "+ "WHERE LOWER(table_name) = ? AND LOWER(column_name) = 'deployed_at'")+ (u/lower-case-en (versions/versions-table-name))])))]+ (case driver/*driver*+ :h2 (is (= "timestamp with time zone" (some-> data_type u/lower-case-en)))+ :postgres (is (= "timestamp with time zone" data_type))+ :mysql (do+ (is (= "timestamp" data_type))+ (is (= 6 datetime_precision)))))))))++(deftest repair-corrects-a-deployment-trimmed-by-a-legacy-rollback-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (testing "a pre-tracking binary's `migrate down` removes changesets but leaves their version rows"+ ;; upgraded 62 -> 64 in one run (backfilled as 64 by the first 65 boot), then rolled back to 63 by 64+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "d1"+ :ran "x.64.0.0"+ :booted ["x.64.2" "x.63.1"]+ ;; The old binary does not know this synthetic row, so it survives+ ;; when that binary rolls the real v64 changeset back.+ :changesets ["v62.00-001" "v63.00-001"+ {:id "v64.legacy-version-tracking"+ :author "version-tracking"+ :filename "legacy-version-tracking"}]}])+ (versions/ensure-version-tracking! conn db)+ (versions/repair-legacy-rollback! conn db ["v64.00-001"])+ (is (= #{["d1" "x.63.0.0" true] ["d1" "x.63.1" false]} (set (version-rows conn)))+ "the ran row is lowered to the highest remaining major, and newer boot rows are dropped")+ (is (= 63 (versions/current-schema-major conn db)))))))))++(deftest repair-leaves-tracked-deployments-alone-test+ (testing "a ran version above the changelog's highest vNN. major is only corrected for a legacy-only deployment"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (doseq [[desc {:keys [ran changesets]}]+ {"the binary's legacy marker keeps the major" {:ran "x.65.0" :changesets ["v64.00-001" "v65.legacy-version-tracking"]}+ "version-less changesets were run by a tracking binary" {:ran "x.66.0" :changesets ["v65.00-001" "2026-10-01-add-thing"]}+ "a development build's version is never corrected" {:ran versions/dev-version :changesets ["v64.00-001"]}}]+ (testing desc+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "d1" :ran ran :changesets changesets}])+ (versions/ensure-version-tracking! conn db)+ (versions/repair-legacy-rollback! conn db [])+ (is (= [["d1" ran true]] (version-rows conn)))))))))))++;;; ------------------------------------------------- deployment reads ---------------------------------------------++(deftest recorded-deployments-follow-the-changelog-test+ (testing "the current deployment is the one owning the newest changelog row, whatever order the version rows were written in"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; version rows land newest-major first, but d64's changeset is the most recently executed one+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "d65" :ran "x.65.0" :changesets ["c65"] :minutes-ago 10}+ {:deployment "d64" :ran "x.64.0" :changesets ["c64"] :minutes-ago 5}])+ (is (= ["d64" "d65"] (map :deployment-id (versions/recorded-deployments conn db))))+ (is (= 64 (versions/current-schema-major conn db)))+ (is (= "x.64.0" (versions/last-deployment-version conn db)))+ (is (= "d65" (versions/previous-deployment-id conn db)))))))))++(deftest rollback-window-test+ (testing "the window is every current-major deployment plus the most recent previous-major one; `all?` is the whole history"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "d58" :ran "x.58.0" :changesets ["c58"]}+ {:deployment "d59" :ran "x.59.0" :changesets ["c59"] :booted ["x.59.3"]}+ {:deployment "d60a" :ran "x.60.0" :changesets ["c60a"]}+ {:deployment "d60b" :ran "x.60.1" :changesets ["c60b"]}])+ (let [deployments (versions/recorded-deployments conn db)]+ (is (= ["d60b" "d60a" "d59" "d58"] (map :deployment-id deployments)) "newest first")+ (is (= 60 (versions/schema-major deployments)))+ (is (= ["d60b" "d60a" "d59"] (map :deployment-id (versions/rollback-window deployments false)))+ "both current-major (v60) deployments and the most recent previous-major (v59) deployment, not the older v58")+ (is (= ["d60b" "d60a" "d59" "d58"] (map :deployment-id (versions/rollback-window deployments true)))+ "with all? the full recorded history (what a forced rollback targets)")+ (is (= #{58 59 60} (versions/recorded-majors deployments)))+ (is (= ["x.59.0" "x.59.3"] (:versions (nth deployments 2))) "ran and booted versions alike")+ (is (= "x.59.0" (:ran-version (nth deployments 2)))))))))))++(deftest previous-recorded-major-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)+ deployments #(versions/recorded-deployments conn db)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "d63" :ran "x.63.0" :changesets ["c63"]}])+ (testing "a single deployment has nothing earlier"+ (is (nil? (versions/previous-recorded-major (deployments) false))))+ (testing "a release that booted without migrations steps back to the schema-bearing release, not past it"+ (versions/record-deployment-version! conn "d63" "x.64.0" false)+ (is (= 63 (versions/previous-recorded-major (deployments) false))))+ (jdbc/execute! {:connection conn}+ [(format "DELETE FROM %s WHERE metabase_version = 'x.64.0'"+ (versions/versions-table-name))])+ ;; the instance then upgraded directly to 65 -- 64 was never a recorded deployment+ (mdb.test-util/fabricate-history! conn changelog-table+ [{:deployment "d65" :ran "x.65.0" :changesets ["c65"]}])+ (testing "the previous *recorded* major, not (dec current), which was never deployed and could not resolve"+ (is (= 63 (versions/previous-recorded-major (deployments) false))))+ (testing "a major recorded only as a boot row counts: it is a nameable boundary at the same deployment"+ (versions/record-deployment-version! conn "d63" "x.64.0" false)+ (is (= 64 (versions/previous-recorded-major (deployments) false)))))))))++;;; ---------------------------------------- the legacy-version-tracking marker -------------------------------------++(deftest latest-applied-major-version-test+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (is (nil? (versions/latest-applied-major-version conn (.getDatabase liquibase))))+ (migrate-without-recording! liquibase)+ (is (< 52 (versions/latest-applied-major-version conn (.getDatabase liquibase))))))))++(deftest record-legacy-version-tracking-test+ (testing "record-legacy-version-tracking! tags the deployment that ran migrations, so older binaries read the right major"+ ;; Older (pre-version-less) binaries detect an unsupported downgrade via latest-applied-major-version (id LIKE 'v%')+ ;; and know nothing about databasechangelog_version. For a version-less major this row is the only signal.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)+ markers (fn [] (mapv :id (jdbc/query {:connection conn}+ [(format "SELECT id FROM %s WHERE id LIKE 'v%%.legacy-version-tracking' ORDER BY id" ct)])))+ row-of (fn [id] (first (jdbc/query {:connection conn}+ [(format "SELECT deployment_id, orderexecuted, liquibase, description FROM %s WHERE id = ?" ct) id])))]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d66" :changesets ["c66"]}])+ (testing "records the row against the deployment that ran, and latest-applied-major-version reads it"+ (versions/record-legacy-version-tracking! db 66 "d66")+ (is (= ["v66.legacy-version-tracking"] (markers)))+ (is (= "d66" (:deployment_id (row-of "v66.legacy-version-tracking")))+ "shares the deployment_id, so the normal rollback path removes it with its deployment")+ (is (= (inc (:orderexecuted (row-of "c66"))) (:orderexecuted (row-of "v66.legacy-version-tracking")))+ "at the next execution position, like a changeset Liquibase ran")+ (is (= (LiquibaseUtil/getBuildVersion) (:liquibase (row-of "v66.legacy-version-tracking")))+ "stamped with the Liquibase version, like a changeset Liquibase ran")+ (is (= "Metabase v66 version marker (not a changeset)" (:description (row-of "v66.legacy-version-tracking")))+ "with a description saying what the row is")+ (with-redefs [config/mb-version-info (tag "v0.66.0")]+ (is (re-find (re-pattern (str "legacy-version-tracking[^;]*'" (LiquibaseUtil/getBuildVersion) "'"))+ (versions/version-tracking-sql conn db 0))+ "and so is the marker in the manual-upgrade SQL"))+ (is (= 66 (versions/latest-applied-major-version conn db))))+ (testing "idempotent when the marker already ends the changelog"+ (versions/record-legacy-version-tracking! db 66 "d66")+ (is (= ["v66.legacy-version-tracking"] (markers)))+ (is (= "d66" (:deployment_id (row-of "v66.legacy-version-tracking")))))+ (testing "a later deployment of the same major moves the row to itself"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d66b" :changesets ["c66b"]}])+ (versions/record-legacy-version-tracking! db 66 "d66b")+ (is (= ["v66.legacy-version-tracking"] (markers)))+ (is (= "d66b" (:deployment_id (row-of "v66.legacy-version-tracking")))))+ (testing "a new major adds its own row; earlier majors' rows are deliberately kept"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d67" :changesets ["c67"]}])+ (versions/record-legacy-version-tracking! db 67 "d67")+ (is (= ["v66.legacy-version-tracking" "v67.legacy-version-tracking"] (markers)))+ (is (= 67 (versions/latest-applied-major-version conn db))+ "older binaries read the highest, i.e. the current major"))+ (testing "the 1-arity records the running binary's major against the current deployment"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d68" :changesets ["c68"]}])+ (with-redefs [config/mb-version-info (tag "v0.68.0")]+ (versions/record-legacy-version-tracking! db))+ (is (= "d68" (:deployment_id (row-of "v68.legacy-version-tracking")))))+ (testing "a dev deployment gets a marker too: a pre-version-less binary pointed at a dev DB must refuse+ cleanly (its id scan reads v9999) instead of picking the legacy changelog off the version-less+ id and crashing on re-running changeset 1"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d9999" :changesets ["devmig"]}])+ (versions/record-legacy-version-tracking! db versions/dev-major "d9999")+ (is (= versions/dev-major (versions/latest-applied-major-version conn db))+ "older binaries read the dev major and refuse"))))))))++(deftest force-migrate-writes-legacy-marker-test+ (testing "migrate force records the vNN.legacy-version-tracking marker just like a normal upgrade, so+ pre-versionless binaries can still detect a downgrade after a force upgrade"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [config/mb-version-info (tag "v0.66.0")+ liquibase/changelog-file "versionless-dev-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :force))+ (let [ct (liquibase/changelog-table-name conn)+ row (fn [id] (first (jdbc/query {:datasource (mdb/data-source)}+ [(format "SELECT deployment_id, orderexecuted FROM %s WHERE id = ?" ct) id])))+ marker (row "v66.legacy-version-tracking")]+ (is (some? marker) "force must leave the same old-binary downgrade signal as a normal upgrade")+ (is (= (:deployment_id (row "dev_run_a")) (:deployment_id marker))+ "as an ordinary row of the deployment that ran the migrations")+ (liquibase/with-liquibase [liquibase conn]+ (is (= 66 (versions/latest-applied-major-version conn (.getDatabase liquibase)))+ "a pre-versionless binary's id scan now reads the upgraded major")))))))++;;; ---------------------------------------------- first runs of re-run changesets ----------------------------------++(deftest re-run-records-where-the-changeset-first-ran-test+ (testing "a re-run keeps where the changeset first ran, which Liquibase overwrites on its changelog row"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, dateexecuted, orderexecuted FROM %s WHERE id = ?" ct) id])))+ first-of (fn [] (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (get (versions/first-runs c) ["migrations/2026/versionless_roc.yaml" "test" "roc_view"])))+ migrate! (fn [changelog]+ (with-redefs [config/mb-version-info (tag "v0.66.0")+ liquibase/changelog-file changelog]+ (mdb/migrate! (mdb/data-source) :up)))]+ (migrate! "versionless-roc-run1.yaml")+ (is (nil? (first-of)) "a changeset that only ran once needs no record")+ (mdb.test-util/age-changelog-rows! spec ct)+ (let [original (row-of "roc_view")]+ (migrate! "versionless-roc-run3.yaml")+ (is (not= (:deployment_id original) (:deployment_id (row-of "roc_view"))) "sanity: the row moved")+ (is (= original (first-of)) "the first run is recorded as the row was before the re-run")+ (mdb.test-util/age-changelog-rows! spec ct)+ (migrate! "versionless-roc-run1.yaml")+ (is (= (:deployment_id original) (:deployment_id (first-of)))+ "a second re-run keeps the first run")))))))++(deftest run-always-records-where-the-changeset-first-ran-test+ (testing "a runAlways changeset re-runs on every update run, so it gets a first-run record too"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (let [ct (liquibase/changelog-table-name conn)+ spec {:datasource (mdb/data-source)}+ row-of (fn [id] (first (jdbc/query spec [(format "SELECT deployment_id, dateexecuted, orderexecuted FROM %s WHERE id = ?" ct) id])))+ first-of (fn [] (with-open [c (.getConnection ^javax.sql.DataSource (mdb/data-source))]+ (get (versions/first-runs c) ["migrations/2026/versionless_always.yaml" "test" "always_view"])))+ migrate! (fn []+ (with-redefs [config/mb-version-info (tag "v0.66.0")+ liquibase/changelog-file "versionless-always.yaml"]+ (mdb/migrate! (mdb/data-source) :up)))]+ (migrate!)+ (mdb.test-util/age-changelog-rows! spec ct)+ (let [original (row-of "always_view")]+ (migrate!)+ (is (not= (:deployment_id original) (:deployment_id (row-of "always_view"))) "sanity: the view re-ran")+ (is (= original (first-of)))))))))++(deftest first-runs-keep-the-changelog-timestamp-type-test+ (testing "a fresh install, which runs UnifyTimeColumnsType over the tables already there, leaves the copied+ dateexecuted with the changelog's type, so the two still compare"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (mdb/migrate! (mdb/data-source) :up)+ (let [column-type (fn [table]+ (-> (jdbc/query {:connection conn}+ [(str "SELECT data_type FROM information_schema.columns "+ "WHERE LOWER(table_name) = ? AND LOWER(column_name) = 'dateexecuted'"+ (when (= :mysql driver/*driver*) " AND table_schema = DATABASE()"))+ (u/lower-case-en table)])+ first :data_type u/lower-case-en))]+ (is (some? (column-type (versions/first-runs-table-name))))+ (is (= (column-type (liquibase/changelog-table-name conn))+ (column-type (versions/first-runs-table-name)))))))))++;;; ------------------------------- binaries from before version-less support (58 - 65) ------------------------------+;;; Version-less support is backported to patch releases of every supported major, so customers move between those+;;; patches and the earlier ("old") patches of the same majors, which know nothing about version tracking.++(defn- old-binary-latest-row+ "The changelog row an old binary's `decide-liquibase-file` reads to pick its changelog: `ORDER BY dateexecuted DESC`+ with no tiebreak. A tie for newest is returned as `::tie`, since the database may then return either row."+ [conn changelog-table]+ (let [newest (jdbc/query {:connection conn}+ [(format "SELECT id FROM %s WHERE dateexecuted = (SELECT MAX(dateexecuted) FROM %s)"+ changelog-table changelog-table)])]+ (if (= 1 (count newest)) (:id (first newest)) ::tie)))++(defn- marker-rows [conn changelog-table]+ (mapv (juxt :id :deployment_id)+ (jdbc/query {:connection conn}+ [(format "SELECT id, deployment_id FROM %s WHERE filename = 'legacy-version-tracking' ORDER BY id"+ changelog-table)])))++(deftest tail-marker-keeps-old-binaries-on-the-current-changelog-test+ (testing "old binaries take a version-less newest row for a pre-4.2 install, so every run ends with a marker they read safely"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58" :ran "x.58.9" :changesets ["v58.2025-01-01T00:00:00" "glossary_entity_id"]}])+ (testing "a versioned major gets the position marker, which never raises the major old binaries read"+ (versions/record-legacy-version-tracking! db 58 "d58")+ (is (= "v56.legacy-version-tracking" (old-binary-latest-row conn ct)))+ (is (= [["v56.legacy-version-tracking" "d58"]] (marker-rows conn ct)))+ (is (= 58 (versions/latest-applied-major-version conn db))))+ (testing "a later patch of the same major moves the marker to its own deployment"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58b" :ran "x.58.10" :changesets ["thing_two"]}])+ (versions/record-legacy-version-tracking! db 58 "d58b")+ (is (= "v56.legacy-version-tracking" (old-binary-latest-row conn ct)))+ (is (= [["v56.legacy-version-tracking" "d58b"]] (marker-rows conn ct))))+ (testing "the marker lands strictly after a row stored in the future (MySQL rounds sub-second timestamps up)"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58c" :ran "x.58.11" :changesets ["thing_three"]}])+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET dateexecuted = CURRENT_TIMESTAMP + INTERVAL '1' SECOND WHERE id = 'thing_three'" ct)])+ (versions/record-legacy-version-tracking! db 58 "d58c")+ (is (= "v56.legacy-version-tracking" (old-binary-latest-row conn ct)))+ (is (= ["d58c"] (map :deployment-id (take 1 (versions/recorded-deployments conn db))))+ "and stays part of the deployment that ran"))+ (testing "a version-less major gets its own major marker, so old binaries read it as newer"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d66" :ran "x.66.0" :changesets ["thing_four"]}])+ (versions/record-legacy-version-tracking! db 66 "d66")+ (is (= "v66.legacy-version-tracking" (old-binary-latest-row conn ct)))+ (is (= 66 (versions/latest-applied-major-version conn db)))+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d66b" :ran "x.66.1" :changesets ["thing_five"]}])+ (versions/record-legacy-version-tracking! db 66 "d66b")+ (is (= "v66.legacy-version-tracking" (old-binary-latest-row conn ct)))+ (is (= [["v56.legacy-version-tracking" "d58c"] ["v66.legacy-version-tracking" "d66b"]] (marker-rows conn ct))))))))))++(deftest ensure-backfills-deployments-old-binaries-ran-test+ (testing "deployments an old binary ran after version tracking started are recorded from their vNN. ids"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; 58.9 (tracking) -> 59.0 (old) -> 60.0 (old) -> back to a tracking binary+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58" :ran "x.58.9" :changesets ["v58.2025-01-01T00:00:00" "thing_one"]}+ {:deployment "d59" :changesets ["v59.2025-02-01T00:00:00"]}+ {:deployment "d60" :changesets ["v60.2025-03-01T00:00:00"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= [["d58" "x.58.9" true] ["d59" "x.59.0.0" true] ["d60" "x.60.0.0" true]]+ (sort (version-rows conn))))+ (is (= 59 (versions/previous-recorded-major (versions/recorded-deployments conn db) false))+ "so `migrate down` steps back to the major the old binary left, not past it")))))))++(deftest ensure-repairs-a-deployment-an-old-binary-rolled-back-test+ (testing "an old binary's `migrate down` removes a major's vNN. changesets but not the version rows recording that major"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; 58 -> 59.9 (tracking: v59 ids, a version-less changeset, the marker), then 59.0 (old) ran `migrate down`,+ ;; which removed the v59 ids and nothing else+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58" :ran "x.58.0.0" :changesets ["v58.2025-01-01T00:00:00"]}+ {:deployment "d59" :ran "x.59.9" :booted ["x.59.10"]+ :changesets ["thing_one"+ {:id "v56.legacy-version-tracking" :author "version-tracking" :filename "legacy-version-tracking"}]}])+ (versions/ensure-version-tracking! conn db)+ (is (= [["d58" "x.58.0.0" true] ["d59" "x.58.0.0" true]] (sort (version-rows conn)))+ "recorded as the major its changesets are back at, so a 58 binary boots against it")+ (is (= 58 (versions/current-schema-major conn db)))))))+ (testing "a deployment whose major's vNN. changesets are still there is left alone"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ ;; 59.0 (old) ran the v59 ids; 59.9 later ran only a version-less changeset+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d59" :ran "x.59.0.0" :changesets ["v59.2025-01-01T00:00:00"]}+ {:deployment "d59b" :ran "x.59.9" :changesets ["thing_one"]}+ {:deployment "d66" :ran "x.66.0" :changesets ["thing_two"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= [["d59" "x.59.0.0" true] ["d59b" "x.59.9" true] ["d66" "x.66.0" true]] (sort (version-rows conn)))+ "and so is a version-less major, which has no vNN. changesets at all"))))))))++(deftest deployments-are-ordered-like-liquibase-orders-them-test+ (testing "rows are ordered by dateexecuted, then orderexecuted, like Liquibase: orderexecuted only increases within+ an update run and can restart, so a later deployment may have lower values"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58" :ran "x.58.0" :changesets ["v58.2025-01-01T00:00:00"]}+ {:deployment "d59" :ran "x.59.0" :changesets ["v59.2025-01-01T00:00:00"]}])+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET orderexecuted = 0 WHERE deployment_id = 'd59'" ct)])+ (is (= ["d59" "d58"] (map :deployment-id (versions/recorded-deployments conn db))))+ (is (= "d59" (versions/last-deployment-id conn db)))+ (is (= 59 (versions/current-schema-major conn db)))))))))++(deftest ensure-heals-a-version-less-newest-row-test+ (testing "whatever left a version-less row newest, the next entry point puts a marker after it"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d58" :ran "x.58.99" :changesets ["v58.2025-01-01T00:00:00" "thing_one"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= "v56.legacy-version-tracking" (old-binary-latest-row conn ct)))+ (testing "and leaves a newest row old binaries already read safely alone"+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d59" :ran "x.59.0.0" :changesets ["v59.2025-01-01T00:00:00"]}])+ (versions/ensure-version-tracking! conn db)+ (is (= "v59.2025-01-01T00:00:00" (old-binary-latest-row conn ct)))+ (is (= [["v56.legacy-version-tracking" "d58"]] (marker-rows conn ct))))))))))++(deftest which-majors-are-versioned-comes-from-the-changelog-test+ (testing "a major is versioned when its own vNN. changesets are in the changelog -- however late a major still shipped some"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)]+ (versions/ensure-version-tracking! conn db)+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d65" :ran "x.65.0" :changesets ["v65.abc123"]}+ {:deployment "d66" :ran "x.66.9" :changesets ["v66.def456" "thing_one"]}])+ (testing "its own ids tell old binaries the major, so it gets the position marker"+ (versions/record-legacy-version-tracking! db 66 "d66")+ (is (= [["v56.legacy-version-tracking" "d66"]] (marker-rows conn ct)))+ (is (= 66 (versions/latest-applied-major-version conn db))))+ (testing "an old 66 binary's `migrate down` removed v66.def456: the deployment is repaired to 65"+ (jdbc/execute! {:connection conn} [(format "DELETE FROM %s WHERE id = 'v66.def456'" ct)])+ (versions/ensure-version-tracking! conn db)+ (is (= [["d65" "x.65.0" true] ["d66" "x.65.0.0" true]] (sort (version-rows conn))))+ (is (= 65 (versions/current-schema-major conn db))))))))))diff --git a/test/metabase/app_db/liquibase_test.clj b/test/metabase/app_db/liquibase_test.cljindex 7bd44d6cc1a..760a1c1c3ca 100644--- a/test/metabase/app_db/liquibase_test.clj+++ b/test/metabase/app_db/liquibase_test.clj@@ -1,10 +1,15 @@(ns ^:mb/driver-tests metabase.app-db.liquibase-test(:require+ [clojure.java.jdbc :as jdbc][clojure.string :as str][clojure.test :refer :all]+ [metabase.app-db.connection :as mdb.connection][metabase.app-db.core :as mdb][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.rollback :as rollback]+ [metabase.app-db.liquibase.versions :as versions][metabase.app-db.test-util :as mdb.test-util]+ [metabase.config.core :as config][metabase.driver :as driver][metabase.driver.sql-jdbc.connection :as sql-jdbc.conn][metabase.driver.sql-jdbc.execute :as sql-jdbc.execute]@@ -12,7 +17,6 @@[next.jdbc :as next.jdbc][toucan2.core :as t2])(:import- (clojure.lang ExceptionInfo)(liquibase Liquibase)(liquibase.lockservice LockServiceFactory)))@@ -39,17 +43,21 @@(doseq [statement ["DROP DATABASE IF EXISTS liquibase_test;""CREATE DATABASE liquibase_test;"]](next.jdbc/execute! conn [statement]))))- (liquibase/with-liquibase [liquibase (->> (mt/dbdef->connection-details :mysql :db {:database-name "liquibase_test"})- (sql-jdbc.conn/connection-details->spec :mysql)- mdb.test-util/->ClojureJDBCSpecDataSource)]- (testing "Make sure *every* line contains ENGINE ... CHARACTER SET ... COLLATE"- (doseq [line (split-migrations-sqls (liquibase/migrations-sql liquibase))- :when (str/starts-with? line "CREATE TABLE")]- (is (true?- (or- (str/includes? line "ENGINE InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci")- (str/includes? line "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci")))- (format "%s should include ENGINE ... CHARACTER SET ... COLLATE ..." (pr-str line)))))))))+ (let [data-source (->> (mt/dbdef->connection-details :mysql :db {:database-name "liquibase_test"})+ (sql-jdbc.conn/connection-details->spec :mysql)+ mdb.test-util/->ClojureJDBCSpecDataSource)]+ ;; the version-tracking tail of the printed SQL uses the application-db dialect, so the application db must+ ;; BE this MySQL database (as it is on every real path that prints migration SQL)+ (binding [mdb.connection/*application-db* (mdb.connection/application-db :mysql data-source)]+ (liquibase/with-liquibase [liquibase data-source]+ (testing "Make sure *every* line contains ENGINE ... CHARACTER SET ... COLLATE"+ (doseq [line (split-migrations-sqls (liquibase/migrations-sql liquibase))+ :when (str/starts-with? line "CREATE TABLE")]+ (is (true?+ (or+ (str/includes? line "ENGINE InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci")+ (str/includes? line "ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci")))+ (format "%s should include ENGINE ... CHARACTER SET ... COLLATE ..." (pr-str line)))))))))))(deftest consolidate-liquibase-changesets-test(mt/test-drivers #{:h2 :mysql :postgres}@@ -134,43 +142,145 @@;; This will fail if the com.github.blagerweij/liquibase-sessionlock dep is not present(is (lock liquibase3) "Can acquire session lock when conn closed without lock release")))))))-(deftest latest-available-major-version- (mt/test-drivers #{:h2}- (mt/with-temp-empty-app-db [conn driver/*driver*]- (liquibase/with-liquibase [liquibase conn]- (is (< 52 (liquibase/latest-available-major-version liquibase)))))))+(deftest decide-liquibase-file-version-less-ids-test+ (testing "decide-liquibase-file treats version-less (year-directory) ids as modern, not as a pre-4.2 install"+ ;; Regression: a version-less latest changeset id (e.g. `aeiagus09e`) has no leading `v`, so the old heuristic+ ;; mistook a modern install for pre-4.2 and loaded the legacy changelog, re-running the earliest migrations and+ ;; bricking the instance on its next boot.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ ;; with-liquibase creates the databasechangelog table, so we are no longer a "fresh install"+ (let [db (.getDatabase liquibase)+ changelog-table (liquibase/changelog-table-name liquibase)+ decide #(#'liquibase/decide-liquibase-file conn db)+ set-latest! (fn [id filename]+ (jdbc/execute! {:connection conn} [(format "DELETE FROM %s" changelog-table)])+ (jdbc/execute! {:connection conn}+ [(format (str "INSERT INTO %s (id, author, filename, dateexecuted, orderexecuted, exectype) "+ "VALUES (?, 't', ?, CURRENT_TIMESTAMP, 1, 'EXECUTED')")+ changelog-table) id filename]))]+ (testing "a year-based directory positively identifies a modern version-less migration"+ ;; this is the case the id-shape heuristic alone cannot get right: an all-digit version-less id would+ ;; otherwise look exactly like a pre-4.2 changeset+ (doseq [id ["aeiagus09e" "12345" "09xyztest"]]+ (set-latest! id "migrations/2026/20260703_workspaces.yaml")+ (is (= liquibase/changelog-file (decide))+ (str "version-less id " id " in a year directory"))))+ (testing "version-less ids -> modern changelog even without the path signal"+ (doseq [id ["aeiagus09e" "ccdevtest01" "09xyztest"]]+ (set-latest! id "f.yaml")+ (is (= liquibase/changelog-file (decide)) id)))+ (testing "genuinely pre-4.2 numeric ids and version < 45 ids -> legacy changelog (unchanged behavior)"+ (doseq [id ["316" "v44.00-042"]]+ (set-latest! id "migrations/000_legacy_migrations.yaml")+ (is (= @#'liquibase/changelog-legacy-file (decide)) id)))+ (testing "modern version ids and the v00 marker -> modern changelog (unchanged behavior)"+ (doseq [id ["v45.00-001" "v63.abc" "v00.00-000"]]+ (set-latest! id "migrations/001_update_migrations.yaml")+ (is (= liquibase/changelog-file (decide)) id)))+ (testing "version-numbered directories (060/) are NOT year directories"+ (set-latest! "v60.abc" "migrations/060/20260101_foo.yaml")+ (is (= liquibase/changelog-file (decide))))+ (testing "rows with the same dateexecuted (e.g. MySQL second precision): higher orderexecuted decides"+ ;; the 'latest changeset' must be resolved on the [dateexecuted orderexecuted] pair like the rest of the+ ;; version machinery, not on dateexecuted alone (which leaves the winner arbitrary among ties)+ (jdbc/execute! {:connection conn} [(format "DELETE FROM %s" changelog-table)])+ (doseq [[id filename oe] [["316" "migrations/000_legacy_migrations.yaml" 1]+ ["v64.abc" "migrations/064/20260101_foo.yaml" 2]]]+ (jdbc/execute! {:connection conn}+ [(format (str "INSERT INTO %s (id, author, filename, dateexecuted, orderexecuted, exectype) "+ "VALUES (?, 't', ?, TIMESTAMP '2020-01-01 00:00:00', ?, 'EXECUTED')")+ changelog-table) id filename oe]))+ (is (= liquibase/changelog-file (decide))+ "the v64 row (orderexecuted 2) outranks the legacy row at the same timestamp"))))))))-(deftest latest-applied-major-version- (mt/test-drivers #{:h2 :mysql :postgres}- (mt/with-temp-empty-app-db [conn driver/*driver*]- (liquibase/with-liquibase [liquibase conn]- (is (nil? (liquibase/latest-applied-major-version conn (.getDatabase liquibase))))- (.update liquibase "")- (is (< 52 (liquibase/latest-applied-major-version conn (.getDatabase liquibase))))))))+(defn- filename-of [conn changelog-table id]+ (:filename (first (jdbc/query {:connection conn} [(format "SELECT filename FROM %s WHERE id = ?" changelog-table) id]))))-(deftest extract-numbers-special-case-test- (testing "when specific migration verison is passed reports different major version"- (is (= 55 (first (#'liquibase/extract-numbers "v56.2025-06-05T16:48:48"))))- (is (= 55 (first (#'liquibase/extract-numbers "v56.2025-05-19T16:48:48"))))- (is (= 60 (first (#'liquibase/extract-numbers "v60.ghdf99efd"))))))+(deftest consolidate-does-not-clobber-version-less-ids-test+ (testing "consolidate-liquibase-changesets! rewrites legacy ids' filenames but leaves version-less ids alone"+ ;; Regression: `consolidate` rewrote the filename of any id sorting before `v45.00-001`, which caught version-less+ ;; ids (e.g. `aeiagus09e` < `v45.00-001`) and pointed them at the legacy changelog file -> Liquibase then re-ran the+ ;; earliest migrations on the next boot.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (liquibase/with-liquibase [liquibase conn]+ (let [ct (liquibase/changelog-table-name liquibase)]+ ;; a legacy row (its filename ends with update_migrations.yaml) and two version-less rows in a year+ ;; directory that must not be touched+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "dep-old" :changesets [{:id "v44.00-042" :author "t" :filename "migrations/001_update_migrations.yaml"}]}+ {:deployment "dep-new" :changesets [{:id "aeiagus09e" :filename "migrations/2026/foo.yaml"}+ {:id "12345" :filename "migrations/2026/foo.yaml"}]}])+ (liquibase/consolidate-liquibase-changesets! conn liquibase)+ (is (= "migrations/000_legacy_migrations.yaml" (filename-of conn ct "v44.00-042"))+ "the legacy v44 changeset is rewritten to the consolidated legacy filename")+ (is (= "migrations/2026/foo.yaml" (filename-of conn ct "aeiagus09e"))+ "the version-less changeset's filename is left untouched")+ (is (= "migrations/2026/foo.yaml" (filename-of conn ct "12345"))+ "an all-digit version-less id in a year directory is recognized by its path, not mistaken for pre-4.2")))))))-(deftest rollback-major-version- (mt/test-drivers #{:h2 :mysql :rollback}- (mt/with-temp-empty-app-db [conn driver/*driver*]- (liquibase/with-liquibase [liquibase conn]- (.update liquibase "")- (let [actual-latest-applied-version (liquibase/latest-applied-major-version conn (.getDatabase liquibase))- actual-latest-available-version (liquibase/latest-available-major-version liquibase)]- (testing "Can downgrade and re-upgrade version"- (liquibase/rollback-major-version! conn liquibase false (dec actual-latest-available-version))- (is (= (dec actual-latest-applied-version) (liquibase/latest-applied-major-version conn (.getDatabase liquibase))))- (liquibase/rollback-major-version! conn liquibase false (- actual-latest-available-version 2))- (is (= (- actual-latest-available-version 2) (liquibase/latest-applied-major-version conn (.getDatabase liquibase))))- (.update liquibase "")- (is (= actual-latest-applied-version (liquibase/latest-applied-major-version conn (.getDatabase liquibase)))))- (testing "Cannot downgrade when there are changests from a newer version already ran which are not in the changelog file"- (mt/with-dynamic-fn-redefs [liquibase/latest-applied-major-version (constantly (inc actual-latest-applied-version))]- (is (thrown-with-msg? ExceptionInfo #"Cannot downgrade.*"- (liquibase/rollback-major-version! conn liquibase false (dec actual-latest-available-version))))- (testing "CAN downgrade if forced"- (liquibase/rollback-major-version! conn liquibase true (dec actual-latest-available-version))))))))))+(deftest consolidate-repairs-version-less-rows-rewritten-by-old-binaries-test+ (testing "consolidate-liquibase-changesets! restores the filename of version-less rows that a pre-version-less binary+ pointed at the legacy changelog"+ ;; Binaries before this change consolidate with an unguarded `WHEN ID < 'v45.00-001' THEN <legacy file>`, which+ ;; catches every version-less id sorting before `v` (most of them). Any `migrate` command of such a binary against+ ;; an upgraded DB -- even one that is then refused as a downgrade -- commits that rewrite via the lock release.+ ;; Left as-is, the newer binary's `migrate down` can no longer match those rows to its changelog: it clears their+ ;; bookkeeping without reversing their DDL, and the next upgrade fails with 'already exists'.+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [liquibase/changelog-file "versionless-dev-run1.yaml"]+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ ct (liquibase/changelog-table-name liquibase)+ legacy "migrations/000_legacy_migrations.yaml"]+ (versions/ensure-version-tracking! conn db)+ ;; the release deployment the dev changeset will be rolled back to, then the dev changeset itself+ (mdb.test-util/fabricate-history! conn ct [{:deployment "d64" :ran "x.64.0.0" :changesets ["boundary_row"]}])+ (liquibase/with-scope-locked liquibase (.update liquibase ""))+ (is (= "migrations/2026/versionless_dev.yaml" (filename-of conn ct "dev_run_a")) "sanity: applied under its year-dir path")+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET deployment_id = 'd65' WHERE id = 'dev_run_a'" ct)])+ (versions/record-deployment-version! conn "d65" "x.65.0.0" true)+ ;; what an old binary's consolidation leaves behind, plus rows it must NOT touch: a pre-4.2 numeric id+ ;; that legitimately lives in the legacy file, the legacy-version-tracking marker, and a version-less row+ ;; that no longer exists in this changelog (nothing to repair it from)+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET filename = ? WHERE id = 'dev_run_a'" ct) legacy])+ (mdb.test-util/fabricate-history! conn ct+ [{:deployment "dep" :minutes-ago 0+ :changesets [{:id "42" :author "legacy" :filename legacy}+ {:id "v65.legacy-version-tracking" :author "version-tracking" :filename "legacy-version-tracking"}+ {:id "gone_from_changelog" :filename legacy}]}])+ (liquibase/consolidate-liquibase-changesets! conn liquibase)+ (is (= "migrations/2026/versionless_dev.yaml" (filename-of conn ct "dev_run_a"))+ "the version-less row is pointed back at the changelog file that defines it")+ (is (= legacy (filename-of conn ct "42")) "a pre-4.2 numeric id stays in the legacy file")+ (is (= "legacy-version-tracking" (filename-of conn ct "v65.legacy-version-tracking")) "the marker is untouched")+ (is (= legacy (filename-of conn ct "gone_from_changelog")) "a row with no changeset to repair from is left alone")+ (testing "and the repaired row is reversed by a rollback again"+ (jdbc/execute! {:connection conn} [(format "DELETE FROM %s WHERE deployment_id = 'dep'" ct)])+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "v0.65.0")]+ (rollback/rollback-major-version! conn liquibase false "64"))+ (is (false? (versions/table-exists? "DEV_RUN_A_TABLE" conn)) "the DDL was reversed, not orphaned")+ (is (nil? (filename-of conn ct "dev_run_a")))))))))))++(deftest filename-repair-leaves-ambiguous-change-set-identities-alone-test+ (testing "filename repair does not guess when [author id] occurs in more than one year-directory file"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [liquibase/changelog-file "versionless-duplicate-ids.yaml"]+ (liquibase/with-liquibase [liquibase conn]+ (let [ct (liquibase/changelog-table-name liquibase)+ legacy "migrations/000_legacy_migrations.yaml"]+ (liquibase/with-scope-locked liquibase (.update liquibase ""))+ (jdbc/execute! {:connection conn}+ [(format "UPDATE %s SET filename = ? WHERE id = 'duplicate_versionless_id'" ct) legacy])+ (mt/with-log-messages-for-level [messages [metabase.app-db.liquibase :warn]]+ (liquibase/repair-version-less-filenames! conn liquibase ct)+ (is (some #(re-find #"Cannot safely restore.*multiple files" (:message %)) (messages))))+ (is (= [legacy legacy]+ (mapv :filename+ (jdbc/query {:connection conn}+ [(format "SELECT filename FROM %s WHERE id = 'duplicate_versionless_id' ORDER BY orderexecuted" ct)])))+ "neither row is incorrectly assigned to the other changeset's file"))))))))diff --git a/test/metabase/app_db/schema_migrations_test.clj b/test/metabase/app_db/schema_migrations_test.cljindex 9d2cce799bd..87566e889e7 100644--- a/test/metabase/app_db/schema_migrations_test.clj+++ b/test/metabase/app_db/schema_migrations_test.clj@@ -18,6 +18,7 @@51.x or older are now 'old'."(:require[clojure.java.jdbc :as jdbc]+ [clojure.set :as set][clojure.string :as str][clojure.test :refer :all][java-time.api :as t]@@ -26,6 +27,7 @@[metabase.app-db.core :as mdb][metabase.app-db.custom-migrations-test :as custom-migrations-test][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.versions :as versions][metabase.app-db.query :as mdb.query][metabase.app-db.schema-migrations-test.impl :as impl][metabase.cmd.dump-to-h2 :as dump-to-h2]@@ -54,6 +56,27 @@(binding [search.ingestion/*disable-updates* true](thunk))))+(deftest run-migrations-in-range-records-per-major-versions-test+ (testing "run-migrations-in-range! gives each legacy major its own deployment_id + databasechangelog_version row,"+ (impl/with-temp-empty-app-db [conn :h2]+ ;; from the start through v46, so the range spans many majors (we assert on two adjacent ones)+ (impl/run-migrations-in-range! conn ["v00.00-000" "v46.00-000"])+ (let [versions-table (versions/versions-table-name)+ rows (jdbc/query {:connection conn}+ [(format "SELECT deployment_id, metabase_version FROM %s" versions-table)])+ major->deps (-> (group-by #(versions/version->major (:metabase_version %)) rows)+ (update-vals #(set (map :deployment_id %))))]+ (testing "both majors in the range are recorded"+ (is (= #{"x.45.0.0"} (set (map :metabase_version (filter #(= 45 (versions/version->major (:metabase_version %))) rows))))+ "major 45 recorded as x.45.0.0")+ (is (= #{"x.46.0.0"} (set (map :metabase_version (filter #(= 46 (versions/version->major (:metabase_version %))) rows))))+ "major 46 recorded as x.46.0.0"))+ (testing "each major gets its own distinct deployment_id"+ (is (= 1 (count (get major->deps 45))))+ (is (= 1 (count (get major->deps 46))))+ (is (empty? (set/intersection (get major->deps 45) (get major->deps 46)))+ "major 45 and 46 use different deployment_ids"))))))+(defn- migrations-versions [](letfn [(form->version [form](cond@@ -173,7 +196,8 @@(impl/test-migrations ["v46.00-001" "v46.00-002"] [migrate!](let [get-last-id (fn [](-> {:datasource (mdb/app-db)}- (jdbc/query ["SELECT id FROM DATABASECHANGELOG ORDER BY ORDEREXECUTED DESC LIMIT 1"])+ ;; skip the version-tracking marker each run and rollback ends with+ (jdbc/query ["SELECT id FROM DATABASECHANGELOG WHERE filename <> 'legacy-version-tracking' ORDER BY ORDEREXECUTED DESC LIMIT 1"])first:id))](migrate!)diff --git a/test/metabase/app_db/schema_migrations_test/impl.clj b/test/metabase/app_db/schema_migrations_test/impl.cljindex 8cd1c30f2f6..d257ca08ed9 100644--- a/test/metabase/app_db/schema_migrations_test/impl.clj+++ b/test/metabase/app_db/schema_migrations_test/impl.clj@@ -17,6 +17,7 @@[metabase.app-db.custom-migrations.util :as custom-migrations.util][metabase.app-db.data-source :as mdb.data-source][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.versions :as versions][metabase.app-db.test-util :as mdb.test-util][metabase.driver :as driver][metabase.driver.sql-jdbc.connection :as sql-jdbc.conn]@@ -107,7 +108,12 @@DB with `jdbc-spec`.Range comparison uses the actual changelog order (index-based), so all ID formats — including hex-style IDs like- `v60.f8c3be` — are handled correctly without numeric conversion."+ `v60.f8c3be` — are handled correctly without numeric conversion.++ After the update, the legacy `vNN.` changesets are split by Metabase major into one synthetic deployment each, with+ a `databasechangelog_version` row apiece (see [[mdb.test-util/split-legacy-majors-into-deployments!]]): a single+ update run covers many majors but records only one deployment, and the deployment-based rollback needs a boundary+ at every major for `migrate! :down N` to find."{:added "0.41.0", :arglists '([conn [start-id end-id]][conn [start-id end-id] {:keys [inclusive-start? inclusive-end?]:or {inclusive-start? true@@ -116,42 +122,39 @@(log/debugf "Finding and running migrations %s" (range-description start-id end-id range-options))(liquibase/with-liquibase [liquibase conn]- (let [database (.getDatabase liquibase)- id->index (into {} (map-indexed (fn [i ^ChangeSet cs] [(.getId cs) i])- (.getChangeSets (.getDatabaseChangeLog liquibase))))- resolve-id (fn [id]- (or (id->index id)- (throw (ex-info (format "Migration ID not found in changelog: %s" id) {:id id}))))+ (let [database (.getDatabase liquibase)+ changelog-table (.getDatabaseChangeLogTableName database)+ changesets (vec (.getChangeSets (.getDatabaseChangeLog liquibase)))+ id->index (into {} (map-indexed (fn [i ^ChangeSet cs] [(.getId cs) i]) changesets))+ resolve-id (fn [id]+ (or (id->index id)+ (throw (ex-info (format "Migration ID not found in changelog: %s" id) {:id id})))){:keys [inclusive-start? inclusive-end?]:or {inclusive-start? true inclusive-end? true}} range-options- start-idx (resolve-id start-id)- end-idx (when end-id (resolve-id end-id))+ start-idx (resolve-id start-id)+ end-idx (when end-id (resolve-id end-id))+ in-range? (fn [idx]+ (and (some? idx)+ (if inclusive-start? (<= start-idx idx) (< start-idx idx))+ (if end-idx (if inclusive-end? (<= idx end-idx) (< idx end-idx)) true)))change-set-filters [(reify ChangeSetFilter(accepts [this change-set](let [id (.getId ^ChangeSet change-set)- idx (id->index id)- accept? (boolean- (and (some? idx)- (if inclusive-start?- (<= start-idx idx)- (< start-idx idx))- (if end-idx- (if inclusive-end?- (<= idx end-idx)- (< idx end-idx))- true)))]+ accept? (boolean (in-range? (id->index id)))](log/tracef "Migration %s in range [%s ↔ %s] %s ? => %s"id start-id end-id(if inclusive-end? "(inclusive)" "(exclusive)")accept?)(ChangeSetFilterResult. accept? "decision according to range" (class this)))))]change-log-service (.getChangeLogService (ChangeLogHistoryServiceFactory/getInstance) database)]+ (versions/ensure-version-tracking! conn database)(liquibase/with-scope-locked liquibase;; Calling .listUnrunChangeSets has the side effect of creating the Liquibase tables;; and initializing checksums so that they match the ones generated in production.(.listUnrunChangeSets liquibase nil (LabelExpression.))(.generateDeploymentId change-log-service)- (liquibase/update-with-change-log liquibase {:change-set-filters change-set-filters})))))+ (liquibase/update-with-change-log liquibase {:change-set-filters change-set-filters}))+ (mdb.test-util/split-legacy-majors-into-deployments! conn changelog-table))))(defn test-migrations-for-driver! [driver [start-id end-id] f](log/debug (u/format-color 'yellow "Testing migrations for driver %s..." driver))@@ -183,10 +186,16 @@:down(do(assert (int? version), "Downgrade requires a version")- (mdb/migrate! (mdb/data-source) :down version)+ ;; force: these legacy tests roll back to arbitrary majors, often further back than the default+ ;; rollback window (current major + previous-major boundary). force widens that window to the full+ ;; recorded history; each major is recorded by run-migrations-in-range! above, so every such target+ ;; is a valid recorded version. The default-window rule itself is covered in metabase.app-db.liquibase-test.+ (mdb/migrate! (mdb/data-source) :down-force version);; We may have rolled back migrations prior to start-id, so its no longer safe to start from there.+ ;; The rollback ends with a version-tracking marker row, which is not a changeset to restart from.(reset! restart-id (t2/select-one-pk (liquibase/changelog-table-name conn)- {:order-by [[:orderexecuted :desc]]}))))))]+ {:where [:not= :filename "legacy-version-tracking"]+ :order-by [[:orderexecuted :desc]]}))))))](f migrate))))(log/debug (u/format-color 'green "Done testing migrations for driver %s." driver)))diff --git a/test/metabase/app_db/setup_test.clj b/test/metabase/app_db/setup_test.cljindex 980876fb312..f1427634a5d 100644--- a/test/metabase/app_db/setup_test.clj+++ b/test/metabase/app_db/setup_test.clj@@ -1,6 +1,7 @@(ns ^:mb/app-db-migrations-test metabase.app-db.setup-test(:require[clojure.java.jdbc :as jdbc]+ [clojure.string :as str][clojure.test :refer :all][metabase.app-db.connection :as mdb.connection][metabase.app-db.connection-pool-setup :as mdb.connection-pool-setup]@@ -9,11 +10,14 @@[metabase.app-db.db :as mdb.db][metabase.app-db.encryption :as mdb.encryption][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.versions :as versions][metabase.app-db.setting :as mdb.setting][metabase.app-db.setup :as mdb.setup][metabase.app-db.test-util :as mdb.test-util]+ [metabase.config.core :as config][metabase.driver :as driver][metabase.test :as mt]+ [metabase.util :as u][metabase.util.encryption :as encryption][metabase.util.encryption-test :as encryption-test][toucan2.core :as t2])@@ -106,8 +110,10 @@(is (= :done(mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) {:create-sample-content? true})))(testing "migrations are executed in the order they are defined"+ ;; the vNN.legacy-version-tracking marker is bookkeeping written by the migration run, not a changeset(is (= (mdb.test-util/all-liquibase-ids false driver/*driver* conn)- (t2/select-pks-vec (liquibase/changelog-table-name conn) {:order-by [[:orderexecuted :asc]]}))))))))+ (remove #(re-find #"legacy-version-tracking$" %)+ (t2/select-pks-vec (liquibase/changelog-table-name conn) {:order-by [[:orderexecuted :asc]]})))))))))(deftest setup-db-no-auto-migrate-test(mt/test-drivers #{:h2 :mysql :postgres}@@ -159,22 +165,147 @@(mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source)))))))(deftest downgrade-detection-test+ ;; Simulates an existing instance that predates version tracking: it has no databasechangelog_version table, so+ ;; downgrade detection lazily backfills the recorded version from the changeset ids. A fresh DB is used per version+ ;; because `update-to-changelog-id` runs migrations without the recording listener -- the backfill snapshots the+ ;; state at first read, so reusing one DB across versions would not reflect later migrations.(mt/test-drivers #{:h2 :mysql :postgres}- (mt/with-temp-empty-app-db [conn driver/*driver*]- ;; migrate to v45+ (doseq [version [45 46]]+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (update-to-changelog-id (format "v%d.00-001" version) conn)+ ;; pretend this binary is v0.44 -- older than the v45/v46 the DB was migrated to, so it must be a downgrade+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "v0.44.0")]+ (is (thrown-with-msg?+ Exception (re-pattern (format "You must run `java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar migrate down` from version %d." version))+ (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source)))))))))++(deftest downgrade-detection-recorded-version-test+ (testing "downgrade detection goes off the recorded version of the last deployment; a synthetic (dev) version warns"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source))+ (liquibase/with-liquibase [liquibase conn]+ (let [db (.getDatabase liquibase)+ versions-table (versions/versions-table-name)+ ;; an arbitrary released major to play "this binary's version" -- every recorded version below is+ ;; fabricated relative to it (999 above it, the dev version further above), so its exact value is irrelevant+ binary-major 64]+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag (format "v0.%d.0" binary-major))]+ (testing "a recorded development version blocks a real binary, pointing at the dev tooling"+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET metabase_version = '%s'" versions-table versions/dev-version)])+ (is (= versions/dev-version (versions/last-deployment-version conn db)))+ (is (thrown-with-msg?+ clojure.lang.ExceptionInfo #"(?s)development build.*dev\.migrate"+ (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source)))))+ (testing "a recorded real version newer than this binary is a genuine downgrade and blocks"+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET metabase_version = 'x.999.0'" versions-table)])+ (is (= "x.999.0" (versions/last-deployment-version conn db)))+ (is (thrown-with-msg?+ Exception #"migrate down` from version 999"+ (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source)))))+ (testing "does not throw when the recorded version is not newer than this binary"+ (jdbc/execute! {:connection conn} [(format "UPDATE %s SET metabase_version = 'x.%d.0'" versions-table binary-major)])+ (is (nil? (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source))))))))))))++(deftest noop-boot-of-newer-binary-preserves-downgrade-test+ (testing "a newer binary that boots without running any migrations must not block the previous binary from booting"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source))+ (liquibase/with-liquibase [_liquibase conn]+ (let [versions-table (versions/versions-table-name)+ ;; an arbitrary released major playing the installing binary's version; the newer binaries below are+ ;; fabricated relative to it, so its exact value is irrelevant+ latest 64]+ ;; make the recorded install version deterministic: this DB was installed by v0.<latest>+ (jdbc/execute! {:connection conn}+ [(format "UPDATE %s SET metabase_version = 'x.%d.0'" versions-table latest)])+ (testing "a no-op boot of the NEXT major is recorded as history, but does not move the schema's version"+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag (format "v0.%d.0" (inc latest)))]+ (is (nil? (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source))))+ (mdb.setup/migrate! (mdb.connection/data-source) :up))+ (is (= 1 (count (jdbc/query {:connection conn}+ [(format "SELECT 1 FROM %s WHERE metabase_version LIKE 'x.%d.%%'"+ versions-table (inc latest))])))+ "the newer major's boot is captured in history"))+ (testing "the previous binary still boots"+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag (format "v0.%d.0" latest))]+ (is (nil? (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source))))))+ (testing "even a much newer boot row on the deployment cannot mask the version that ran it"+ (let [last-dep (:deployment_id (first (jdbc/query {:connection conn}+ [(format "SELECT deployment_id FROM %s LIMIT 1" versions-table)])))]+ (versions/record-deployment-version! conn last-dep (format "x.%d.0" (+ latest 2)) false))+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag (format "v0.%d.0" latest))]+ (is (nil? (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source)))+ "the stamp does not block the binary whose major actually ran the deployment")))))))))++(deftest downgrade-check-creates-persistent-version-table-test+ (testing "the version table lazily created during downgrade detection persists (committed) for other connections"+ ;; This guards the in-memory \"already created\" tracking in ensure-databasechangelog-versions-table!: if the+ ;; CREATE ran on a pooled, autocommit-off connection and were rolled back on close, a later connection that trusts+ ;; the in-memory flag would skip creation and then fail. Most relevant on Postgres (transactional DDL).+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ ;; simulate an existing instance (has changesets, no version table yet)+ (update-to-changelog-id "v45.00-001" conn)+ ;; latest-available >> 45, so this does NOT throw; it lazily creates + backfills the version table+ (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source))+ (testing "version table exists and is populated when read from a separate connection"+ (is (seq (jdbc/query {:datasource (mdb.connection/data-source)}+ [(format "SELECT deployment_id, metabase_version FROM %s"+ (versions/versions-table-name))]))))))))++(deftest migrations-sql-includes-version-tracking-test+ (testing "the manual-upgrade SQL (`migrate print`) records the version bookkeeping"+ ;; Regression: the version row and the vNN.legacy-version-tracking marker are written by the exec listener, which+ ;; only runs when Metabase itself executes the migrations. A manual `migrate print` + apply-the-SQL upgrade left+ ;; neither, silently disabling downgrade detection for both older binaries (which read the marker) and newer ones+ ;; (which read databasechangelog_version).+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ ;; an existing pre-version-tracking install partway through history, with plenty of unrun changesets+ (update-to-changelog-id "v45.00-001" conn)+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "v0.64.0")]+ (liquibase/with-liquibase [liquibase conn]+ (let [sql (liquibase/migrations-sql liquibase)]+ (testing "backfills the pre-upgrade version for the existing install"+ (is (re-find #"(?i)INSERT (IGNORE )?INTO databasechangelog_version[^;]+x\.45\.0\.0" sql)))+ (testing "records the upgrading version"+ (is (re-find #"(?i)INSERT (IGNORE )?INTO databasechangelog_version[^;]+x\.64\.0" sql)))+ (testing "ends with the marker older binaries read safely as the newest row (a versioned major's own ids+ give them its major, so it is the low position marker)"+ (is (re-find #"v56\.legacy-version-tracking" sql)))+ (testing "the version table is created by the printed SQL, not on the live database"+ (is (re-find #"(?i)CREATE TABLE IF NOT EXISTS databasechangelog_version" sql))+ (is (false? (versions/versions-table-exists? conn))+ "`migrate print` must not mutate the database"))))))))+ (testing "a changeset that the printed SQL re-runs has its first run recorded before Liquibase's UPDATE moves it"+ (mt/test-drivers #{:h2 :mysql :postgres}+ (mt/with-temp-empty-app-db [conn driver/*driver*]+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "v0.66.0")+ liquibase/changelog-file "versionless-roc-run1.yaml"]+ (mdb/migrate! (mdb/data-source) :up))+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "v0.66.1")+ liquibase/changelog-file "versionless-roc-run3.yaml"]+ (liquibase/with-liquibase [liquibase conn]+ (let [sql (liquibase/migrations-sql liquibase)+ capture (some-> (re-find #"(?i)INSERT INTO databasechangelog_first_run[^;]+'roc_view'" sql) u/lower-case-en)+ update (re-find #"(?i)UPDATE [^;]*databasechangelog[^;]+roc_view" sql)]+ (is (some? capture))+ (is (some? update) "sanity: Liquibase re-runs the view")+ (is (< (str/index-of (u/lower-case-en sql) capture) (str/index-of sql update)))+ (is (re-find #"(?i)CREATE TABLE IF NOT EXISTS databasechangelog_first_run" sql))))))))+ (testing "a dev build's SQL records the dev version and a v9999 marker, like any other version"+ (mt/with-temp-empty-app-db [conn :h2](update-to-changelog-id "v45.00-001" conn)- ;; the latest changeSet in `000_legacy_migrations.yaml` is `v44.00-044`. We can simulate a downgrade to that- ;; version by telling Liquibase that's the migrations file.- (mt/with-dynamic-fn-redefs [liquibase/decide-liquibase-file (fn [& _args] "liquibase_legacy_migrations.yaml")]- (is (thrown-with-msg?- Exception #"You must run `java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar migrate down` from version 45."- (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source)))))- ;; check that the error correctly reports the version to run `downgrade` from- (update-to-changelog-id "v46.00-001" conn)- (mt/with-dynamic-fn-redefs [liquibase/decide-liquibase-file (fn [& _args] "liquibase_legacy_migrations.yaml")]- (is (thrown-with-msg?- Exception #"You must run `java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar migrate down` from version 46."- (#'mdb.setup/error-if-downgrade-required! (mdb.connection/data-source))))))))+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag "vLOCAL_DEV")]+ (liquibase/with-liquibase [liquibase conn]+ (let [sql (liquibase/migrations-sql liquibase)]+ (is (re-find #"(?i)INSERT INTO databasechangelog_version" sql))+ (is (re-find #"'x\.9999\.0\.0'" sql))+ (is (re-find #"v9999\.legacy-version-tracking" sql))+ (is (false? (versions/versions-table-exists? conn))+ "computing the dev synthetic version on the print path must not create the table")))))))(deftest changesets-from-later-version-test(mt/test-drivers #{:h2 :mysql :postgres}@@ -192,15 +323,33 @@VALUES (?, 'test', 'test.yaml', CURRENT_TIMESTAMP, ?, 'EXECUTED', 'fake')"table)id (+ 99990 i)]))+ ;; ... plus a version-less changeset applied by a later deployment: nothing in its id reveals its version,+ ;; only its deployment's recorded ran-version does+ (jdbc/execute! db-conn+ [(format "INSERT INTO %s (id, author, filename, dateexecuted, orderexecuted, exectype, md5sum, deployment_id)+ VALUES ('zz_versionless_cs', 'test', 'migrations/2026/test.yaml', CURRENT_TIMESTAMP, 99993, 'EXECUTED', 'fake', 'futuredep')"+ table)])+ ;; ... plus an OLD changeset that merely re-ran (RERAN) under that later deployment (an edited runOnChange+ ;; changeset): it is not "from" the later version and must not be listed+ (jdbc/execute! db-conn+ [(format "INSERT INTO %s (id, author, filename, dateexecuted, orderexecuted, exectype, md5sum, deployment_id)+ VALUES ('zz_reran_cs', 'test', 'migrations/2026/test.yaml', CURRENT_TIMESTAMP, 99994, 'RERAN', 'fake', 'futuredep')"+ table)])+ (versions/record-deployment-version! conn "futuredep" "x.999.1" true)(try- (let [later (liquibase/changesets-from-later-version conn (.getDatabase liquibase) 997 999)]- (testing "returns exactly the fake changeset IDs in execution order"- (is (= fake-ids later))))+ (let [later (versions/changesets-from-later-version conn (.getDatabase liquibase) 997 999)]+ (testing "returns the versioned AND version-less changeset IDs in execution order"+ (is (= (conj fake-ids "zz_versionless_cs") later)))+ (testing "re-run (RERAN) rows of the later deployment are not listed"+ (is (not (some #{"zz_reran_cs"} later)))))(finally;; Clean up fake rows- (doseq [id fake-ids]+ (doseq [id (conj fake-ids "zz_versionless_cs" "zz_reran_cs")](jdbc/execute! db-conn- [(format "DELETE FROM %s WHERE id = ?" table) id])))))))))+ [(format "DELETE FROM %s WHERE id = ?" table) id]))+ (jdbc/execute! db-conn+ [(format "DELETE FROM %s WHERE deployment_id = 'futuredep'"+ (versions/versions-table-name))]))))))));; `delete!` below is ok in a parallel test since it's not actually executing anything#_{:clj-kondo/ignore [:metabase/validate-deftest]}diff --git a/test/metabase/app_db/test_util.clj b/test/metabase/app_db/test_util.cljindex 28284032b4e..c7bba5e9ce3 100644--- a/test/metabase/app_db/test_util.clj+++ b/test/metabase/app_db/test_util.clj@@ -5,6 +5,7 @@[clojure.string :as str][clojure.test :refer :all][metabase.app-db.core :as mdb]+ [metabase.app-db.liquibase.versions :as versions][metabase.driver.test-util :as driver.tu][metabase.test.util.timezone :as test.tz][metabase.util.random :as u.random]@@ -99,14 +100,15 @@;; Per-release migration files (v56-v59 pattern)(filter io/resource (for [n (range 56 100)](format "migrations/%03d_update_migrations.yaml" n)))- ;; Directory-based migration files (v60+ pattern)+ ;; Directory-based migration files: version-numbered directories (v60-v65, `060/...`) and the year-based+ ;; directories of version-less changesets (`2026/...`)(let [migrations-dir (io/resource "migrations")](when migrations-dir(->> (io/file migrations-dir)file-seq(filter (fn [^java.io.File f](and (.isFile f)- (re-matches #".*\d{3}/\d{8}_.+\.yaml$" (str f)))))+ (re-matches #".*(\d{3}|\d{4})/\d{8}_.+\.yaml$" (str f)))))sort(map (fn [^java.io.File f](str "migrations/" (.getName (.getParentFile f)) "/" (.getName f))))))))))@@ -115,3 +117,88 @@"Returns a set of all changeset IDs from all migration files."[include-legacy? driver conn](apply concat (map #(liquibase-file->included-ids % driver conn) (all-migration-files include-legacy?))))++;;; ------------------------------------------ fabricated deployment histories ------------------------------------++(defn age-changelog-rows!+ "Rewind every existing `databasechangelog` row's `dateexecuted` by `minutes` (default 1); their relative order is+ preserved by the `orderexecuted` tiebreak. Required before layering fabricated rows or a second migration run on top+ of rows Liquibase just wrote: MySQL's second-precision DATEEXECUTED **rounds** sub-second JDBC timestamps, so a row+ Liquibase wrote at xx.5s is stored one second in the FUTURE, and a row written shortly after it (CURRENT_TIMESTAMP+ truncates) can sort at-or-before it. Real deployments are separated by real wall-clock time; this restores that+ separation for back-to-back test writes.++ The rewind is computed ON THE SERVER (`CURRENT_TIMESTAMP - INTERVAL '<n>' MINUTE`, valid on H2/Postgres/MySQL): a+ client-side `java.sql.Timestamp` parameter is rendered in the JVM's zone while Liquibase's rows and+ `CURRENT_TIMESTAMP` land in the server's frame, so on a non-UTC dev machine against a UTC MySQL the aged rows would+ jump hours into the future instead.++ The first runs copied from changelog rows (see [[versions/first-runs]]) are aged with them, so they keep comparing+ with the changelog the same way."+ ([spec changelog-table]+ (age-changelog-rows! spec changelog-table 1))+ ([spec changelog-table minutes]+ (let [first-runs? (if-let [conn (:connection spec)]+ (versions/table-exists? (versions/first-runs-table-name) conn)+ (with-open [conn (.getConnection ^javax.sql.DataSource (:datasource spec))]+ (versions/table-exists? (versions/first-runs-table-name) conn)))]+ (doseq [table (cond-> [changelog-table] first-runs? (conj (versions/first-runs-table-name)))]+ (jdbc/execute! spec [(format "UPDATE %s SET dateexecuted = CURRENT_TIMESTAMP - INTERVAL '%d' MINUTE"+ table (long minutes))])))))++(defn- insert-changelog-row!+ [conn changelog-table {:keys [id author filename exectype] :or {author "test", filename "synthetic.yaml", exectype "EXECUTED"}}+ deployment-id minutes-ago]+ (jdbc/execute! {:connection conn}+ [(format (str "INSERT INTO %s (id, author, filename, dateexecuted, orderexecuted, exectype, deployment_id) "+ "SELECT ?, ?, ?, CURRENT_TIMESTAMP - INTERVAL '%d' MINUTE, COALESCE(MAX(orderexecuted), 0) + 1, ?, ? FROM %s")+ changelog-table (long minutes-ago) changelog-table)+ id author filename exectype deployment-id]))++(defn fabricate-history!+ "Insert a fabricated deployment history into the application database on `conn`, oldest entry first, as the newest+ history of the changelog (rows already there are first aged to before the oldest entry). Each entry is++ {:deployment \"d64\" ; its deployment_id+ :ran \"x.64.0\" ; the version recorded as having run it (optional)+ :booted [\"x.65.0\"] ; versions recorded as having booted against it (optional)+ :changesets [\"c64\" {:id \"v64.abc\", :author \"t\", :filename \"f.yaml\", :exectype \"RERAN\"}] ; its changelog rows+ :minutes-ago 20} ; dateexecuted of those rows; defaults to ten minutes per step++ A changeset given as a string is its id with the defaults `test`/`synthetic.yaml`/`EXECUTED`. The version table must+ exist (see [[versions/ensure-version-tracking!]])."+ [conn changelog-table entries]+ (let [n (count entries)+ minutes-ago (fn [i {:keys [minutes-ago]}] (or minutes-ago (* 10 (- n i))))+ oldest (apply max 1 (map-indexed minutes-ago entries))]+ (age-changelog-rows! {:connection conn} changelog-table (inc oldest))+ (doseq [[i {:keys [deployment ran booted changesets] :as entry}] (map-indexed vector entries)+ :let [ago (minutes-ago i entry)]]+ (doseq [cs changesets]+ (insert-changelog-row! conn changelog-table (if (string? cs) {:id cs} cs) deployment ago))+ (when ran+ (versions/record-deployment-version! conn deployment ran true))+ (doseq [version booted]+ (versions/record-deployment-version! conn deployment version false)))))++(defn split-legacy-majors-into-deployments!+ "Give every legacy `vNN.` major in the changelog its own deployment (`mjr0NN`) with an `x.NN.0.0` ran-version row, as+ if the database had been upgraded release by release: a single update run covers many majors but records only one+ deployment and one version, while the deployment-based rollback needs a boundary at every major. Version-less ids+ keep their deployment. Version rows of deployments left without any changelog row are removed. Idempotent."+ [conn changelog-table]+ (let [ids (map :id (jdbc/query {:connection conn} [(format "SELECT id FROM %s" changelog-table)]))+ major-of (fn [id] (some-> (re-find #"^v(\d+)\." id) second parse-long))]+ (doseq [[major block-ids] (sort-by key (group-by major-of (filter major-of ids)))]+ (let [deployment-id (format "mjr%03d" major)]+ (jdbc/execute! {:connection conn}+ (into [(format "UPDATE %s SET deployment_id = ? WHERE id IN (%s)"+ changelog-table+ (str/join ", " (repeat (count block-ids) "?")))+ deployment-id]+ block-ids))+ (versions/record-deployment-version! conn deployment-id (versions/major->version major) true)))+ (jdbc/execute! {:connection conn}+ [(format "DELETE FROM %s WHERE deployment_id NOT IN (SELECT DISTINCT deployment_id FROM %s WHERE deployment_id IS NOT NULL)"+ (versions/versions-table-name)+ changelog-table)])))diff --git a/test/metabase/cmd/compare_h2_dbs_test_util.clj b/test/metabase/cmd/compare_h2_dbs_test_util.cljindex 68a5662bef5..2126cd36e71 100644--- a/test/metabase/cmd/compare_h2_dbs_test_util.clj+++ b/test/metabase/cmd/compare_h2_dbs_test_util.clj@@ -25,6 +25,7 @@"Set of Table names to skip diffing (e.g. because they're not ones we migrate.)"#{"CORE_SESSION""DATABASECHANGELOG"+ "DATABASECHANGELOG_VERSION""QRTZ_BLOB_TRIGGERS""QRTZ_CALENDARS""QRTZ_CRON_TRIGGERS"diff --git a/test/metabase/cmd/load_from_h2_test.clj b/test/metabase/cmd/load_from_h2_test.cljindex 3ad4cf1a8b2..3b70ce60b6d 100644--- a/test/metabase/cmd/load_from_h2_test.clj+++ b/test/metabase/cmd/load_from_h2_test.clj@@ -5,12 +5,15 @@[clojure.test :refer :all][metabase.app-db.connection :as mdb.connection][metabase.app-db.liquibase :as liquibase]+ [metabase.app-db.liquibase.rollback :as rollback]+ [metabase.app-db.liquibase.versions :as versions][metabase.app-db.setup :as mdb.setup][metabase.app-db.test-util :as mdb.test-util][metabase.cmd.copy :as copy][metabase.cmd.dump-to-h2 :as dump-to-h2][metabase.cmd.load-from-h2 :as load-from-h2][metabase.cmd.test-util :as cmd.test-util]+ [metabase.config.core :as config][metabase.driver :as driver][metabase.driver.sql-jdbc.connection :as sql-jdbc.conn][metabase.search.core :as search]@@ -82,15 +85,14 @@(t2.conn/with-connection [conn](liquibase/with-liquibase [liquibase conn](let [change-sets (.. liquibase getDatabaseChangeLog getChangeSets)- size (count change-sets)]- (if (pos? size)- (let [change-set-id (.getId ^ChangeSet (.get change-sets (dec size)))- [_ major] (re-find #"v(\d+).*" change-set-id)]- (if major- (parse-long major)- (throw (ex-info "couldn't parse major version from change-set-id " change-set-id- {:change-set-id change-set-id}))))- (throw (ex-info "no changesets found" {})))))))+ ;; the highest `vNN.` major in the changelog. Year-directory migrations (`migrations/2026/...`) have+ ;; version-less ids and sort last, so "the last changeset's id" no longer necessarily carries a version.+ majors (keep (fn [^ChangeSet cs]+ (some-> (re-find #"^v(\d+)\." (.getId cs)) second parse-long))+ change-sets)]+ (if (seq majors)+ (apply max majors)+ (throw (ex-info "no versioned changesets found" {})))))))(def ^:private current-major-version;; We are interested in the latest version we started preparing@@ -100,6 +102,16 @@;; interesting.)(delay (liquibase-latest-major-version)))+(defn- fabricate-per-major-version-history!+ "A fresh install records only its own version, and [[rollback/rollback-major-version!]] only accepts majors recorded+ for a deployment -- so split the install's single deployment into one per legacy `vNN.` major, as if this database+ had really been upgraded release by release, giving the rollback a boundary at every major."+ []+ (t2.conn/with-connection [conn]+ (liquibase/with-liquibase [liquibase conn]+ (versions/ensure-version-tracking! conn (.getDatabase liquibase))+ (mdb.test-util/split-legacy-majors-into-deployments! conn (liquibase/changelog-table-name liquibase)))))+(defn- migrate-down-then-up-and-create-dump(t2.conn/with-connection [conn](liquibase/with-liquibase [liquibase conn]- (liquibase/rollback-major-version! conn liquibase false version))))+ ;; force: targets more than one major back are outside the default rollback window (current major ++ ;; previous-major boundary); force widens it to the full recorded history. Every target is a recorded+ ;; version courtesy of [[fabricate-per-major-version-history!]]. The default-window rule itself is+ ;; covered in metabase.app-db.liquibase-test.+ (rollback/rollback-major-version! conn liquibase true version))))(log/info "creating dump" filename);; this migrates the DB back to the newest and creates a dump(dump-to-h2/dump-to-h2! filename)@@ -161,10 +177,15 @@db-type driver/*driver*source-db-def {:database-name "load-test-source"}data-source (get-data-source db-type source-db-def)]- ;; Create "load-test-source" once and then reuse it because creating it for each supported version, running all- ;; migrations and populating with data takes a lot of time.- (log/info "creating database")- (create-current-database! db-type source-db-def data-source)- (doseq [version versions]- (migrate-down-then-up-and-create-dump! source-db-def h2-filename version)- (load-dump! "load-test-target" h2-filename version))))))+ ;; migrate as a release of the current major: every deployment a development build makes records the same dev+ ;; version, which `migrate down` refuses to roll back from+ (with-redefs [config/mb-version-info (assoc config/mb-version-info :tag (format "v0.%d.0" current-version))]+ ;; Create "load-test-source" once and then reuse it because creating it for each supported version, running+ ;; all migrations and populating with data takes a lot of time.+ (log/info "creating database")+ (create-current-database! db-type source-db-def data-source)+ (binding [mdb.connection/*application-db* (mdb.connection/application-db db-type data-source)]+ (fabricate-per-major-version-history!))+ (doseq [version versions]+ (migrate-down-then-up-and-create-dump! source-db-def h2-filename version)+ (load-dump! "load-test-target" h2-filename version)))))))diff --git a/test_resources/duplicate_identity/2026/versionless_duplicate_a.yaml b/test_resources/duplicate_identity/2026/versionless_duplicate_a.yamlnew file mode 100644index 00000000000..c43beee93c5--- /dev/null+++ b/test_resources/duplicate_identity/2026/versionless_duplicate_a.yaml@@ -0,0 +1,11 @@+databaseChangeLog:+ - changeSet:+ id: duplicate_versionless_id+ author: test+ changes:+ - createTable:+ tableName: duplicate_versionless_a+ columns:+ - column:+ name: id+ type: intdiff --git a/test_resources/duplicate_identity/2027/versionless_duplicate_b.yaml b/test_resources/duplicate_identity/2027/versionless_duplicate_b.yamlnew file mode 100644index 00000000000..d38db932f6a--- /dev/null+++ b/test_resources/duplicate_identity/2027/versionless_duplicate_b.yaml@@ -0,0 +1,11 @@+databaseChangeLog:+ - changeSet:+ id: duplicate_versionless_id+ author: test+ changes:+ - createTable:+ tableName: duplicate_versionless_b+ columns:+ - column:+ name: id+ type: intdiff --git a/test_resources/versioned-roc-run1.yaml b/test_resources/versioned-roc-run1.yamlnew file mode 100644index 00000000000..8a7d32feb61--- /dev/null+++ b/test_resources/versioned-roc-run1.yaml@@ -0,0 +1,45 @@+# Test changelog for metabase.app-db.liquibase.rollback-test/rollback-reverses-reran-changesets-newer-than-the-target-test.+# Simulates 65.0: a table, a column and a runOnChange view over that column, all with v65 ids. `versioned-roc-run2.yaml`+# is 65.1's changelog, where only the view's SQL was edited in place (same changeset identity, changed checksum).+databaseChangeLog:+ - logicalFilePath: migrations/065/versioned_roc.yaml+ - changeSet:+ id: v65.roc-table+ author: test+ comment: table of 65.0 (scenario test)+ changes:+ - createTable:+ tableName: vroc_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: vroc_table+ - changeSet:+ id: v65.roc-column+ author: test+ comment: column of 65.0 that the view reads (scenario test)+ changes:+ - addColumn:+ tableName: vroc_table+ columns:+ - column:+ name: extra+ type: int+ rollback:+ - dropColumn:+ tableName: vroc_table+ columnName: extra+ - changeSet:+ id: v65.roc-view+ author: test+ runOnChange: true+ comment: runOnChange view over the 65.0 column (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_vroc AS SELECT extra, 1 AS x FROM vroc_table+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_vrocdiff --git a/test_resources/versioned-roc-run2.yaml b/test_resources/versioned-roc-run2.yamlnew file mode 100644index 00000000000..1aef1a58e1f--- /dev/null+++ b/test_resources/versioned-roc-run2.yaml@@ -0,0 +1,45 @@+# Test changelog for metabase.app-db.liquibase.rollback-test/rollback-reverses-reran-changesets-newer-than-the-target-test.+# `versioned-roc-run1.yaml` as shipped by 65.1: only the runOnChange view's SQL was edited in place, so Liquibase+# re-runs it (RERAN) and moves its changelog row into 65.1's deployment.+databaseChangeLog:+ - logicalFilePath: migrations/065/versioned_roc.yaml+ - changeSet:+ id: v65.roc-table+ author: test+ comment: table of 65.0 (scenario test)+ changes:+ - createTable:+ tableName: vroc_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: vroc_table+ - changeSet:+ id: v65.roc-column+ author: test+ comment: column of 65.0 that the view reads (scenario test)+ changes:+ - addColumn:+ tableName: vroc_table+ columns:+ - column:+ name: extra+ type: int+ rollback:+ - dropColumn:+ tableName: vroc_table+ columnName: extra+ - changeSet:+ id: v65.roc-view+ author: test+ runOnChange: true+ comment: runOnChange view over the 65.0 column (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_vroc AS SELECT extra, 2 AS x FROM vroc_table+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_vrocdiff --git a/test_resources/versionless-always.yaml b/test_resources/versionless-always.yamlnew file mode 100644index 00000000000..7374fb1973f--- /dev/null+++ b/test_resources/versionless-always.yaml@@ -0,0 +1,30 @@+# Test changelog for the runAlways tests in metabase.app-db.liquibase.versions-test and rollback-test.+# A base changeset and a runAlways view: every update run re-runs the view (Liquibase passes ALREADY_RAN, not+# RUN_AGAIN, for it) and moves its changelog row into that run's deployment.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_always.yaml+ - changeSet:+ id: always_base+ author: test+ comment: base migration (scenario test)+ changes:+ - createTable:+ tableName: always_base_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: always_base_table+ - changeSet:+ id: always_view+ author: test+ runAlways: true+ comment: runAlways view (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_always AS SELECT 1 AS x+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_alwaysdiff --git a/test_resources/versionless-dev-run1.yaml b/test_resources/versionless-dev-run1.yamlnew file mode 100644index 00000000000..e81e2753b9f--- /dev/null+++ b/test_resources/versionless-dev-run1.yaml@@ -0,0 +1,16 @@+# Test changelog for metabase.app-db.liquibase-test/dev-consecutive-migration-runs-are-separate-deployments-test.+# Simulates a dev instance's FIRST migration run. `versionless-dev-run2.yaml` is the same changelog after the+# developer adds a second migration; a shared logicalFilePath keeps changeset identity stable across the two files.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_dev.yaml+ - changeSet:+ id: dev_run_a+ author: test+ comment: first dev migration run (scenario test)+ changes:+ - createTable:+ tableName: dev_run_a_table+ columns:+ - column:+ name: id+ type: intdiff --git a/test_resources/versionless-dev-run2.yaml b/test_resources/versionless-dev-run2.yamlnew file mode 100644index 00000000000..f09e9385dfc--- /dev/null+++ b/test_resources/versionless-dev-run2.yaml@@ -0,0 +1,27 @@+# Test changelog for metabase.app-db.liquibase-test/dev-consecutive-migration-runs-are-separate-deployments-test.+# `versionless-dev-run1.yaml` plus a second migration the developer added; the shared logicalFilePath keeps+# changeset identity stable across the two files.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_dev.yaml+ - changeSet:+ id: dev_run_a+ author: test+ comment: first dev migration run (scenario test)+ changes:+ - createTable:+ tableName: dev_run_a_table+ columns:+ - column:+ name: id+ type: int+ - changeSet:+ id: dev_run_b+ author: test+ comment: second dev migration run (scenario test)+ changes:+ - createTable:+ tableName: dev_run_b_table+ columns:+ - column:+ name: id+ type: intdiff --git a/test_resources/versionless-duplicate-ids.yaml b/test_resources/versionless-duplicate-ids.yamlnew file mode 100644index 00000000000..e2297f14035--- /dev/null+++ b/test_resources/versionless-duplicate-ids.yaml@@ -0,0 +1,5 @@+databaseChangeLog:+ - include:+ file: duplicate_identity/2026/versionless_duplicate_a.yaml+ - include:+ file: duplicate_identity/2027/versionless_duplicate_b.yamldiff --git a/test_resources/versionless-late-roc-run1.yaml b/test_resources/versionless-late-roc-run1.yamlnew file mode 100644index 00000000000..17f1ab0d820--- /dev/null+++ b/test_resources/versionless-late-roc-run1.yaml@@ -0,0 +1,45 @@+# Test changelog for metabase.app-db.liquibase.rollback-test/rollback-reverses-version-less-reran-changesets-newer-than-the-target-test.+# Simulates 65.0: a table, a column and a runOnChange view over that column, all version-less. `versionless-late-roc-run2.yaml`+# is 65.1's changelog, where only the view's SQL was edited in place (same changeset identity, changed checksum).+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_late_roc.yaml+ - changeSet:+ id: late_roc_table+ author: test+ comment: table of 65.0 (scenario test)+ changes:+ - createTable:+ tableName: lroc_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: lroc_table+ - changeSet:+ id: late_roc_column+ author: test+ comment: column of 65.0 that the view reads (scenario test)+ changes:+ - addColumn:+ tableName: lroc_table+ columns:+ - column:+ name: extra+ type: int+ rollback:+ - dropColumn:+ tableName: lroc_table+ columnName: extra+ - changeSet:+ id: late_roc_view+ author: test+ runOnChange: true+ comment: runOnChange view over the 65.0 column (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_lroc AS SELECT extra, 1 AS x FROM lroc_table+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_lrocdiff --git a/test_resources/versionless-late-roc-run2.yaml b/test_resources/versionless-late-roc-run2.yamlnew file mode 100644index 00000000000..60b691a5f3d--- /dev/null+++ b/test_resources/versionless-late-roc-run2.yaml@@ -0,0 +1,44 @@+# `versionless-late-roc-run1.yaml` as shipped by 65.1: only the runOnChange view's SQL was edited in place, so Liquibase+# re-runs it (RERAN) and moves its changelog row into 65.1's deployment.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_late_roc.yaml+ - changeSet:+ id: late_roc_table+ author: test+ comment: table of 65.0 (scenario test)+ changes:+ - createTable:+ tableName: lroc_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: lroc_table+ - changeSet:+ id: late_roc_column+ author: test+ comment: column of 65.0 that the view reads (scenario test)+ changes:+ - addColumn:+ tableName: lroc_table+ columns:+ - column:+ name: extra+ type: int+ rollback:+ - dropColumn:+ tableName: lroc_table+ columnName: extra+ - changeSet:+ id: late_roc_view+ author: test+ runOnChange: true+ comment: runOnChange view over the 65.0 column (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_lroc AS SELECT extra, 2 AS x FROM lroc_table+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_lrocdiff --git a/test_resources/versionless-roc-run1.yaml b/test_resources/versionless-roc-run1.yamlnew file mode 100644index 00000000000..7f52d6d2e84--- /dev/null+++ b/test_resources/versionless-roc-run1.yaml@@ -0,0 +1,31 @@+# Test changelog for metabase.app-db.liquibase-test/rollback-does-not-reverse-reran-changesets-test.+# Simulates the OLDER release: a base migration plus a runOnChange view. `versionless-roc-run2.yaml` is the+# next release's changelog, where the view SQL was edited in place (same changeset identity, changed checksum)+# and a new changeset added; a shared logicalFilePath keeps changeset identity stable across the two files.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_roc.yaml+ - changeSet:+ id: roc_base+ author: test+ comment: base migration of the older release (scenario test)+ changes:+ - createTable:+ tableName: roc_base_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: roc_base_table+ - changeSet:+ id: roc_view+ author: test+ runOnChange: true+ comment: runOnChange view (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_roc AS SELECT 1 AS x+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_rocdiff --git a/test_resources/versionless-roc-run2.yaml b/test_resources/versionless-roc-run2.yamlnew file mode 100644index 00000000000..d57b4395bf3--- /dev/null+++ b/test_resources/versionless-roc-run2.yaml@@ -0,0 +1,45 @@+# Test changelog for metabase.app-db.liquibase-test/rollback-does-not-reverse-reran-changesets-test.+# `versionless-roc-run1.yaml` as shipped by the NEXT release: the runOnChange view's SQL was edited in place+# (same changeset identity, changed checksum -- Liquibase will re-run it and move its changelog row into this+# release's deployment) and a genuinely new changeset was added.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_roc.yaml+ - changeSet:+ id: roc_base+ author: test+ comment: base migration of the older release (scenario test)+ changes:+ - createTable:+ tableName: roc_base_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: roc_base_table+ - changeSet:+ id: roc_view+ author: test+ runOnChange: true+ comment: runOnChange view, edited in place by the newer release (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_roc AS SELECT 2 AS x+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_roc+ - changeSet:+ id: roc_new+ author: test+ comment: new changeset of the newer release (scenario test)+ changes:+ - createTable:+ tableName: roc_new_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: roc_new_tablediff --git a/test_resources/versionless-roc-run3.yaml b/test_resources/versionless-roc-run3.yamlnew file mode 100644index 00000000000..a3fa166da8c--- /dev/null+++ b/test_resources/versionless-roc-run3.yaml@@ -0,0 +1,30 @@+# Test changelog for metabase.app-db.liquibase-test/rollback-of-reran-only-deployment-test.+# `versionless-roc-run1.yaml` after the developer edits ONLY the runOnChange view (no new changesets), so the+# next migrate up produces a deployment that contains nothing but the re-run (RERAN) row.+databaseChangeLog:+ - logicalFilePath: migrations/2026/versionless_roc.yaml+ - changeSet:+ id: roc_base+ author: test+ comment: base migration of the older release (scenario test)+ changes:+ - createTable:+ tableName: roc_base_table+ columns:+ - column:+ name: id+ type: int+ rollback:+ - dropTable:+ tableName: roc_base_table+ - changeSet:+ id: roc_view+ author: test+ runOnChange: true+ comment: runOnChange view, edited again with no accompanying new changeset (scenario test)+ changes:+ - sql:+ sql: CREATE OR REPLACE VIEW v_roc AS SELECT 3 AS x+ rollback:+ - sql:+ sql: DROP VIEW IF EXISTS v_roc