The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Never encrypt the app DB on startup when there is existing data; add `enable-encryption` command (#81237)
Alexander Polyankin6b9bcaca8e62
diff --git a/.clj-kondo/config/modules/config.edn b/.clj-kondo/config/modules/config.ednindex 7e01dd218f9..d0b400081a5 100644--- a/.clj-kondo/config/modules/config.edn+++ b/.clj-kondo/config/modules/config.edn@@ -356,7 +356,7 @@connection-pooltaskutil}- :model-imports #{:model/Field :model/QueryCache :model/Setting}}+ :model-imports #{:model/Field :model/QueryCache}}appearance{:team "UX West"diff --git a/.clj-kondo/config/test-helpers/config.edn b/.clj-kondo/config/test-helpers/config.ednindex 44d0c9a4f48..95a44ec8957 100644--- a/.clj-kondo/config/test-helpers/config.edn+++ b/.clj-kondo/config/test-helpers/config.edn@@ -58,6 +58,7 @@metabase.app-db.core/finish-db-setup!metabase.app-db.core/migrate!metabase.app-db.core/setup-db!+ metabase.app-db.encryption-test-util/do-with-encrypted-app-db!metabase.app-db.schema-migrations-test.impl/run-migrations-in-range!metabase.app-db.setup/migrate!metabase.app-db.setup/setup-db!diff --git a/docs/databases/encrypting-details-at-rest.md b/docs/databases/encrypting-details-at-rest.mdindex 741bdcebd2c..b3bd31e7d20 100644--- a/docs/databases/encrypting-details-at-rest.md+++ b/docs/databases/encrypting-details-at-rest.md@@ -29,11 +29,24 @@ Metabase stores connection information for the various databases you add in theMB_ENCRYPTION_SECRET_KEY="IYqrSi5QDthvFWe4/WdAxhnra5DZC3RKx3ZSrOJDKsM=" java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar```-Once you set the `MB_ENCRYPTION_SECRET_KEY` value, Metabase will automatically encrypt and store the connection details for each new database that you add. To encrypt existing connections, see the next section.+Once you set the `MB_ENCRYPTION_SECRET_KEY` value, Metabase will automatically encrypt and store the connection details for each new database that you add. If you set the key on a new Metabase, that's all you need to do. To encrypt an existing Metabase, see the next section.-## Encrypting an existing connection+## Encrypting an existing Metabase-If you added databases before setting the `MB_ENCRYPTION_SECRET_KEY` value, you can encrypt the connection details by going to each one of those databases in **Admin** > **Databases** and clicking on the **Save** button. Existing databases with unencrypted details will continue to work normally.+If you're adding an encryption key to a Metabase that already has data, you need to encrypt that data once with the `enable-encryption` command. Metabase won't start while `MB_ENCRYPTION_SECRET_KEY` is set but the application database isn't encrypted with it; it never encrypts existing data on its own.++1. We recommend that you [backup](../installation-and-operation/backing-up-metabase-application-data.md) your data before enabling encryption.+2. Stop running your Metabase app.+3. Run the CLI command `enable-encryption` with the key set as `MB_ENCRYPTION_SECRET_KEY`.+4. Start Metabase with the same `MB_ENCRYPTION_SECRET_KEY`.++### Example command for enabling encryption++```+MB_ENCRYPTION_SECRET_KEY=your-key java --add-opens java.base/java.nio=ALL-UNNAMED -jar metabase.jar enable-encryption+```++If Metabase reports that the database isn't encrypted but you didn't just add the key, don't run `enable-encryption`: someone has changed the application database directly. Restore it from a backup. (A wrong key is reported separately, as a key mismatch.)## Rotating an encryption keydiff --git a/docs/installation-and-operation/commands.md b/docs/installation-and-operation/commands.mdindex ddf2d23a860..f7c9114ec97 100644--- a/docs/installation-and-operation/commands.md+++ b/docs/installation-and-operation/commands.md@@ -43,6 +43,10 @@ Options:- `-k, --keep-existing` - Do not delete target H2 file if it exists.- `-p, --dump-plaintext` - Do not encrypt dumped contents.+## `enable-encryption`++Encrypts data in the metabase database with the key in the MB_ENCRYPTION_SECRET_KEY environment variable. Run this once, with Metabase stopped, after adding the key to an existing instance: Metabase refuses to start while the key is set but the database is not encrypted with it.+## `environment-variables-documentation`Generates a markdown file containing documentation for environment variables relevant to configuring Metabase. The command only includes environment variables registered as defsettings. For a full list of environment variables, see https://www.metabase.com/docs/latest/configuring-metabase/environment-variables.diff --git a/enterprise/backend/src/metabase_enterprise/advanced_config/settings.clj b/enterprise/backend/src/metabase_enterprise/advanced_config/settings.cljindex eb566e22ad8..81fec175a81 100644--- a/enterprise/backend/src/metabase_enterprise/advanced_config/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/advanced_config/settings.clj@@ -18,7 +18,7 @@(str "Allowed email address domain(s) for new Dashboard Subscriptions and Alerts. ""To specify multiple domains, separate each domain with a comma, with no space in between. ""To allow all domains, leave the field empty. This setting doesn’t affect existing subscriptions."))- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:export? true:feature :email-allow-listdiff --git a/enterprise/backend/src/metabase_enterprise/metabot/settings.clj b/enterprise/backend/src/metabase_enterprise/metabot/settings.cljindex c41e0d06273..72aac16f2f1 100644--- a/enterprise/backend/src/metabase_enterprise/metabot/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/metabot/settings.clj@@ -45,7 +45,7 @@:type :string:default "You have reached your AI usage limit for the current period. Please contact your administrator.":visibility :settings-manager- :encryption :no+ :encryption :when-encryption-key-set:export? true:feature :ai-controls)diff --git a/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj b/enterprise/backend/src/metabase_enterprise/remote_sync/settings.cljindex 32fdde1ba20..c5a4e178b5a 100644--- a/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/remote_sync/settings.clj@@ -25,7 +25,7 @@(deferred-tru "The remote branch to sync with, e.g. `main`"):type :string:visibility :admin- :encryption :no+ :encryption :when-encryption-key-set:export? false:can-read-from-env? true)@@ -44,7 +44,7 @@(deferred-tru "The location of your git repository, e.g. https://github.com/acme-inco/metabase.git"):type :string:visibility :admin- :encryption :no+ :encryption :when-encryption-key-set:export? false:can-read-from-env? true)@@ -198,7 +198,7 @@:type :string:visibility :internal:export? false- :encryption :no+ :encryption :when-encryption-key-set:doc false)(defn check-and-update-remote-settings!diff --git a/enterprise/backend/src/metabase_enterprise/security_center/settings.clj b/enterprise/backend/src/metabase_enterprise/security_center/settings.cljindex b0a8964410d..cf473ebcffc 100644--- a/enterprise/backend/src/metabase_enterprise/security_center/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/security_center/settings.clj@@ -9,7 +9,7 @@(deferred-tru "List of email recipients for Security Center notifications."):type :json:default nil- :encryption :no+ :encryption :when-encryption-key-set:feature :admin-security-center:visibility :admin:export? false@@ -32,7 +32,7 @@(deferred-tru "Slack channel for Security Center notifications. Null means Slack notifications are disabled."):type :string:default nil- :encryption :no+ :encryption :when-encryption-key-set:feature :admin-security-center:visibility :admin:export? falsediff --git a/enterprise/backend/src/metabase_enterprise/semantic_search/settings.clj b/enterprise/backend/src/metabase_enterprise/semantic_search/settings.cljindex 073fd6c0a91..69303d9b3b8 100644--- a/enterprise/backend/src/metabase_enterprise/semantic_search/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/semantic_search/settings.clj@@ -13,7 +13,7 @@(defsetting ee-embedding-provider(deferred-tru "The registered embedding provider to use")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "ai-service":type :string@@ -70,7 +70,7 @@(defsetting ee-embedding-service-base-url(deferred-tru "URL of the OpenAI-compatible embedding service (e.g. a LiteLLM proxy).")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default nil:export? falsediff --git a/enterprise/backend/src/metabase_enterprise/sso/settings.clj b/enterprise/backend/src/metabase_enterprise/sso/settings.cljindex 1fd5b1fa3dc..e935f714474 100644--- a/enterprise/backend/src/metabase_enterprise/sso/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/sso/settings.clj@@ -73,7 +73,7 @@ using, this usually looks like `https://your-org-name.example.com` or `https://eopen it in a text editor, then copy and paste the certificate''s contents here."):feature :sso-saml:audit :no-value- :encryption :no+ :encryption :when-encryption-key-set:setter (fn [new-value];; when setting the idp cert validate that it's something we(when new-value@@ -344,14 +344,14 @@ on your IdP, this usually looks something like `http://www.example.com/141xkex60;; TODO - maybe we want to add a csv setting type?(defsetting ldap-sync-user-attributes-blacklist(deferred-tru "Comma-separated list of user attributes to skip syncing for LDAP users.")- :encryption :no+ :encryption :when-encryption-key-set:default "userPassword,dn,distinguishedName":type :csv:audit :getter)(defsetting ldap-group-membership-filter(deferred-tru "Group membership lookup filter. The placeholders '{dn}' and '{uid}' will be replaced by the user''s Distinguished Name and UID, respectively.")- :encryption :no+ :encryption :when-encryption-key-set:default "(member={dn})":audit :getter)diff --git a/enterprise/backend/src/metabase_enterprise/transforms_python/settings.clj b/enterprise/backend/src/metabase_enterprise/transforms_python/settings.cljindex 83150cd2647..8962178758b 100644--- a/enterprise/backend/src/metabase_enterprise/transforms_python/settings.clj+++ b/enterprise/backend/src/metabase_enterprise/transforms_python/settings.clj@@ -12,7 +12,7 @@:feature :transforms-python:doc false:export? false- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(setting/defsetting python-runner-api-token@@ -35,7 +35,7 @@:feature :transforms-python:doc false:export? false- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(setting/defsetting python-storage-s-3-region@@ -80,7 +80,7 @@:sensitive? true:doc false:export? false- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(setting/defsetting python-storage-s-3-secret-key@@ -103,7 +103,7 @@:feature :transforms-python:doc false:export? false- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(setting/defsetting python-storage-s-3-path-style-accessdiff --git a/enterprise/backend/test/metabase_enterprise/sso/integrations/slack_connect_test.clj b/enterprise/backend/test/metabase_enterprise/sso/integrations/slack_connect_test.cljindex 0fe47a42876..b32d69aa8df 100644--- a/enterprise/backend/test/metabase_enterprise/sso/integrations/slack_connect_test.clj+++ b/enterprise/backend/test/metabase_enterprise/sso/integrations/slack_connect_test.clj@@ -3,6 +3,7 @@(:require[clojure.string :as str][clojure.test :refer :all]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.sso.integrations.slack-connect-test :as slack-connect-test][metabase.sso.test-helpers :as sso.test-helpers][metabase.test :as mt]@@ -11,10 +12,12 @@(set! *warn-on-reflection* true)-(use-fixtures :once (fixtures/initialize :test-users))+(use-fixtures :once+ (fixtures/initialize :test-users)+ (encryption-tu/with-encrypted-app-db-fixture slack-connect-test/test-secret))(use-fixtures :each slack-connect-test/do-with-url-prefix-disabled)-(deftest multiple-sso-methods-test+(deftest ^:synchronized multiple-sso-methods-test(testing "with SAML and Slack Connect configured, a GET request to /auth/sso/slack-connect should redirect to Slack"(slack-connect-test/with-test-encryption!(sso.test-helpers/with-slack-default-setup!diff --git a/resources/migrations/058_update_migrations.yaml b/resources/migrations/058_update_migrations.yamlindex 130057a1e01..4cddfc82466 100644--- a/resources/migrations/058_update_migrations.yaml+++ b/resources/migrations/058_update_migrations.yaml@@ -1336,10 +1336,10 @@ databaseChangeLog:author: ranquildcomment: >-Encrypt any plaintext api_key.key bcrypt hashes at rest, so the column can be read strictly once- MB_ENCRYPTION_SECRET_KEY is set. Otherwise an attacker with SQL write access could inject the bcrypt hash of a- key they know (bcrypt is keyless) and authenticate as any user; encrypting the hash means such a plaintext value- is rejected on the strict read. Already-encrypted values are left untouched. On rollback the values are- decrypted back to the plaintext hash, so a downgrade to code that reads the column directly can still verify keys.+ MB_ENCRYPTION_SECRET_KEY is set: authentication must only accept hashes written through the application, and+ the strict read enforces that by rejecting a plaintext value written directly to the app DB.+ Already-encrypted values are left untouched. On rollback the values are decrypted back to the plaintext hash,+ so a downgrade to code that reads the column directly can still verify keys.changes:- customChange:class: metabase.app_db.custom_migrations.EncryptApiKeys@@ -1601,9 +1601,9 @@ databaseChangeLog:author: ranquildcomment: >-Encrypt any plaintext public_uuid values at rest for cards, dashboards, actions, and documents, so the column- can be read strictly once MB_ENCRYPTION_SECRET_KEY is set. Otherwise an attacker with SQL write access could- forge a public link by inserting a chosen public_uuid; encrypting it means such a plaintext value is rejected on- the strict read. Already-encrypted values are left untouched.+ can be read strictly once MB_ENCRYPTION_SECRET_KEY is set: public links must only resolve to values issued by+ the application, and the strict read enforces that by rejecting a plaintext value written directly to the app+ DB. Already-encrypted values are left untouched.changes:- customChange:class: metabase.app_db.custom_migrations.EncryptPublicUuids@@ -1620,6 +1620,32 @@ databaseChangeLog:- customChange:class: metabase.app_db.custom_migrations.EncryptNotificationAndPulseChannelDetails+ - changeSet:+ id: v58.2026-08-28T00:00:00+ author: ranquild+ comment: >-+ Encrypt at rest any plaintext value of the settings this release newly marks+ :encryption :when-encryption-key-set (they were previously stored plaintext), so the strict decrypting read+ added for encrypted settings accepts them once MB_ENCRYPTION_SECRET_KEY is set. Already-encrypted values are+ left untouched. Rollback decrypts them again, so a downgraded version that reads them as :encryption :no+ still sees the values.+ changes:+ - customChange:+ class: metabase.app_db.custom_migrations.EncryptSettingsV58++ - changeSet:+ id: v58.2026-08-28T14:00:00+ author: ranquild+ comment: >-+ Encrypt at rest any plaintext values left in the columns that were encrypted long before any backfill existed:+ metabase_database details/settings, core_user.settings, channel.details, and secret.value.+ Encryption used to be write-time only, and the old startup auto-encrypt and key rotation covered only some of+ these columns, so an upgraded database can hold a mix of plaintext and encrypted rows that the strict reads+ would reject. Already-encrypted values are left untouched; without MB_ENCRYPTION_SECRET_KEY this is a no-op.+ changes:+ - customChange:+ class: metabase.app_db.custom_migrations.EncryptRemainingColumns+# >>>>>>>>>> DO NOT ADD NEW MIGRATIONS BELOW THIS LINE! ADD THEM ABOVE <<<<<<<<<<########################################################################################################################diff --git a/src/metabase/analytics/settings.clj b/src/metabase/analytics/settings.cljindex d14be069769..c68a4a36df1 100644--- a/src/metabase/analytics/settings.clj+++ b/src/metabase/analytics/settings.clj@@ -56,14 +56,14 @@(defsetting metaplow-url(deferred-tru "The URL of the Metaplow collector to send analytics events to.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:audit :never:doc false)(defsetting snowplow-url(deferred-tru "The URL of the Snowplow collector to send analytics events to.")- :encryption :no+ :encryption :when-encryption-key-set:default (if config/is-prod?"https://sp.metabase.com";; See the iglu-schema-registry repo for instructions on how to run Snowplow Micro locally for developmentdiff --git a/src/metabase/app_db/core.clj b/src/metabase/app_db/core.cljindex 1a43d3e396c..a56ebff9e01 100644--- a/src/metabase/app_db/core.clj+++ b/src/metabase/app_db/core.clj@@ -58,7 +58,8 @@clob->str][mdb.encryptiondecrypt-db- encrypt-db]+ encrypt-db+ encryption-check-status][metabase.app-db.formatformat-sql][mdb.setup@@ -115,8 +116,8 @@Skips:- Liquibase migrations.- - [[metabase.app-db.setup/check-encryption]], whose auto-encrypt branch can silently rewrite- every encrypted `setting` row when an encryption key is configured."+ - [[metabase.app-db.setup/check-encryption]], which may write the `encryption-check` sentinel or+ refuse to start when an encryption key is configured."[](when-not (db-is-set-up?)(verify-application-db-connection!)@@ -133,8 +134,12 @@database migrations. If DB is already set up, this function will no-op. Thread-safe.Callers must explicitly decide whether or not to create sample content during migrations with the`create-sample-content?` keyword argument. This should usually be `true` but is `false` for load-from-h2,- serialization imports, and in some tests because the sample content makes tests slow enough to cause timeouts."- [& {:keys [create-sample-content?]}]+ serialization imports, and in some tests because the sample content makes tests slow enough to cause timeouts.++ `manage-encryption-state?` (default `true`) verifies MB_ENCRYPTION_SECRET_KEY against the database before migrations+ run and records the resulting encryption state afterwards; it is `false` for the commands that handle the+ encryption state themselves (`enable-encryption`, [[metabase.cmd.copy/copy!]])."+ [& {:keys [create-sample-content? manage-encryption-state?] :or {manage-encryption-state? true}}]{:pre [(some? create-sample-content?)]}(when-not (db-is-set-up?);; It doesn't really matter too much what we lock on, as long as the lock is per-application-DB e.g. so we can run@@ -146,7 +151,9 @@(let [db-type (db-type)data-source (data-source)auto-migrate? (config/config-bool :mb-db-automigrate)]- (mdb.setup/setup-db! db-type data-source auto-migrate? create-sample-content?))+ (mdb.setup/setup-db! db-type data-source {:auto-migrate? auto-migrate?+ :create-sample-content? create-sample-content?+ :manage-encryption-state? manage-encryption-state?}))(finish-db-setup!)))):done)diff --git a/src/metabase/app_db/custom_migrations.clj b/src/metabase/app_db/custom_migrations.cljindex 386932fa617..4ea426dde09 100644--- a/src/metabase/app_db/custom_migrations.clj+++ b/src/metabase/app_db/custom_migrations.clj@@ -1210,7 +1210,8 @@example-dashboard-id 1example-collection-id 2 ;; trash collection is 1expected-sample-db-id 1- dbs (table-name->rows table-name->raw-rows :metabase_database)+ dbs (map #(update % :details encryption/maybe-encrypt)+ (table-name->rows table-name->raw-rows :metabase_database))_ (t2/query {:insert-into :metabase_database :values dbs})db-ids (set (map :id (t2/query {:select :id :from :metabase_database})))];; If that did not succeed in creating the metabase_database rows we could be reusing a database that@@ -1240,9 +1241,10 @@:perm_type "perms/collection-access":perm_value "read-and-write":collection_id example-collection-id}]}))+ ;; `example-dashboard-id` is encrypted at rest (`:setter :none` defaults to `:when-encryption-key-set`)(t2/query {:insert-into :setting:values [{:key "example-dashboard-id"- :value (str example-dashboard-id)}]})))))))+ :value (encryption/maybe-encrypt (str example-dashboard-id))}]})))))))(comment;; How to create `resources/sample-content.edn` used in `CreateSampleContent`@@ -2247,46 +2249,138 @@(define-migration EncryptAuthIdentityCredentials(when (encryption/default-encryption-enabled?)(run! (fn [{:keys [id credentials]}]- (when (and (string? credentials)- (not (str/blank? credentials))- (not (encryption/possibly-encrypted-string? credentials)))+ (when (not (encryption/decryptable-string? credentials))(t2/query {:update :auth_identity- :set {:credentials (encryption/maybe-encrypt credentials)}+ :set {:credentials (encryption/encrypt credentials)}:where [:= :id id]})))(t2/reducible-query {:select [:id :credentials]- :from [:auth_identity]}))))+ :from [:auth_identity]+ :where [:!= :credentials nil]}))))(define-reversible-migration EncryptApiKeys(when (encryption/default-encryption-enabled?)(run! (fn [{:keys [id] k :key}]- (when (and (string? k)- (not (str/blank? k))- (not (encryption/possibly-encrypted-string? k)))+ (when (not (encryption/decryptable-string? k))(t2/query {:update :api_key- :set {:key (encryption/maybe-encrypt k)}+ :set {:key (encryption/encrypt k)}:where [:= :id id]})))(t2/reducible-query {:select [:id :key]- :from [:api_key]})))+ :from [:api_key]+ :where [:!= :key nil]})))(when (encryption/default-encryption-enabled?)(run! (fn [{:keys [id] k :key}]- (when (and (string? k)- (not (str/blank? k))- (encryption/possibly-encrypted-string? k))+ (when (encryption/decryptable-string? k)(t2/query {:update :api_key- :set {:key (encryption/maybe-decrypt k)}+ :set {:key (encryption/decrypt k)}:where [:= :id id]})))(t2/reducible-query {:select [:id :key]- :from [:api_key]}))))+ :from [:api_key]+ :where [:!= :key nil]}))))++(defn encrypt-settings+ "Encrypt at rest the plaintext value of every setting in `setting-keys`, so the strict decrypting read of an+ `:encryption :when-encryption-key-set` setting accepts it. A value already encrypted with the current key (e.g. by a+ key rotation, which re-encrypts every setting) is left untouched -- decided by decrypting it, never by its shape,+ since plaintext can look like ciphertext (see [[metabase.util.encryption/possibly-encrypted-string?]]). A blank value+ is encrypted too, since a strict read would reject it as plaintext. No-op without an encryption key. Use it as the+ forward body of a migration that marks existing settings as encrypted, paired with [[decrypt-settings]].++ `setting-keys` must be exactly the settings whose `:encryption` went from `:no` to `:when-encryption-key-set` in the+ release the migration ships in: only those were stored plaintext by the previous release."+ [setting-keys]+ (when (encryption/default-encryption-enabled?)+ (run! (fn [{:keys [key value]}]+ (when (not (encryption/decryptable-string? value))+ (t2/query {:update :setting+ :set {:value (encryption/encrypt value)}+ :where [:= :key key]})))+ (t2/reducible-query {:select [:key :value]+ :from [:setting]+ :where [:and [:in :key setting-keys] [:!= :value nil]]}))))++(defn decrypt-settings+ "Reverse of [[encrypt-settings]]: store the plaintext value of every setting in `setting-keys` that is encrypted with+ the current key, so a downgraded version that reads them as `:encryption :no` still sees them. Plaintext values,+ including ones that merely look like ciphertext, are left untouched."+ [setting-keys]+ (when (encryption/default-encryption-enabled?)+ (run! (fn [{:keys [key value]}]+ (when (encryption/decryptable-string? value)+ (t2/query {:update :setting+ :set {:value (encryption/decrypt value)}+ :where [:= :key key]})))+ (t2/reducible-query {:select [:key :value]+ :from [:setting]+ :where [:and [:in :key setting-keys] [:!= :value nil]]}))))++(def ^:private encrypted-settings-v58+ "Every registered setting stored in the setting table that is encrypted at rest as of v58: the ones whose+ `:encryption` went from `:no` to `:when-encryption-key-set` in v58, and the previously-encrypted ones, whose rows+ can still be plaintext from the era when encryption was write-time only. Settings with `:setter :none` are not+ listed, since they have no row to encrypt; rows for settings that no longer exist are left alone.+ `encrypt-settings-test` checks this list against the registry."+ ["admin-email" "ai-service-base-url" "allowed-iframe-hosts"+ "api-key" "application-colors" "application-favicon-url"+ "application-font-files" "application-logo-url" "csp-img-allowed-hosts"+ "custom-formatting" "custom-geojson" "database-replication-connections"+ "ee-embedding-provider" "ee-embedding-service-api-key" "ee-embedding-service-base-url"+ "email-from-address" "email-from-address-override" "email-from-name"+ "email-reply-to" "email-smtp-host" "email-smtp-host-override"+ "email-smtp-password" "email-smtp-password-override" "email-smtp-port"+ "email-smtp-port-override" "email-smtp-security" "email-smtp-security-override"+ "email-smtp-username" "email-smtp-username-override" "embedding-app-origins-interactive"+ "embedding-app-origins-sdk" "embedding-secret-key" "google-auth-auto-create-accounts-domain"+ "google-auth-client-id" "gsheets" "help-link-custom-destination"+ "jwt-attribute-email" "jwt-attribute-firstname" "jwt-attribute-groups"+ "jwt-attribute-lastname" "jwt-attribute-tenant" "jwt-attribute-tenant-attributes"+ "jwt-group-mappings" "jwt-identity-provider-uri" "jwt-shared-secret"+ "landing-page" "landing-page-illustration-custom" "ldap-attribute-email"+ "ldap-attribute-firstname" "ldap-attribute-lastname" "ldap-bind-dn"+ "ldap-group-base" "ldap-group-mappings" "ldap-group-membership-filter"+ "ldap-host" "ldap-password" "ldap-port"+ "ldap-sync-user-attributes-blacklist" "ldap-user-base" "ldap-user-filter"+ "llm-anthropic-api-base-url" "llm-anthropic-api-key" "llm-azure-api-base-url"+ "llm-azure-api-key" "llm-bedrock-access-key-id" "llm-bedrock-secret-access-key"+ "llm-bedrock-session-token" "llm-deepseek-api-base-url" "llm-deepseek-api-key"+ "llm-google-api-base-url" "llm-google-oauth-access-token" "llm-google-service-account-key"+ "llm-mistral-api-base-url" "llm-mistral-api-key" "llm-moonshot-api-base-url"+ "llm-moonshot-api-key" "llm-openai-api-base-url" "llm-openai-api-key"+ "llm-openrouter-api-base-url" "llm-openrouter-api-key" "llm-providers"+ "llm-proxy-base-url" "llm-vllm-api-base-url" "llm-vllm-api-key"+ "llm-zai-api-base-url" "llm-zai-api-key" "locked-meters"+ "login-page-illustration-custom" "map-tile-server-url" "mcp-apps-cors-custom-origins"+ "mcp-apps-cors-enabled-clients" "metabot-chat-system-prompt" "metabot-nlq-system-prompt"+ "metabot-quota-reached-message" "metabot-slack-signing-secret" "metabot-sql-system-prompt"+ "metaplow-url" "mfa-challenge-signing-key" "migration-dump-file"+ "no-data-illustration-custom" "no-object-illustration-custom" "notification-link-base-url"+ "oidc-providers" "premium-embedding-token" "python-runner-api-token"+ "python-runner-url" "python-storage-s-3-access-key" "python-storage-s-3-container-endpoint"+ "python-storage-s-3-endpoint" "python-storage-s-3-secret-key" "remote-sync-allow"+ "remote-sync-branch" "remote-sync-token" "remote-sync-url"+ "saml-application-name" "saml-attribute-email" "saml-attribute-firstname"+ "saml-attribute-group" "saml-attribute-lastname" "saml-attribute-tenant"+ "saml-group-mappings" "saml-identity-provider-certificate" "saml-identity-provider-issuer"+ "saml-identity-provider-slo-uri" "saml-identity-provider-uri" "saml-keystore-alias"+ "saml-keystore-password" "saml-keystore-path" "sdk-encryption-validation-key"+ "search-language" "security-center-email-recipients" "security-center-slack-channel"+ "session-timeout" "site-url" "slack-app-token"+ "slack-bug-report-channel" "slack-cached-channels-and-usernames" "slack-connect-attribute-team-id"+ "slack-connect-authentication-mode" "slack-connect-client-id" "slack-connect-client-secret"+ "slack-files-channel" "snowplow-url" "source-address-header"+ "store-api-url" "store-url" "subscription-allowed-domains"+ "uploads-settings"])++(define-reversible-migration EncryptSettingsV58+ (encrypt-settings encrypted-settings-v58)+ (decrypt-settings encrypted-settings-v58))(define-reversible-migration EncryptPublicUuids(when (encryption/default-encryption-enabled?)(doseq [table [:report_card :report_dashboard :action :document]](run! (fn [{:keys [id public_uuid]}]- (when (and (string? public_uuid)- (not (str/blank? public_uuid))- (not (encryption/possibly-encrypted-string? public_uuid)))+ (when (not (encryption/decryptable-string? public_uuid))(t2/query {:update table- :set {:public_uuid (encryption/maybe-encrypt public_uuid)}+ :set {:public_uuid (encryption/encrypt public_uuid)}:where [:= :id id]})))(t2/reducible-query {:select [:id :public_uuid]:from [table]@@ -2294,11 +2388,9 @@(when (encryption/default-encryption-enabled?)(doseq [table [:report_card :report_dashboard :action :document]](run! (fn [{:keys [id public_uuid]}]- (when (and (string? public_uuid)- (not (str/blank? public_uuid))- (encryption/possibly-encrypted-string? public_uuid))+ (when (encryption/decryptable-string? public_uuid)(t2/query {:update table- :set {:public_uuid (encryption/maybe-decrypt public_uuid)}+ :set {:public_uuid (encryption/decrypt public_uuid)}:where [:= :id id]})))(t2/reducible-query {:select [:id :public_uuid]:from [table]@@ -2308,10 +2400,9 @@(when (encryption/default-encryption-enabled?)(doseq [table [:notification_recipient :pulse_channel]](run! (fn [{:keys [id details]}]- (when (and (string? details)- (not (encryption/possibly-encrypted-string? details)))+ (when (not (encryption/decryptable-string? details))(t2/query {:update table- :set {:details (encryption/maybe-encrypt details)}+ :set {:details (encryption/encrypt details)}:where [:= :id id]})))(t2/reducible-query {:select [:id :details]:from [table]@@ -2319,11 +2410,49 @@(when (encryption/default-encryption-enabled?)(doseq [table [:notification_recipient :pulse_channel]](run! (fn [{:keys [id details]}]- (when (and (string? details)- (encryption/possibly-encrypted-string? details))+ (when (encryption/decryptable-string? details)(t2/query {:update table- :set {:details (encryption/maybe-decrypt details)}+ :set {:details (encryption/decrypt details)}:where [:= :id id]})))(t2/reducible-query {:select [:id :details]:from [table]:where [:!= :details nil]})))))++(def ^:private encrypt-remaining-columns-v58+ "Encrypted-at-rest string columns that predate any backfill: pre-v53 encryption was write-time only, and the+ v53-v62 startup auto-encrypt and pre-v63 key rotation covered only `metabase_database.details`, `setting`, and+ `secret`, so an upgraded database can hold a mix of plaintext and encrypted rows within one of these columns.+ Columns newer than v58 (`metabase_database.write_data_details` and later) are not listed: they do not exist yet+ when this changeset runs on an older database."+ [[:metabase_database :details]+ [:metabase_database :settings]+ [:core_user :settings]+ [:channel :details]])++(defn- secret-value->bytes ^bytes [v]+ (cond+ (bytes? v) v+ (instance? java.sql.Blob v) (.getBytes ^java.sql.Blob v 0 (.length ^java.sql.Blob v))+ :else nil))++(define-migration EncryptRemainingColumns+ (when (encryption/default-encryption-enabled?)+ (doseq [[table column] encrypt-remaining-columns-v58]+ (run! (fn [{:keys [id value]}]+ (when (and (string? value)+ (not (encryption/possibly-encrypted-string? value)))+ (t2/query {:update table+ :set {column (encryption/maybe-encrypt value)}+ :where [:= :id id]})))+ (t2/reducible-query {:select [:id [column :value]]+ :from [table]+ :where [:!= column nil]})))+ (run! (fn [{:keys [id value]}]+ (let [value (secret-value->bytes value)]+ (when (and value (not (encryption/possibly-encrypted-bytes? value)))+ (t2/query {:update :secret+ :set {:value (encryption/maybe-encrypt-bytes value)}+ :where [:= :id id]}))))+ (t2/reducible-query {:select [:id :value]+ :from [:secret]+ :where [:!= :value nil]}))))diff --git a/src/metabase/app_db/encryption.clj b/src/metabase/app_db/encryption.cljindex e7227388c3f..0dfb15a159f 100644--- a/src/metabase/app_db/encryption.clj+++ b/src/metabase/app_db/encryption.clj@@ -1,9 +1,11 @@(ns metabase.app-db.encryption(:require+ [metabase.util :as u][metabase.util.encryption :as encryption][metabase.util.honey-sql-2 :as h2x][metabase.util.i18n :refer [trs]][metabase.util.log :as log]+ [metabase.util.malli :as mu][metabase.util.string :as string][toucan2.core :as t2])(:import@@ -60,27 +62,187 @@(def ^:private clearable-when-undecryptable#{[:core_user :settings]})-(defn- encryption-check-status+(def ^:private encryption-check-key "encryption-check")++(mu/defn encryption-check-status :- [:enum :valid :invalid :absent]"Whether the current MB_ENCRYPTION_SECRET_KEY is the right key for this database, according to the- `encryption-check` sentinel setting (a random UUID stored encrypted whenever the database is encrypted):+ `encryption-check` sentinel setting -- a random UUID encrypted under the key, present if and only if the database is encrypted:- :valid - the sentinel decrypts to a UUID, so the key is correct- :invalid - the sentinel exists but does not decrypt, so the key is wrong (or unset) for this database- :unknown - no sentinel (database predates it), or the database is marked unencrypted"+ :valid - a key is set and the sentinel decrypts to a UUID with it, so the key is correct+ :invalid - the sentinel exists but does not decrypt (wrong or unset key, corruption)+ :absent - no sentinel: a `setting` table that does not exist yet (before migrations on a fresh database), no+ row, or the plaintext \"unencrypted\" marker (inserted by the v53 migration, and written back when+ the database is decrypted) -- an explicit statement of the same thing a missing row means"[]- (let [raw (t2/select-one-fn :value :setting :key "encryption-check")]+ (let [raw (u/ignore-exceptions (t2/select-one-fn :value :setting :key encryption-check-key))](cond(or (nil? raw) (= raw "unencrypted"))- :unknown+ :absent- (and (encryption/possibly-encrypted-string? raw)- (try (string/valid-uuid? (encryption/maybe-decrypt-accepting-plaintext raw))- (catch Throwable _ false)))+ (and (encryption/default-encryption-enabled?)+ (u/ignore-exceptions (string/valid-uuid? (encryption/maybe-decrypt raw)))):valid:else:invalid)))+(defn- column-exists?+ "Whether `table`.`column` exists in the connection's own catalog and schema, per JDBC metadata (a same-named table+ in another schema of the database must not count). Identifiers are matched as stored and upper-cased, since H2+ upper-cases unquoted names."+ [table column]+ (t2/with-connection [^java.sql.Connection conn]+ (let [metadata (.getMetaData conn)+ catalog (.getCatalog conn)+ schema (.getSchema conn)]+ (boolean (some (fn [[t c]]+ (with-open [rs (.getColumns metadata catalog schema t c)]+ (.next rs)))+ [[(name table) (name column)]+ [(u/upper-case-en (name table)) (u/upper-case-en (name column))]])))))++(mu/defn- column-content-status :- [:enum :none :decryptable :not-decryptable]+ "Whether `table`.`column` holds no values, only values `decryptable?` accepts, or at least one value it does not.+ Streams the column and stops at the first value that does not decrypt. A table or column that does not exist yet+ (this runs before migrations, on a fresh or old database) counts as empty; any other failure to read it propagates,+ since answering \"empty\" wrongly could mark an unencrypted database as encrypted."+ [table :- :keyword+ column :- :keyword+ decryptable? :- fn?]+ (try+ (reduce (fn [acc {:keys [value]}]+ (cond+ (nil? value) acc+ (decryptable? value) :decryptable+ :else (reduced :not-decryptable)))+ :none+ (t2/reducible-select [table [column :value]]))+ (catch Exception e+ (if (column-exists? table column)+ (throw e)+ :none))))++(mu/defn- encrypted-content-status :- [:enum :none :decryptable :not-decryptable]+ "Whether the encrypted-at-rest columns hold content, and whether all of it was encrypted under the current key:++ :none - no encrypted-at-rest column holds any value: a database that has never held such content+ :decryptable - every value in every encrypted-at-rest column decrypts with the current key, which only a+ database encrypted under exactly this key can produce+ :not-decryptable - some value does not decrypt: plaintext waiting for `enable-encryption`, ciphertext under some+ other key, or a mix++ This only runs in the one-shot \"no sentinel but content exists\" state, so it can afford to stream every column+ fully (stopping at the first value that does not decrypt) rather than sampling; a partially encrypted column can+ therefore never read as `:decryptable`. The `setting` table is not counted: whether a given setting is encrypted at+ rest is decided per setting, so its values prove nothing about the database."+ []+ (reduce (fn [acc [table column decryptable?]]+ (case (column-content-status table column decryptable?)+ :not-decryptable (reduced :not-decryptable)+ :decryptable :decryptable+ :none acc))+ :none+ (concat (map #(conj % encryption/decryptable-string?) encrypted-string-columns)+ (map #(conj % (comp encryption/decryptable-bytes? maybe-blob->bytes)) encrypted-bytes-columns))))++(defn- replace-encryption-check!+ "Replace the `encryption-check` sentinel on `conn`: with a fresh UUID encrypted by `encrypt-fn`, or with the+ plaintext \"unencrypted\" marker when `encrypt-fn` is nil (the database is being decrypted)."+ [conn encrypt-fn]+ (t2/delete! :conn conn :setting :key encryption-check-key)+ (t2/insert! :conn conn :setting {:key encryption-check-key+ :value (if encrypt-fn+ (encrypt-fn (str (random-uuid)))+ "unencrypted")}))++(defn- write-encryption-check!+ "Record that the database is encrypted under the current MB_ENCRYPTION_SECRET_KEY by replacing the `encryption-check`+ sentinel with a fresh UUID encrypted under it. Only ever writes the sentinel -- never touches any other row."+ []+ (t2/with-transaction [conn]+ (replace-encryption-check! conn encryption/encrypt)))++(def ^:private EncryptionState+ [:enum :encrypted :unencrypted :fresh :pre-sentinel :missing-key :wrong-key :not-decryptable])++(mu/defn encryption-state :- EncryptionState+ "The encryption state of the database, judged from MB_ENCRYPTION_SECRET_KEY, the `encryption-check` sentinel+ setting (a random UUID encrypted under the key, present if and only if the database is encrypted, read and written+ raw rather than through `defsetting` -- see [[encryption-check-status]]), and the encrypted-at-rest content itself.+ Never throws:++ :encrypted - the key is set and the sentinel decrypts with it+ :unencrypted - no key and no sentinel+ :fresh - the key is set, no sentinel, and the database has never held encrypted-at-rest content+ :pre-sentinel - the key is set, no sentinel, and every encrypted-at-rest value already decrypts with the key --+ a state only a database encrypted under exactly this key can produce, e.g. one from before the+ sentinel existed+ :missing-key - the sentinel is present but no key is set+ :wrong-key - the key is set but the sentinel does not decrypt with it+ :not-decryptable - the key is set, no sentinel, and some content does not decrypt: plaintext waiting for+ `enable-encryption`, ciphertext under some other key, or a mix"+ []+ (let [status (encryption-check-status)]+ (if-not (encryption/default-encryption-enabled?)+ (if (= status :absent) :unencrypted :missing-key)+ (case status+ :valid :encrypted+ :invalid :wrong-key+ :absent (case (encrypted-content-status)+ :none :fresh+ :decryptable :pre-sentinel+ :not-decryptable :not-decryptable)))))++(mu/defn check-encryption :- :nil+ "Refuse to run with a `db-state` MB_ENCRYPTION_SECRET_KEY cannot work with. This runs before migrations, which+ encrypt whatever they write or backfill with the current key: none of these states may reach them -- in particular+ a wrong key would re-encrypt existing ciphertext, irreversibly.++ Startup never encrypts existing data: content the key does not decrypt (`:not-decryptable`) refuses to run, and the+ admin has to run `enable-encryption` deliberately (or fix the key). The strict reads rely on existing rows only+ ever being encrypted by that deliberate command."+ [db-state :- EncryptionState]+ (case db-state+ :encrypted+ (log/info "Database encrypted and MB_ENCRYPTION_SECRET_KEY correctly configured")++ :unencrypted+ (log/info "Database not encrypted and MB_ENCRYPTION_SECRET_KEY env variable not set.")++ (:fresh :pre-sentinel)+ nil++ :missing-key+ (throw (ex-info "Database is encrypted but the MB_ENCRYPTION_SECRET_KEY environment variable was NOT set" {}))++ :wrong-key+ (throw (ex-info (str "Database was encrypted with a different key than the MB_ENCRYPTION_SECRET_KEY "+ "environment contains")+ {}))++ :not-decryptable+ (throw (ex-info (str "MB_ENCRYPTION_SECRET_KEY is set but the database is not marked as encrypted and already "+ "contains data the key does not decrypt. If you have just added the key to an existing "+ "instance, stop Metabase and run `enable-encryption` to encrypt the database. If this "+ "database was already encrypted, it has been modified directly: do NOT run "+ "`enable-encryption`; check the key or restore from a backup.")+ {}))))++(mu/defn record-encryption-state!+ "Record post-migrations the state [[encryption-state]] found pre-migrations: for a `:fresh` or `:pre-sentinel`+ database (both provably encrypted under the current key, or holding nothing at all), replace the `encryption-check`+ sentinel with a fresh UUID encrypted under MB_ENCRYPTION_SECRET_KEY; every other state is already recorded+ correctly. Runs after migrations because on a fresh database the `setting` table does not exist before them. Only+ ever writes the sentinel, never another row."+ [db-state :- EncryptionState]+ (when (#{:fresh :pre-sentinel} db-state)+ (write-encryption-check!)+ (log/info (case db-state+ :fresh "MB_ENCRYPTION_SECRET_KEY set on a new database. Marked database as encrypted."+ :pre-sentinel (str "MB_ENCRYPTION_SECRET_KEY decrypts the existing data but the database "+ "predates the encryption sentinel. Marked database as encrypted."))+ (u/emoji "✅"))))+(defn- reencrypt-encrypted-column!"Re-encrypt `column` for every row in `table` using `encrypt-str-fn`. See `encrypted-string-columns`. Streams therows so a large column does not have to be held in memory all at once.@@ -138,15 +300,20 @@(reencrypt-encrypted-column! conn table column encrypt-str-fn(and (= check-status :valid)(contains? clearable-when-undecryptable [table column])))))- (doseq [[key value] (t2/select-fn->fn :key :value :model/Setting)]+ ;; Read the settings raw (via `:setting`, not `:model/Setting`) to bypass the model's strict decrypt-on-read:+ ;; a setting that is plaintext at rest while a key is configured (e.g. one newly designated encrypted but not yet+ ;; re-encrypted) is exactly what this operation exists to fix, so we decrypt it leniently here rather than reject+ ;; it. A value that looks encrypted but can't be decrypted with the current key still aborts.+ (doseq [[key value] (t2/select-fn->fn :key :value :setting)](case key"settings-last-updated" (let [current-timestamp-as-string-honeysql (h2x/cast (if (= db-type :mysql) :char :text)(h2x/current-datetime-honeysql-form db-type))](t2/update! :conn conn :setting {:key key} {:value current-timestamp-as-string-honeysql}))- "encryption-check" (t2/update! :conn conn :setting {:key key} {:value (if encrypting? (encrypt-str-fn (str (random-uuid))) "unencrypted")})+ "encryption-check" nil(t2/update! :conn conn :setting{:key key}- {:value (encrypt-str-fn value)})))+ {:value (encrypt-str-fn (encryption/maybe-decrypt-accepting-plaintext value))})))+ (replace-encryption-check! conn (when encrypting? encrypt-str-fn))(doseq [[table column] encrypted-bytes-columns](reencrypt-encrypted-bytes-column! conn table column encrypt-bytes-fn))(t2/delete! :conn conn :model/QueryCache))))@@ -157,6 +324,8 @@[db-type data-source to-key](when (and (not (nil? to-key)) (empty? to-key))(throw (ex-info "Cannot encrypt database with an empty key" {})))+ (when (and (nil? to-key) (not (encryption/default-encryption-enabled?)))+ (throw (ex-info "Cannot encrypt database: MB_ENCRYPTION_SECRET_KEY is not set" {})))(do-encryption db-type data-source true (fn [maybe-encrypt-fn](if(nil? to-key) maybe-encrypt-fndiff --git a/src/metabase/app_db/setup.clj b/src/metabase/app_db/setup.cljindex e5ba6591641..9879703b37c 100644--- a/src/metabase/app_db/setup.clj+++ b/src/metabase/app_db/setup.clj@@ -19,15 +19,12 @@[metabase.app-db.liquibase :as liquibase][metabase.config.core :as config][metabase.util :as u]- [metabase.util.encryption :as encryption][metabase.util.honey-sql-2][metabase.util.i18n :refer [trs]][metabase.util.log :as log][metabase.util.malli :as mu][metabase.util.malli.schema :as ms]- [metabase.util.string :as string][methodical.core :as methodical]- [toucan2.core :as t2][toucan2.honeysql2 :as t2.honeysql][toucan2.jdbc.options :as t2.jdbc.options][toucan2.pipeline :as t2.pipeline])@@ -192,32 +189,6 @@"https://www.metabase.com/docs/latest/installation-and-operation/migrating-from-h2#supported-databases-for-storing-your-metabase-application-data"]){}))))))-(mu/defn- check-encryption- "Ensure encryption env variable is correctly set if needed, and encrypt the database if it needs to be- Encryption status is tracked by an 'encryption-check' value in the settings table.- NOTE: the encryption-check setting is not managed like most settings with 'defsetting' so we can manage checking the raw values in the database"- []- (let [raw (try (t2/select-one-fn :value :setting :key "encryption-check")- (catch Throwable e (log/warnf "Error checking encryption status, assuming unencrypted: %s" (ex-message e))))- looks-encrypted (not= raw "unencrypted")]- (log/debug "Checking encryption configuration")- (when-not (nil? raw)- (if looks-encrypted- (do- (when-not (encryption/default-encryption-enabled?)- (throw (ex-info "Database is encrypted but the MB_ENCRYPTION_SECRET_KEY environment variable was NOT set" {})))- (when-not (and (encryption/possibly-encrypted-string? raw)- (u/ignore-exceptions- (string/valid-uuid? (encryption/maybe-decrypt-accepting-plaintext raw))))- (throw (ex-info "Database was encrypted with a different key than the MB_ENCRYPTION_SECRET_KEY environment contains" {})))- (log/debug "Database encrypted and MB_ENCRYPTION_SECRET_KEY correctly configured"))- (if (encryption/default-encryption-enabled?)- (do- (log/info "New MB_ENCRYPTION_SECRET_KEY environment variable set. Encrypting database...")- (mdb.encryption/encrypt-db (:db-type mdb.connection/*application-db*) (:data-source mdb.connection/*application-db*) nil)- (log/info "Database encrypted..." (u/emoji "✅")))- (log/debug "Database not encrypted and MB_ENCRYPTION_SECRET_KEY env variable not set."))))))-(mu/defn- error-if-downgrade-required)@@ -255,21 +226,41 @@;; TODO -- consider renaming to something like `verify-connection-and-migrate!`(mu/defn setup-db!"Connects to db and runs migrations. Don't use this directly, unless you know what you're doing;- use [[metabase.app-db.setup-db!]] instead, which can be called more than once without issue and is thread-safe."- [db-type :- :keyword- data-source :- (ms/InstanceOfClass javax.sql.DataSource)- auto-migrate? :- :boolean- create-sample-content? :- :boolean]- (u/profile (trs "Database setup")- (u/with-us-locale- (binding [mdb.connection/*application-db* (mdb.connection/application-db db-type data-source :create-pool? false) ; should already be a pool- config/*disable-setting-cache* true- custom-migrations/*create-sample-content* create-sample-content?]- (verify-db-connection db-type data-source)- (error-if-downgrade-required! data-source)- (run-schema-migrations! data-source auto-migrate?)- (check-encryption))))- :done)+ use [[metabase.app-db.setup-db!]] instead, which can be called more than once without issue and is thread-safe.++ Options:+ - `:auto-migrate?` (default `true`): run pending migrations, otherwise only print them.+ - `:create-sample-content?` (default `false`): create the sample content on a fresh install.+ - `:manage-encryption-state?` (default `true`): verify MB_ENCRYPTION_SECRET_KEY against the database before+ migrations run (see [[mdb.encryption/encryption-state]] and [[mdb.encryption/check-encryption]]) and record the+ state afterwards (see [[mdb.encryption/record-encryption-state!]]). Turned off by the `enable-encryption` command+ and by [[metabase.cmd.copy/copy!]],+ which handle the encryption state themselves."+ ([db-type data-source]+ (setup-db! db-type data-source {}))++ ([db-type :- :keyword+ data-source :- (ms/InstanceOfClass javax.sql.DataSource)+ {:keys [auto-migrate? create-sample-content? manage-encryption-state?]+ :or {auto-migrate? true, create-sample-content? false, manage-encryption-state? true}}+ :- [:map+ [:auto-migrate? {:optional true} :boolean]+ [:create-sample-content? {:optional true} :boolean]+ [:manage-encryption-state? {:optional true} :boolean]]]+ (u/profile (trs "Database setup")+ (u/with-us-locale+ (binding [mdb.connection/*application-db* (mdb.connection/application-db db-type data-source :create-pool? false) ; should already be a pool+ config/*disable-setting-cache* true+ custom-migrations/*create-sample-content* create-sample-content?]+ (verify-db-connection db-type data-source)+ (error-if-downgrade-required! data-source)+ (let [db-state (mdb.encryption/encryption-state)]+ (when manage-encryption-state?+ (mdb.encryption/check-encryption db-state))+ (run-schema-migrations! data-source auto-migrate?)+ (when manage-encryption-state?+ (mdb.encryption/record-encryption-state! db-state))))))+ :done))(defn release-migration-locks!"Wait up to `timeout-seconds` for the current process to release all migration locks, otherwise force release them."diff --git a/src/metabase/appearance/settings.clj b/src/metabase/appearance/settings.cljindex f1c807f5cc6..89cc963b734 100644--- a/src/metabase/appearance/settings.clj+++ b/src/metabase/appearance/settings.clj@@ -58,7 +58,7 @@(defsetting landing-page(deferred-tru "Enter a relative URL like /dashboard/1 or /collection/2.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -98,7 +98,7 @@(defsetting application-colors(deferred-tru "Choose the colors used in the user interface throughout Metabase and others specifically for the charts. You need to refresh your browser to see your changes take effect.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :json@@ -147,7 +147,7 @@ To change the chart colors:(defsetting application-font-files(deferred-tru "Tell us where to find the file for each font weight. You don’t need to include all of them, but it’ll look better if you do.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :json@@ -184,7 +184,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting application-logo-url(deferred-tru "Upload a file to replace the Metabase logo on the top bar.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -195,7 +195,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting application-favicon-url(deferred-tru "Upload a file to use as the favicon.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -225,7 +225,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting login-page-illustration-custom(deferred-tru "The custom illustration for the login page.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -244,7 +244,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting landing-page-illustration-custom(deferred-tru "The custom illustration for the landing page.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -263,7 +263,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting no-data-illustration-custom(deferred-tru "The custom illustration for when there are no results after running a question.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -282,7 +282,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting no-object-illustration-custom(deferred-tru "The custom illustration for when there are no results after searching.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:export? true:type :string@@ -324,7 +324,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting help-link-custom-destination(deferred-tru "Custom URL for the help link.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :public:type :string:audit :getter@@ -362,7 +362,7 @@ See [fonts](../configuring-metabase/fonts.md).")(defsetting custom-formatting(deferred-tru "Object keyed by type, containing formatting settings")- :encryption :no+ :encryption :when-encryption-key-set:type :json:export? true:default {}diff --git a/src/metabase/channel/settings.clj b/src/metabase/channel/settings.cljindex 8d0358b8c34..b598dddc404 100644--- a/src/metabase/channel/settings.clj+++ b/src/metabase/channel/settings.clj@@ -78,7 +78,7 @@(deferred-tru "The name of the channel to which Metabase files should be initially uploaded"):deprecated "0.54.0":default "metabase_files"- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:audit :getter:setter (fn [channel-name]@@ -87,7 +87,7 @@(defsetting slack-bug-report-channel(deferred-tru "The name of the channel where bug reports should be posted"):default "metabase-bugs"- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:audit :getter:export? false@@ -117,7 +117,7 @@(defsetting notification-link-base-url(deferred-tru "By default \"Site Url\" is used in notification links, but can be overridden.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :internal:type :string:feature :whitelabel@@ -127,14 +127,14 @@(defsetting email-from-address(deferred-tru "The email address you want to use for the sender of emails.")- :encryption :no+ :encryption :when-encryption-key-set:default "[email protected]":visibility :settings-manager:audit :getter)(defsetting email-from-address-override(deferred-tru "The email address you want to use for the sender of emails from your custom SMTP server.")- :encryption :no+ :encryption :when-encryption-key-set:feature :cloud-custom-smtp:default "[email protected]":visibility :settings-manager@@ -143,7 +143,7 @@(defsetting email-from-name(deferred-tru "The name you want to use for the sender of emails.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:audit :getter:setter (fn [new-value]@@ -166,7 +166,7 @@(defsetting email-reply-to(deferred-tru "The email address you want the replies to go to, if different from the from address.")- :encryption :no+ :encryption :when-encryption-key-set:type :json:visibility :settings-manager:audit :getterdiff --git a/src/metabase/cloud_migration/settings.clj b/src/metabase/cloud_migration/settings.cljindex e92708af212..852082a9d6b 100644--- a/src/metabase/cloud_migration/settings.clj+++ b/src/metabase/cloud_migration/settings.clj@@ -35,7 +35,7 @@(defsetting store-url(deferred-tru "Store URL."):type :string- :encryption :no+ :encryption :when-encryption-key-set;; should be :internal, but FE doesn't get internal settings. -- Someone else;;;; OK, then no it shouldn't be internal at all, internal is literally for Settings that are only visible to backend@@ -47,7 +47,7 @@(defsetting migration-dump-file(deferred-tru "Dump file for migrations.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :internal:default nil:doc falsediff --git a/src/metabase/cmd/copy.clj b/src/metabase/cmd/copy.cljindex 888127ff15e..8ac16046e42 100644--- a/src/metabase/cmd/copy.clj+++ b/src/metabase/cmd/copy.clj@@ -14,6 +14,7 @@[metabase.models.init][metabase.models.resolution :as models.resolution][metabase.util :as u]+ [metabase.util.encryption :as encryption][metabase.util.i18n :refer [trs]][metabase.util.log :as log][metabase.util.malli :as mu]@@ -219,7 +220,7 @@(cond-> database(or (:is_attached_dwh database)(and (not *copy-h2-database-details*)- (= (:engine database) "h2"))) (assoc :details "{}"))))+ (= (:engine database) "h2"))) (assoc :details (encryption/maybe-encrypt "{}"))))):model/Setting;; Never create dumps with read-only-mode turned on.@@ -456,14 +457,18 @@(not config/ee-available?)(remove #(str/starts-with? (str %) "metabase-enterprise")))](classloader/require ns-symb))- ;; make sure the source database is up-do-date+ ;; make sure the source database is up-do-date. Skip the encryption check: the source may legitimately be unencrypted+ ;; while MB_ENCRYPTION_SECRET_KEY is set for the target (enabling encryption while migrating off H2); rows are copied+ ;; as-is and [[metabase.cmd.load-from-h2/load-from-h2!]] encrypts the target afterwards.(step (trs "Set up {0} source database and run migrations..." (name source-db-type))- (mdb.setup/setup-db! source-db-type source-data-source true false))+ (mdb.setup/setup-db! source-db-type source-data-source {:manage-encryption-state? false}));; make sure the dest DB is up-to-date;;- ;; don't need or want to run data migrations in the target DB, since the data is already migrated appropriately+ ;; don't need or want to run data migrations in the target DB, since the data is already migrated appropriately.+ ;; Skip the encryption check too: whatever it would write is truncated below along with the other migration-created+ ;; rows, and the caller decides the target's encryption state from the copied sentinel afterwards.(step (trs "Set up {0} target database and run migrations..." (name target-db-type))- (mdb.setup/setup-db! target-db-type target-data-source true false))+ (mdb.setup/setup-db! target-db-type target-data-source {:manage-encryption-state? false}));; make sure target DB is empty(step (trs "Testing if target {0} database is already populated..." (name target-db-type))(assert-has-no-users target-data-source))diff --git a/src/metabase/cmd/core.clj b/src/metabase/cmd/core.cljindex 6fe08f88ca0..23da694b716 100644--- a/src/metabase/cmd/core.clj+++ b/src/metabase/cmd/core.clj@@ -220,6 +220,23 @@(log/errorf "ERROR ROTATING KEY: %s" (ex-message e))(system-exit! 1))))+(defn ^:command enable-encryption+ "Encrypts data in the metabase database with the key in the MB_ENCRYPTION_SECRET_KEY environment variable. Run this+ once, with Metabase stopped, after adding the key to an existing instance: Metabase refuses to start while the key is+ set but the database is not encrypted with it."+ []+ (classloader/require 'metabase.cmd.enable-encryption)+ (when-not (encryption/default-encryption-enabled?)+ (log/error "MB_ENCRYPTION_SECRET_KEY environment variable has not been set")+ (system-exit! 1))+ (try+ ((resolve 'metabase.cmd.enable-encryption/enable-encryption!))+ (log/info "Encryption enabled OK.")+ (system-exit! 0)+ (catch Throwable e+ (log/errorf "ERROR ENABLING ENCRYPTION: %s" (ex-message e))+ (system-exit! 1))))+(defn ^:command remove-encryption"Decrypts data in the metabase database. The MB_ENCRYPTION_SECRET_KEY environment variable has to be set tothe current key"diff --git a/src/metabase/cmd/enable_encryption.clj b/src/metabase/cmd/enable_encryption.cljnew file mode 100644index 00000000000..b3568a670d1--- /dev/null+++ b/src/metabase/cmd/enable_encryption.clj@@ -0,0 +1,20 @@+(ns metabase.cmd.enable-encryption+ (:require+ [metabase.app-db.core :as mdb]+ [metabase.util.log :as log]))++(set! *warn-on-reflection* true)++(defn enable-encryption!+ "Encrypts the current configured db with the key in the `MB_ENCRYPTION_SECRET_KEY` env var. This is the only path+ that turns existing plaintext rows into ciphertext: startup refuses to run while the key is set but the database is+ not encrypted with it, so that existing rows are only ever encrypted when an admin explicitly asks for it."+ []+ (when-not (mdb/db-is-set-up?)+ (log/info "Checking database configuration prior to encryption")+ (mdb/setup-db! :create-sample-content? true :manage-encryption-state? false))+ (log/infof "Connected to: %s | %s" (mdb/db-type) (mdb/db-file))+ (if (= :valid (mdb/encryption-check-status))+ (log/info "Database is already encrypted with MB_ENCRYPTION_SECRET_KEY; nothing to do.")+ ;; an :invalid sentinel makes encrypt-db abort before touching any row+ (mdb/encrypt-db (mdb/db-type) (mdb/data-source) nil)))diff --git a/src/metabase/cmd/load_from_h2.clj b/src/metabase/cmd/load_from_h2.cljindex 36032d8a7cd..c678395adf8 100644--- a/src/metabase/cmd/load_from_h2.clj+++ b/src/metabase/cmd/load_from_h2.clj@@ -21,7 +21,9 @@[metabase.app-db.core :as mdb][metabase.cmd.copy :as copy][metabase.cmd.copy.h2 :as copy.h2]- [metabase.search.core :as search]))+ [metabase.search.core :as search]+ [metabase.util.encryption :as encryption]+ [metabase.util.log :as log]))(defn load-from-h2!"Transfer data from existing H2 database to a newly created (presumably MySQL or Postgres) DB. Intended as a tool for@@ -34,4 +36,16 @@(let [h2-filename (str h2-filename ";IFEXISTS=TRUE")h2-data-source (copy.h2/h2-data-source h2-filename)](copy/copy! :h2 h2-data-source (mdb/db-type) (mdb/data-source))+ ;; The rows were copied verbatim, sentinel included, so the target is in whatever state the H2 file was in.+ (let [status (mdb/encryption-check-status)]+ (cond+ (= status :invalid)+ (throw (ex-info (str "The H2 database is encrypted with a different key than the MB_ENCRYPTION_SECRET_KEY "+ "environment contains (or the key is not set)")+ {}))++ (and (= status :absent) (encryption/default-encryption-enabled?))+ (do+ (log/info "MB_ENCRYPTION_SECRET_KEY is set and the loaded data is not encrypted. Encrypting...")+ (mdb/encrypt-db (mdb/db-type) (mdb/data-source) nil))))(search/reset-tracking!))))diff --git a/src/metabase/embedding/settings.clj b/src/metabase/embedding/settings.cljindex 53b2e97dffc..1ce80f4e408 100644--- a/src/metabase/embedding/settings.clj+++ b/src/metabase/embedding/settings.clj@@ -128,7 +128,7 @@:feature :embedding:export? false:visibility :public- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(defsetting enable-embedding-static@@ -183,7 +183,7 @@:export? false:visibility :public:default ""- :encryption :no+ :encryption :when-encryption-key-set:audit :getter:getter #'-embedding-app-origins-sdk:setter #'-embedding-app-origins-sdk!)diff --git a/src/metabase/geojson/settings.clj b/src/metabase/geojson/settings.cljindex 24ac7a01461..45c81d5ca81 100644--- a/src/metabase/geojson/settings.clj+++ b/src/metabase/geojson/settings.clj@@ -133,7 +133,7 @@(defsetting custom-geojson(deferred-tru "JSON containing information about custom GeoJSON files for use in map visualizations instead of the default US State or World GeoJSON.")- :encryption :no+ :encryption :when-encryption-key-set:type :json:getter (fn [] (merge (setting/get-value-of-type :json :custom-geojson) (builtin-geojson))):setter (fn [new-value]diff --git a/src/metabase/llm/settings.clj b/src/metabase/llm/settings.cljindex bc152fba21c..4e116975838 100644--- a/src/metabase/llm/settings.clj+++ b/src/metabase/llm/settings.clj@@ -87,7 +87,7 @@(defsetting llm-anthropic-api-base-url(deferred-tru "The Anthropic API base URL.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://api.anthropic.com":export? false@@ -116,7 +116,7 @@(defsetting llm-openai-api-base-url(deferred-tru "The OpenAI API base URL.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://api.openai.com":export? false@@ -139,7 +139,7 @@(defsetting llm-openrouter-api-base-url(deferred-tru "The OpenRouter API base URL used for Chat Completions.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://openrouter.ai/api":export? false@@ -162,7 +162,7 @@(defsetting llm-zai-api-base-url(deferred-tru "The Z.AI API base URL used for Chat Completions.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://api.z.ai/api/paas/v4":export? false@@ -185,7 +185,7 @@(defsetting llm-mistral-api-base-url(deferred-tru "The Mistral API base URL used for Chat Completions.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://api.mistral.ai/v1":export? false@@ -206,7 +206,7 @@(defsetting llm-moonshot-api-base-url(deferred-tru "The Moonshot AI API base URL used for Chat Completions. Repoint this to use the `.cn` platform; keys are not interchangeable between the two.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://api.moonshot.ai/v1":export? false@@ -227,7 +227,7 @@(defsetting llm-deepseek-api-base-url(deferred-tru "The DeepSeek API base URL. Both the Anthropic-compatible Messages surface (`/anthropic/v1/messages`) and the model catalog (`/models`) are served off this root, so do not include `/anthropic` or `/v1`.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default "https://api.deepseek.com":export? false@@ -322,7 +322,7 @@(defsetting llm-google-api-base-url(deferred-tru "The Gemini Enterprise Agent Platform API base URL. Leave unset to derive it from the location.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:default google-global-api-base-url:export? false@@ -383,7 +383,7 @@(defsetting llm-azure-api-base-url(deferred-tru "The base URL of the Azure resource''s OpenAI- or Anthropic-compatible surface, e.g. `https://<resource>.services.ai.azure.com/openai`.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:export? false:getter (connection-field-getter :llm-azure-api-base-url)@@ -412,7 +412,7 @@(defsetting llm-vllm-api-base-url(deferred-tru "The base URL of your vLLM server''s OpenAI-compatible API, e.g. `http://vllm.internal:8000/v1`.")- :encryption :no+ :encryption :when-encryption-key-set:visibility :settings-manager:export? false:getter (connection-field-getter :llm-vllm-api-base-url)@@ -464,7 +464,7 @@ Configuring a provider through the single-provider variables (`MB_LLM_ANTHROPIC_:enabled? #(or (premium-features/has-feature? :metabase-ai-managed)(premium-features/has-feature? :offer-metabase-ai-managed)(premium-features/has-feature? :metabot-v3))- :encryption :no+ :encryption :when-encryption-key-set:visibility :internal:default nil:export? false@@ -474,7 +474,7 @@ Configuring a provider through the single-provider variables (`MB_LLM_ANTHROPIC_(deferred-tru "Base URL for the managed Metabase AI service."):enabled? #(or (premium-features/has-feature? :metabase-ai-managed)(premium-features/has-feature? :metabot-v3))- :encryption :no+ :encryption :when-encryption-key-set:visibility :internal:default nil:export? falsediff --git a/src/metabase/mcp/settings.clj b/src/metabase/mcp/settings.cljindex 4b42c6a3709..d0bda554492 100644--- a/src/metabase/mcp/settings.clj+++ b/src/metabase/mcp/settings.clj@@ -62,7 +62,7 @@:default []:visibility :admin:export? false- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(defn- strip-scheme@@ -105,7 +105,7 @@:default "":visibility :admin:export? false- :encryption :no+ :encryption :when-encryption-key-set:audit :getter:setter #'-mcp-apps-cors-custom-origins!)diff --git a/src/metabase/metabot/settings.clj b/src/metabase/metabot/settings.cljindex 7f2b710aaf5..c0e8cc7e825 100644--- a/src/metabase/metabot/settings.clj+++ b/src/metabase/metabot/settings.clj@@ -61,7 +61,7 @@:type :string:default "":visibility :admin- :encryption :no+ :encryption :when-encryption-key-set:export? true:feature :ai-controls)@@ -70,7 +70,7 @@:type :string:default "":visibility :admin- :encryption :no+ :encryption :when-encryption-key-set:export? true:feature :ai-controls)@@ -79,7 +79,7 @@:type :string:default "":visibility :admin- :encryption :no+ :encryption :when-encryption-key-set:export? true:feature :ai-controls)diff --git a/src/metabase/premium_features/settings.clj b/src/metabase/premium_features/settings.cljindex 332c9715bd3..4e66208e3f9 100644--- a/src/metabase/premium_features/settings.clj+++ b/src/metabase/premium_features/settings.clj@@ -41,7 +41,7 @@(defsetting locked-meters(deferred-tru "Locally-mirrored is-locked state per meter, refreshed on each successful token-check.")- :encryption :no+ :encryption :when-encryption-key-set:type :json:visibility :internal:audit :neverdiff --git a/src/metabase/request/settings.clj b/src/metabase/request/settings.cljindex a02ec862a9d..9c46204d224 100644--- a/src/metabase/request/settings.clj+++ b/src/metabase/request/settings.clj@@ -7,7 +7,7 @@(defsetting source-address-header(deferred-tru "Identify the source of HTTP requests by this header''s value, instead of its remote address.")- :encryption :no+ :encryption :when-encryption-key-set:default "X-Forwarded-For":export? true:audit :getter@@ -104,7 +104,7 @@;; Should be in the form "{\"amount\":60,\"unit\":\"minutes\"}" where the unit is one of "seconds", "minutes";; or "hours". The amount is nillable.(deferred-tru "Time before inactive users are logged out. By default, sessions last indefinitely.")- :encryption :no+ :encryption :when-encryption-key-set:type :json:default nil:getter #'-session-timeoutdiff --git a/src/metabase/search/settings.clj b/src/metabase/search/settings.cljindex ac856c29e79..6b10e78e9e4 100644--- a/src/metabase/search/settings.clj+++ b/src/metabase/search/settings.clj@@ -67,6 +67,6 @@Value must be a valid configured language option in your database such as ''english'' or ''simple''"):visibility :internal:export? false- :encryption :no+ :encryption :when-encryption-key-set:default nil:type :string)diff --git a/src/metabase/server/settings.clj b/src/metabase/server/settings.cljindex 98ec24bae1b..c4f1ace3b3d 100644--- a/src/metabase/server/settings.clj+++ b/src/metabase/server/settings.clj@@ -35,7 +35,7 @@ x.com")(defsetting allowed-iframe-hosts(deferred-tru "Allowed iframe hosts")- :encryption :no+ :encryption :when-encryption-key-set:default default-allowed-iframe-hosts:audit :getter:visibility :public@@ -43,7 +43,7 @@ x.com")(defsetting csp-img-allowed-hosts(deferred-tru "Comma-separated list of hosts that images may load from (e.g. in dashboard text, entity descriptions, and custom visualizations) when `csp-img-enabled` is on. Empty by default, which restricts images to this Metabase instance and the map tile server used by map visualizations.")- :encryption :no+ :encryption :when-encryption-key-set:default "":audit :getter:visibility :publicdiff --git a/src/metabase/settings/models/setting.clj b/src/metabase/settings/models/setting.cljindex a3396d73626..7c786e9ada2 100644--- a/src/metabase/settings/models/setting.clj+++ b/src/metabase/settings/models/setting.clj@@ -339,8 +339,17 @@(when (allows-database-local-values? setting)(core/get *database-local-values* setting-name))))-(defn- prohibits-encryption? [setting-or-name]- (= :no (:encryption (resolve-setting setting-or-name))))+(defn- maybe-resolve-setting+ "Like [[resolve-setting]] but returns nil for a setting with no code definition (e.g. one written straight to the DB+ in a test) instead of throwing."+ [setting-or-name]+ (try (resolve-setting setting-or-name)+ (catch clojure.lang.ExceptionInfo e+ (when-not (::unknown-setting-error (ex-data e))+ (throw e)))))++(defn- encrypts? [setting-or-name]+ (not= :no (:encryption (resolve-setting setting-or-name))))(defn- allows-user-local-values? [setting](#{:only :allowed} (:user-local (resolve-setting setting))))@@ -1734,7 +1743,7 @@;; we can't do anything about it (since we can't decrypt it). If stuff isn't decrypted in the DB, we have nothing to;; do.(when (encryption/default-encryption-enabled?)- (let [settings (filter prohibits-encryption? (vals @registered-settings))]+ (let [settings (remove encrypts? (vals @registered-settings))](t2/with-transaction [_conn](doseq [{v :value k :key}(t2/select :setting {:for :update :where [:and@@ -1751,13 +1760,10 @@;; Don't do any automatic handling of the "encryption-check" special setting used by mdb.encryption(if (= "encryption-check" (:key setting-model))setting-model- (let [resolved (try (resolve-setting (:key setting-model))- (catch clojure.lang.ExceptionInfo e- (when (not (::unknown-setting-error (ex-data e)))- (throw e))))]+ (let [resolved (maybe-resolve-setting (:key setting-model))](cond-> setting-model(or (nil? resolved)- (not (prohibits-encryption? resolved)))+ (encrypts? resolved))(update :value encryption/maybe-encrypt)))))(t2/define-before-update :model/Setting@@ -1769,18 +1775,23 @@(maybe-encrypt setting))(defn- decrypt-setting-value-on-read- "Decrypt a Setting `value` on read, tolerating an undecryptable value by returning it unchanged so a stale row (e.g.- left by a botched key change) degrades gracefully at the reader instead of throwing on every read. The strict throw- from [[encryption/maybe-decrypt-accepting-plaintext]] still applies where it matters — the key-rotation path, which- does not swallow it."- [value]- (try- (encryption/maybe-decrypt-accepting-plaintext value)- (catch Throwable _ value)))+ "Decrypt a Setting's `:value` on read. A setting whose `:encryption` is not `:no` is stored encrypted at rest, so it+ is read strictly with [[encryption/maybe-decrypt]]: a plaintext value — forged via a direct DB write, or a legacy row+ from before the setting became encrypted — is rejected rather than trusted. A `:no` setting (or one with no code+ definition, e.g. in tests) is intentionally plaintext, so it is read leniently with+ [[encryption/maybe-decrypt-accepting-plaintext]], which returns a plaintext value unchanged."+ [setting]+ (let [resolved (maybe-resolve-setting (:key setting))+ decrypt (if (or (nil? resolved) (not (encrypts? resolved)))+ encryption/maybe-decrypt-accepting-plaintext+ encryption/maybe-decrypt)]+ (update setting :value decrypt)))(t2/define-after-select :model/Setting[setting]- ;; Don't do any automatic handling of the "encryption-check" special setting used by mdb.encryption- (if (= "encryption-check" (:key setting))+ ;; Skip aggregate results (e.g. a `count` row) that carry no `:key` to resolve or `:value` to decrypt, and don't do+ ;; any automatic handling of the "encryption-check" special setting used by mdb.encryption.+ (if (or (nil? (:key setting))+ (= "encryption-check" (:key setting)))setting- (update setting :value decrypt-setting-value-on-read)))+ (decrypt-setting-value-on-read setting)))diff --git a/src/metabase/sso/settings.clj b/src/metabase/sso/settings.cljindex 53cc824f680..f70da58c965 100644--- a/src/metabase/sso/settings.clj+++ b/src/metabase/sso/settings.clj@@ -48,19 +48,19 @@(defsetting ldap-user-base(deferred-tru "Search base for users. (Will be searched recursively)")- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(defsetting ldap-user-filter(deferred-tru "User lookup filter. The placeholder '''{login}''' will be replaced by the user supplied login."):default "(&(objectClass=inetOrgPerson)(|(uid={login})(mail={login})))"- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(defsetting ldap-attribute-email(deferred-tru "Attribute to use for the user''s email. (usually ''mail'', ''email'' or ''userPrincipalName'')"):default "mail"- :encryption :no+ :encryption :when-encryption-key-set:getter (fn [] (u/lower-case-en (setting/get-value-of-type :string :ldap-attribute-email))):audit :getter)@@ -68,12 +68,12 @@(deferred-tru "Attribute to use for the user''s first name. (usually ''givenName'')"):default "givenName":getter (fn [] (u/lower-case-en (setting/get-value-of-type :string :ldap-attribute-firstname)))- :encryption :no+ :encryption :when-encryption-key-set:audit :getter)(defsetting ldap-attribute-lastname(deferred-tru "Attribute to use for the user''s last name. (usually ''sn'')")- :encryption :no+ :encryption :when-encryption-key-set:default "sn":getter (fn [] (u/lower-case-en (setting/get-value-of-type :string :ldap-attribute-lastname))):audit :getter)@@ -87,13 +87,13 @@(defsetting ldap-group-base(deferred-tru "Search base for groups. Not required for LDAP directories that provide a ''memberOf'' overlay, such as Active Directory. (Will be searched recursively)"):audit :getter- :encryption :no)+ :encryption :when-encryption-key-set)(defsetting ldap-group-mappings;; Should be in the form: {"cn=Some Group,dc=...": [1, 2, 3]} where keys are LDAP group DNs and values are lists of;; MB groups IDs(deferred-tru "JSON containing LDAP to Metabase group mappings.")- :encryption :no+ :encryption :when-encryption-key-set:type :json:cache? false:default {}@@ -181,7 +181,7 @@:export? false:default slack-connect-auth-mode-link-only:audit :getter- :encryption :no+ :encryption :when-encryption-key-set:setter (fn [new-value](when (and new-value(not (contains? #{slack-connect-auth-mode-sso slack-connect-auth-mode-link-only} new-value)))diff --git a/src/metabase/store_api/settings.clj b/src/metabase/store_api/settings.cljindex f919bdcd9f0..ae535a1a807 100644--- a/src/metabase/store_api/settings.clj+++ b/src/metabase/store_api/settings.clj@@ -16,7 +16,7 @@(defsetting store-api-url(deferred-tru "Store API URL."):type :string- :encryption :no+ :encryption :when-encryption-key-set:visibility :internal:default (str "https://store-api" (when (default-to-staging?) ".staging") ".metabase.com"):doc falsediff --git a/src/metabase/tiles/settings.clj b/src/metabase/tiles/settings.cljindex e6c5a4c3f3e..350184c4951 100644--- a/src/metabase/tiles/settings.clj+++ b/src/metabase/tiles/settings.clj@@ -5,7 +5,7 @@(defsetting map-tile-server-url(i18n/deferred-tru "The map tile server URL template used in map visualizations, for example from OpenStreetMaps or MapBox.")- :encryption :no+ :encryption :when-encryption-key-set:default "https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png":visibility :public:audit :getter)diff --git a/src/metabase/util/encryption.clj b/src/metabase/util/encryption.cljindex 0b411cfd8ff..42eaab52bac 100644--- a/src/metabase/util/encryption.clj+++ b/src/metabase/util/encryption.clj@@ -203,28 +203,58 @@(def ^:private ^:const aes256-block-size 16)(defn possibly-encrypted-bytes?- "Returns true if it's likely that `b` is an encrypted byte array. To compute this, we need the number of bytes in- the input, subtract the bytes used by the cipher type tag (`aes256-tag-length`) and what is left should be divisible- by the cipher's block size (`aes256-block-size`). If it's not divisible by that number it is either not encrypted or- it has been corrupted as it must always have a multiple of the block size or it won't decrypt."+ "Whether `b` has the shape of an encrypted byte array: at least the length of the shortest ciphertext, and a length+ that leaves a multiple of the cipher block size (`aes256-block-size`) once the tag (`aes256-tag-length`) is taken off.++ This is a shape check, and its only guarantee is one-sided:++ - `false`: `b` is definitely NOT encrypted (or is corrupted beyond decrypting).+ - `true`: `b` is encrypted, OR it is plaintext that merely happens to have the right length.++ So it can rule encryption out, never in. Anything that decides whether a value gets encrypted, decrypted, skipped+ or trusted must use [[decryptable-bytes?]], which actually decrypts."[^bytes b]- (if (nil? b)- false- (u/ignore-exceptions- (let [byte-length (alength b)]- (zero? (mod (- byte-length aes256-tag-length)- aes256-block-size))))))+ (boolean+ (when b+ (u/ignore-exceptions+ (let [byte-length (alength b)]+ ;; IV + at least one cipher block + tag: anything shorter cannot be ciphertext, and `mod` alone would accept+ ;; 16- and 32-byte plaintext too+ (and (>= byte-length (+ (* 2 aes256-block-size) aes256-tag-length))+ (zero? (mod (- byte-length aes256-tag-length)+ aes256-block-size))))))))(defn possibly-encrypted-string?- "Returns true if it's likely that `s` is an encrypted string. Specifically we need `s` to be a non-blank, base64- encoded string of the correct length. See docstring for `possibly-encrypted-bytes?` for an explanation of correct- length."+ "Whether `s` has the shape of an encrypted string: non-blank base64 that decodes to something+ [[possibly-encrypted-bytes?]]. The same one-sided guarantee applies: `false` means definitely not encrypted, `true`+ means encrypted or plaintext that merely matches the pattern (a 64-character hex digest, for one). Use+ [[decryptable-string?]] wherever the answer decides what happens to the value."[^String s]- (u/ignore-exceptions- (when-let [b (and (not (str/blank? s))- (u/base64-string? s)- (codec/base64-decode s))]- (possibly-encrypted-bytes? b))))+ (boolean+ (u/ignore-exceptions+ (when-let [b (and (not (str/blank? s))+ (u/base64-string? s)+ (codec/base64-decode s))]+ (possibly-encrypted-bytes? b)))))++(defn decryptable-bytes?+ "Whether `b` is encrypted with `secret-key`: it has the shape of ciphertext *and* decrypts. The ciphertext is+ authenticated, so a wrong key, corruption, and plaintext that merely looks like ciphertext all yield false. Always+ false when no key is set. Use this, not [[possibly-encrypted-bytes?]], wherever the answer decides whether a value is+ encrypted, decrypted, or left alone."+ ([^bytes b] (decryptable-bytes? default-secret-key b))+ ([secret-key ^bytes b]+ (boolean (and secret-key+ (possibly-encrypted-bytes? b)+ (u/ignore-exceptions (decrypt-bytes secret-key b) true)))))++(defn decryptable-string?+ "Whether `s` is encrypted with `secret-key`: it has the shape of ciphertext *and* decrypts. See [[decryptable-bytes?]]."+ ([^String s] (decryptable-string? default-secret-key s))+ ([secret-key ^String s]+ (boolean (and secret-key+ (possibly-encrypted-string? s)+ (u/ignore-exceptions (decrypt secret-key s) true)))))(defn maybe-decrypt-accepting-plaintext"Plaintext-tolerant decrypt of a String `s`. If `MB_ENCRYPTION_SECRET_KEY` is set and `s` is encrypted, decrypt it;diff --git a/test/metabase/app_db/custom_migrations_test.clj b/test/metabase/app_db/custom_migrations_test.cljindex baf3bef1e32..1ab94d055d5 100644--- a/test/metabase/app_db/custom_migrations_test.clj+++ b/test/metabase/app_db/custom_migrations_test.clj@@ -31,7 +31,7 @@[metabase.pulse.models.pulse-channel-test :as pulse-channel-test][metabase.pulse.task.send-pulses :as task.send-pulses][metabase.search.ingestion :as search.ingestion]- [metabase.settings.models.setting]+ [metabase.settings.models.setting :as setting][metabase.sync.task.sync-databases-test :as task.sync-databases-test][metabase.task.core :as task][metabase.task.impl :as task.impl]@@ -1525,7 +1525,7 @@(defmacro ^:private with-ldap-and-sso-configured!"Run body with ldap and SSO configured, in which SSO will only be configured if enterprise is available"[ldap-group-mappings sso-group-mappings & body]- (binding [metabase.settings.models.setting/*allow-retired-setting-names* true]+ (binding [setting/*allow-retired-setting-names* true]`(call-with-ldap-and-sso-configured! ~ldap-group-mappings ~sso-group-mappings (fn [] ~@body))));; The `remove-admin-from-group-mapping-if-needed` migration is written to run in OSS version@@ -2719,7 +2719,7 @@(is (not (encryption/possibly-encrypted-string? (raw-cred plain-id))))(migrate!)(testing "plaintext credentials are encrypted and decrypt to the original value"- (is (encryption/possibly-encrypted-string? (raw-cred plain-id)))+ (is (encryption/decryptable-string? (raw-cred plain-id)))(is (= {:password_hash "h" :password_salt "s"}(json/decode+kw (encryption/maybe-decrypt-accepting-plaintext (raw-cred plain-id))))))(testing "already-encrypted credentials row is left unchanged"@@ -2744,7 +2744,7 @@(is (not (encryption/possibly-encrypted-string? (raw-key plain-id))))(migrate!)(testing "plaintext key is encrypted and decrypts to the original hash"- (is (encryption/possibly-encrypted-string? (raw-key plain-id)))+ (is (encryption/decryptable-string? (raw-key plain-id)))(is (= "plaintext-bcrypt-hash"(encryption/maybe-decrypt-accepting-plaintext (raw-key plain-id)))))(testing "already-encrypted key row is left unchanged"@@ -2772,9 +2772,95 @@(encryption-test/with-secret-key "dont-tell-anyone-about-this"(migrate!)(testing "encrypted after migration, and still decrypts to the original value"- (is (true? (encryption/possibly-encrypted-string? (raw-pc))))+ (is (encryption/decryptable-string? (raw-pc)))(is (= pc-details (encryption/maybe-decrypt (raw-pc))))))))))+(deftest encrypt-remaining-columns-test+ (testing "v58.2026-08-28T14:00:00 : plaintext rows in the historically mixable columns are encrypted, encrypted rows untouched"+ (encryption-test/with-secret-key "encrypt-remaining-test-key-1234"+ (impl/test-migrations ["v58.2026-08-28T14:00:00"] [migrate!]+ (let [plain-details (json/encode {:db "/plain.db"})+ enc-details (encryption/maybe-encrypt (json/encode {:db "/enc.db"}))+ plain-db-id (:id (new-instance-with-default :metabase_database {:details plain-details}))+ enc-db-id (:id (new-instance-with-default :metabase_database {:details enc-details}))+ user-settings (json/encode {:locale "en"})+ user-id (:id (new-instance-with-default :core_user {:settings user-settings}))+ secret-bytes (.getBytes "sooper-secret" "UTF-8")+ secret-id (t2/insert-returning-pk! :secret {:name "s"+ :kind "password"+ :value secret-bytes+ :version 1+ :creator_id user-id+ :created_at :%now+ :updated_at :%now})+ raw (fn [table column id]+ (:value (t2/query-one {:select [[column :value]] :from [table] :where [:= :id id]})))+ ;; convert the blob inside the reduction, while its connection is still open+ raw-secret (fn [id]+ (first (into []+ (map (fn [{:keys [value]}] (#'custom-migrations/secret-value->bytes value)))+ (t2/reducible-query {:select [:value] :from [:secret] :where [:= :id id]}))))]+ (is (not (encryption/possibly-encrypted-string? (raw :metabase_database :details plain-db-id))))+ (migrate!)+ (testing "plaintext string values are encrypted and decrypt to the original"+ (is (encryption/possibly-encrypted-string? (raw :metabase_database :details plain-db-id)))+ (is (= plain-details (encryption/maybe-decrypt (raw :metabase_database :details plain-db-id))))+ (is (encryption/possibly-encrypted-string? (raw :core_user :settings user-id)))+ (is (= user-settings (encryption/maybe-decrypt (raw :core_user :settings user-id)))))+ (testing "already-encrypted values are untouched"+ (is (= enc-details (raw :metabase_database :details enc-db-id))))+ (testing "plaintext secret bytes are encrypted and decrypt to the original"+ (let [v (raw-secret secret-id)]+ (is (encryption/possibly-encrypted-bytes? v))+ (is (= "sooper-secret" (String. (encryption/maybe-decrypt-bytes v) "UTF-8"))))))))))++(deftest encrypt-settings-test+ ;; some of the settings are enterprise-only, so they are registered only when the EE namespaces are loaded+ (testing "a few known encrypted settings are in the migration list"+ (doseq [k ["email-smtp-password" "ldap-password" "saml-keystore-password" "site-url" "snowplow-url"]]+ (testing k+ (is (some #{k} @#'custom-migrations/encrypted-settings-v58)))))+ (testing "v58.2026-08-28T00:00:00 : plaintext values of newly-encrypted settings are encrypted at rest, others untouched"+ (encryption-test/with-secret-key "encrypt-settings-test-key-1234"+ (impl/test-migrations "v58.2026-08-28T00:00:00" [migrate!]+ (let [ins! (fn [k v] (t2/query {:insert-into :setting :values [{:key k :value v}]}))+ raw (fn [k] (t2/select-one-fn :value :setting :key k))+ enc-str (encryption/maybe-encrypt "https://already.example")+ ;; base64 of 64 bytes: plaintext with exactly the shape of ciphertext+ shaped-str (str (apply str (repeat 86 "a")) "==")]+ (ins! "snowplow-url" "https://plain.example")+ (ins! "ldap-user-filter" " ")+ (ins! "email-reply-to" "")+ (ins! "map-tile-server-url" enc-str)+ (ins! "store-url" shaped-str)+ (ins! "site-name" "Metabase")+ (is (not (encryption/possibly-encrypted-string? (raw "snowplow-url"))))+ (is (encryption/possibly-encrypted-string? shaped-str))+ (migrate!)+ (testing "a plaintext value of a listed setting is encrypted and decrypts back"+ (is (encryption/decryptable-string? (raw "snowplow-url")))+ (is (= "https://plain.example" (encryption/maybe-decrypt (raw "snowplow-url")))))+ (testing "a blank value of a listed setting is encrypted too, since a strict read would reject plaintext"+ (is (encryption/decryptable-string? (raw "ldap-user-filter")))+ (is (= " " (encryption/maybe-decrypt (raw "ldap-user-filter"))))+ (is (encryption/decryptable-string? (raw "email-reply-to")))+ (is (= "" (encryption/maybe-decrypt (raw "email-reply-to")))))+ (testing "a plaintext value that merely looks like ciphertext is encrypted, not skipped"+ (is (encryption/decryptable-string? (raw "store-url")))+ (is (= shaped-str (encryption/maybe-decrypt (raw "store-url")))))+ (testing "an already-encrypted value is left unchanged"+ (is (= enc-str (raw "map-tile-server-url"))))+ (testing "a setting not in the list is left plaintext"+ (is (= "Metabase" (raw "site-name"))))+ (testing "rollback decrypts the listed settings back to plaintext, others untouched"+ (migrate! :down 57)+ (is (= "https://plain.example" (raw "snowplow-url")))+ (is (= " " (raw "ldap-user-filter")))+ (is (= "" (raw "email-reply-to")))+ (is (= shaped-str (raw "store-url")))+ (is (= "https://already.example" (raw "map-tile-server-url")))+ (is (= "Metabase" (raw "site-name")))))))))+(deftest backfill-transform-target-db-id-test(testing "v59.2026-01-31T12:01:23 : backfill target_db_id from target and source JSON"(impl/test-migrations ["v59.2026-01-31T12:01:23"] [migrate!]diff --git a/test/metabase/app_db/encryption_test_util.clj b/test/metabase/app_db/encryption_test_util.cljnew file mode 100644index 00000000000..7bedae7ae1e--- /dev/null+++ b/test/metabase/app_db/encryption_test_util.clj@@ -0,0 +1,74 @@+(ns metabase.app-db.encryption-test-util+ (:require+ [metabase.app-db.connection :as mdb.connection]+ [metabase.app-db.core :as mdb]+ [metabase.app-db.custom-migrations.util :as custom-migrations.util]+ [metabase.app-db.data-source :as mdb.data-source]+ [metabase.app-db.encryption :as mdb.encryption]+ [metabase.test.initialize :as initialize]+ [metabase.test.initialize.test-users :as init.test-users]+ [metabase.util.encryption :as encryption]))++(set! *warn-on-reflection* true)++(defonce ^{:doc "Holds `{:app-db .. :secret-key ..}` for the isolated, encrypted app DB prepared by the namespace whose+ `:once` [[with-encrypted-app-db-fixture]] is running, so [[with-encrypted-app-db]] can reach it. A global (rather than+ a dynamic binding) because the runner runs a namespace's vars, and the server/`future` threads serving `mt/client`,+ on threads that would not see a thread-local binding. Safe as a single global: the runner runs whole namespaces+ sequentially, and the tests that opt in are `^:synchronized`, so at most one prepared DB is live at a time."}+ prepared-encrypted-db+ (atom nil))++(defn do-with-encrypted-app-db!+ "Build one isolated, empty H2 application DB encrypted with `secret-key` (the hashed value bound to+ [[metabase.util.encryption/default-secret-key]]) and hold it in [[prepared-encrypted-db]] for the extent of `thunk`.++ Running `encrypt-db` after `setup-db!` mirrors production startup, where setting `MB_ENCRYPTION_SECRET_KEY`+ re-encrypts any setting written before the key was active. Intended as a namespace's outermost `:once` fixture: the+ ambient app DB is left untouched, so tests run as usual against the shared DB unless they opt into the isolated DB+ with [[with-encrypted-app-db]]. That opt-in is for the tests that activate an encryption key — reading a setting+ under an active key strictly decrypts it, which throws on a plaintext value another namespace may have left in the+ shared DB. Standard test users are seeded so `:rasta` &c. resolve there (`user->id` memoizes per application DB)."+ [secret-key thunk]+ ;; initialize the ambient test app DB first: seeding the isolated DB below forces the one-time :db initialization+ ;; while *application-db* is bound to it, which would mark :db initialized without ever migrating the ambient DB --+ ;; the first test to touch the ambient DB would then find it empty.+ (initialize/initialize-if-needed! :db)+ (let [data-source (mdb.data-source/raw-connection-string->DataSource+ (str "jdbc:h2:mem:" (gensym "encryption-test-db-")))]+ ;; hold one connection open for the whole fixture so the in-memory DB survives until every test has run+ (with-open [_conn (.getConnection data-source)]+ (let [app-db (mdb.connection/application-db :h2 data-source)]+ (binding [mdb.connection/*application-db* app-db+ custom-migrations.util/*allow-temp-scheduling* false]+ (mdb/setup-db! :create-sample-content? false)+ (init.test-users/init!)+ (with-redefs [encryption/default-secret-key secret-key]+ (mdb.encryption/encrypt-db (mdb.connection/db-type) data-source nil)))+ (reset! prepared-encrypted-db {:app-db app-db :secret-key secret-key})+ (try+ (thunk)+ (finally+ (reset! prepared-encrypted-db nil)))))))++(defn with-encrypted-app-db-fixture+ "Return a `:once` [[clojure.test/use-fixtures]] function that prepares an encrypted app DB for the namespace via+ [[do-with-encrypted-app-db!]]. Tests opt in with [[with-encrypted-app-db]]."+ [secret-key]+ (fn [thunk]+ (do-with-encrypted-app-db! secret-key thunk)))++(defmacro with-encrypted-app-db+ "Run `body` against the namespace's prepared encrypted app DB (see [[with-encrypted-app-db-fixture]]) with the+ encryption key active. `*application-db*` is redefined (not bound) so the isolated DB is visible on the server/`future`+ threads that serve `mt/client`; the opt-in test must therefore be marked `^:synchronized`. Use for a test that+ activates an encryption key, so its strict decrypting reads meet only the settings in this already-encrypted DB."+ [& body]+ `(let [prepared# (or @prepared-encrypted-db+ (throw (ex-info (str "with-encrypted-app-db requires a `with-encrypted-app-db-fixture` :once "+ "fixture on this namespace")+ {})))+ {app-db# :app-db secret-key# :secret-key} prepared#]+ (with-redefs [mdb.connection/*application-db* app-db#+ encryption/default-secret-key secret-key#]+ ~@body)))diff --git a/test/metabase/app_db/query_test.clj b/test/metabase/app_db/query_test.cljindex 3130c8f446c..e5195552079 100644--- a/test/metabase/app_db/query_test.clj+++ b/test/metabase/app_db/query_test.clj@@ -9,6 +9,7 @@[metabase.query-processor.compile :as qp.compile][metabase.query-processor.test :as qp][metabase.test :as mt]+ [metabase.test.fixtures :as fixtures][metabase.util :as u][toucan2.core :as t2])(:import@@ -16,6 +17,8 @@(set! *warn-on-reflection* true)+(use-fixtures :once (fixtures/initialize :db))+(defn- verify-same-query"Ensure that the formatted native query derived from an mbql query produce the same results."[q]diff --git a/test/metabase/app_db/setup_test.clj b/test/metabase/app_db/setup_test.cljindex ef6a748f8b3..18351c80ba0 100644--- a/test/metabase/app_db/setup_test.clj+++ b/test/metabase/app_db/setup_test.clj@@ -56,7 +56,7 @@(testing "Should be able to set up an arbitrary application DB"(letfn [(test* [data-source](is (= :done- (mdb.setup/setup-db! :h2 data-source true true)))+ (mdb.setup/setup-db! :h2 data-source {:create-sample-content? true})))(is (= ["Administrators" "All Users" "All tenant users" "Data Analysts"](mapv :name (jdbc/query {:datasource data-source}"SELECT name FROM permissions_group ORDER BY name ASC;")))))]@@ -72,7 +72,7 @@(testing "test `create-sample-content?` arg works"(doseq [create-sample-content? [true false]](let [data-source (mdb.data-source/raw-connection-string->DataSource (str "jdbc:h2:" (subname)))]- (mdb.setup/setup-db! :h2 data-source true create-sample-content?)+ (mdb.setup/setup-db! :h2 data-source {:create-sample-content? create-sample-content?})(is (= (if create-sample-content?["E-commerce Insights"][])@@ -84,7 +84,7 @@(testing "can setup a fresh db"(mt/with-temp-empty-app-db [conn driver/*driver*](is (= :done- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) true true)))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) {:create-sample-content? true})))(testing "migrations are executed in the order they are defined"(is (= (mdb.test-util/all-liquibase-ids false driver/*driver* conn)(t2/select-pks-vec (liquibase/changelog-table-name conn) {:order-by [[:orderexecuted :asc]]}))))))))@@ -94,20 +94,20 @@(mt/with-temp-empty-app-db [_conn driver/*driver*](testing "Running setup with `auto-migrate?`=false should pass if no migrations exist which need to be run"(is (= :done- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) true false)))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source))))(is (= :done- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) false false)))))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) {:auto-migrate? false})))))(testing "Setting up DB with `auto-migrate?`=false should exit if any migrations exist which need to be run";; Use a migration file that intentionally errors with failOnError: false, so that a migration is still unrun;; when we re-run `setup-db!`(with-redefs [liquibase/changelog-file "error-migration.yaml"](mt/with-temp-empty-app-db [_conn driver/*driver*](is (= :done- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) true false)))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source))))(is (thrown-with-msg?Exception#"Database requires manual upgrade."- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) false false))))))))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) {:auto-migrate? false}))))))))(defn- update-to-changelog-id[change-log-id conn]@@ -127,7 +127,7 @@;; set up a db in a way we have a MB instance running metabase 44(update-to-changelog-id "v44.00-000" conn))(is (= :done- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) true false))))))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source)))))))(deftest setup-a-mb-instance-running-version-greater-than-45(mt/test-drivers #{:h2 :mysql :postgres}@@ -136,7 +136,7 @@;; set up a db in a way we have a MB instance running metabase 45(update-to-changelog-id "v45.00-001" conn))(is (= :done- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) true false))))))+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source)))))))(deftest downgrade-detection-test(mt/test-drivers #{:h2 :mysql :postgres}@@ -160,7 +160,7 @@(mt/test-drivers #{:h2 :mysql :postgres}(mt/with-temp-empty-app-db [conn driver/*driver*];; Run all real migrations first so the changelog table exists- (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source) true false)+ (mdb.setup/setup-db! driver/*driver* (mdb.connection/data-source))(liquibase/with-liquibase [liquibase conn](let [table (liquibase/changelog-table-name liquibase)db-conn {:connection conn}diff --git a/test/metabase/cmd/dump_to_h2_test.clj b/test/metabase/cmd/dump_to_h2_test.cljindex 50e22a65d34..79aa7d930b8 100644--- a/test/metabase/cmd/dump_to_h2_test.clj+++ b/test/metabase/cmd/dump_to_h2_test.clj@@ -75,8 +75,8 @@(binding [copy/*copy-h2-database-details* true](load-from-h2/load-from-h2! h2-fixture-db-file)(encryption-test/with-secret-key "89ulvIGoiYw6mNELuOoEZphQafnF/zYe+3vT+v70D1A="- ;; the fixture was loaded unencrypted; encrypt it under the key first (as startup's- ;; check-encryption does) so no encrypted column is left plaintext for the strict model reads+ ;; the fixture was loaded unencrypted; encrypt it under the key first (as `enable-encryption`+ ;; does) so no encrypted column is left plaintext for the strict model reads;; the update and dump below trigger(mdb.encryption/encrypt-db driver/*driver* (:data-source mdb.connection/*application-db*) nil)(t2/insert! :model/Setting {:key "my-site-admin", :value "baz"})@@ -124,6 +124,9 @@(binding [copy/*copy-h2-database-details* true](load-from-h2/load-from-h2! h2-fixture-db-file)(encryption-test/with-secret-key "89ulvIGoiYw6mNELuOoEZphQafnF/zYe+3vT+v70D1A="+ ;; the fixture was loaded unencrypted; encrypt it under the key first (as `enable-encryption`+ ;; does), otherwise the dump would hold data under an absent sentinel and its setup would refuse+ (mdb.encryption/encrypt-db driver/*driver* (:data-source mdb.connection/*application-db*) nil)(t2/insert! :model/Database {:engine "h2":name "normal-db":details {:db "/tmp/test.db"}diff --git a/test/metabase/cmd/enable_encryption_test.clj b/test/metabase/cmd/enable_encryption_test.cljnew file mode 100644index 00000000000..65d4912a23b--- /dev/null+++ b/test/metabase/cmd/enable_encryption_test.clj@@ -0,0 +1,57 @@+(ns metabase.cmd.enable-encryption-test+ (:require+ [clojure.test :refer :all]+ [metabase.app-db.connection :as mdb.connection]+ [metabase.app-db.core :as mdb]+ [metabase.cmd.core :as cmd]+ [metabase.cmd.enable-encryption :refer [enable-encryption!]]+ [metabase.test :as mt]+ [metabase.test.fixtures :as fixtures]+ [metabase.util.encryption :as encryption]+ [metabase.util.encryption-test :as encryption-test]+ [toucan2.core :as t2]))++(set! *warn-on-reflection* true)++(use-fixtures :once (fixtures/initialize :db))++(defn- raw-value [k]+ (t2/select-one-fn :value :setting :key k))++(defn- restart! []+ (reset! (:status mdb.connection/*application-db*) ::not-set-up)+ (mdb/setup-db! :create-sample-content? false))++(deftest cmd-enable-encryption-errors-when-failed-test+ (mt/with-dynamic-fn-redefs [enable-encryption! #(throw (Exception. "err"))+ cmd/system-exit! identity]+ (is (= 1 (cmd/enable-encryption)))))++(deftest enable-encryption!-test+ (testing "enabling encryption on an existing, unencrypted instance"+ (encryption-test/with-secret-key nil+ (mt/with-temp-empty-app-db [_conn :h2]+ (mdb/setup-db! :create-sample-content? true)+ (t2/insert! :model/Setting {:key "test-setting", :value "plain value"})+ ;; the v53 migration's legacy plaintext marker; new code never writes it and reads it as "no sentinel"+ (is (= "unencrypted" (raw-value "encryption-check")))+ (encryption-test/with-secret-key "key1"+ (testing "startup refuses until the command has been run"+ (is (thrown-with-msg? Exception #"already contains data.*run `enable-encryption`" (restart!)))+ (is (= "unencrypted" (raw-value "encryption-check"))))+ (testing "the command encrypts everything and writes the sentinel"+ (enable-encryption!)+ (is (encryption/decryptable-string? (raw-value "encryption-check")))+ (is (encryption/decryptable-string? (raw-value "test-setting")))+ (is (= "plain value" (t2/select-one-fn :value :model/Setting :key "test-setting")))+ (is (encryption/decryptable-string? (t2/select-one-fn :details :metabase_database)))+ (is (map? (t2/select-one-fn :details :model/Database))))+ (testing "startup now succeeds"+ (is (= :done (restart!))))+ (testing "running it again is a no-op"+ (let [sentinel (raw-value "encryption-check")]+ (enable-encryption!)+ (is (= sentinel (raw-value "encryption-check"))))))+ (testing "running it with a different key than the database was encrypted with aborts"+ (encryption-test/with-secret-key "key2"+ (is (thrown-with-msg? Exception #"encrypted with a different key" (enable-encryption!)))))))))diff --git a/test/metabase/cmd/load_from_h2_test.clj b/test/metabase/cmd/load_from_h2_test.cljindex 3748a0ba627..3ad4cf1a8b2 100644--- a/test/metabase/cmd/load_from_h2_test.clj+++ b/test/metabase/cmd/load_from_h2_test.clj@@ -70,7 +70,7 @@[db-type db-def data-source](tx/destroy-db! db-type db-def)(tx/create-db! db-type db-def)- (mdb.setup/setup-db! db-type data-source true false)+ (mdb.setup/setup-db! db-type data-source)(search/reset-tracking!))(defn- dump-filenamediff --git a/test/metabase/cmd/remove_encryption_test.clj b/test/metabase/cmd/remove_encryption_test.cljindex 86d6b347c61..bd20bfd48f8 100644--- a/test/metabase/cmd/remove_encryption_test.clj+++ b/test/metabase/cmd/remove_encryption_test.clj@@ -32,8 +32,8 @@(mt/with-temp-empty-app-db [_conn :h2](mdb/setup-db! :create-sample-content? true)(t2/insert! :model/Setting {:key "test-setting", :value "unencrypted value"})- (is (encryption/possibly-encrypted-string? (raw-value _conn "encryption-check")))- (is (encryption/possibly-encrypted-string? (raw-value _conn "test-setting")))+ (is (encryption/decryptable-string? (raw-value _conn "encryption-check")))+ (is (encryption/decryptable-string? (raw-value _conn "test-setting")))(remove-encryption!)(is (= "unencrypted" (raw-value _conn "encryption-check")))(is (not (encryption/possibly-encrypted-string? (raw-value _conn "test-setting"))))))))diff --git a/test/metabase/cmd/rotate_encryption_key_test.clj b/test/metabase/cmd/rotate_encryption_key_test.cljindex 488c09ea77d..4c4fe9532ed 100644--- a/test/metabase/cmd/rotate_encryption_key_test.clj+++ b/test/metabase/cmd/rotate_encryption_key_test.clj@@ -135,9 +135,9 @@(is (mt/secret-value-equals? secret-val (t2/select-one-fn :value :model/Secret :id @secret-id-unenc)))))(testing "but not with old key"(encryption-test/with-secret-key k1- ;; the Setting after-select is lenient: an undecryptable value comes back as-is (the raw ciphertext)- ;; rather than throwing, so the reader degrades gracefully instead of failing on every settings read- (is (not= "unencrypted value" (t2/select-one-fn :value :model/Setting :key "nocrypt")))+ ;; reading a value that looks encrypted but can't be decrypted with the current key throws rather+ ;; than returning the raw ciphertext, so it can never be mistaken for a real plaintext value+ (is (thrown? clojure.lang.ExceptionInfo (t2/select-one-fn :value :model/Setting :key "nocrypt")))(is (thrown? clojure.lang.ExceptionInfo (t2/select-one-fn :details :model/Database :id 1)))(is (thrown? clojure.lang.ExceptionInfo (t2/select-one-fn :settings :model/Database :id 1)))(is (thrown? clojure.lang.ExceptionInfo (t2/select-one-fn :settings :model/User :id @user-id)))diff --git a/test/metabase/public_sharing_rest/api_test.clj b/test/metabase/public_sharing_rest/api_test.cljindex f7629124a68..912611d3916 100644--- a/test/metabase/public_sharing_rest/api_test.clj+++ b/test/metabase/public_sharing_rest/api_test.clj@@ -9,6 +9,7 @@[dk.ative.docjure.spreadsheet :as spreadsheet][metabase.analytics.snowplow-test :as snowplow-test][metabase.analytics.stats :as stats]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.dashboards-rest.api-test :as api.dashboard-test][metabase.lib.core :as lib][metabase.lib.metadata :as lib.metadata]@@ -29,7 +30,6 @@[metabase.tiles.api-test :as tiles.api-test][metabase.util :as u][metabase.util.encryption :as encryption]- [metabase.util.encryption-test :as encryption-test][metabase.util.json :as json][metabase.warehouse-schema.models.field-values :as field-values][throttle.core :as throttle]@@ -93,6 +93,9 @@(def ^:private encryption-test-secret-key "public-uuid-encryption-test-key")+(use-fixtures :once+ (encryption-tu/with-encrypted-app-db-fixture (encryption/secret-key->hash encryption-test-secret-key)))+(defn- raw-public-uuid"Read the `public_uuid` column straight from the DB (raw ciphertext), bypassing the model's decrypting transform."[model id]@@ -123,7 +126,7 @@(testing "sharing encrypts the uuid at rest and derives the prefix"(t2/update! model id {:public_uuid uuid})(let [raw (raw-public-uuid model id)]- (is (encryption/possibly-encrypted-string? raw) "public_uuid is stored as ciphertext")+ (is (encryption/decryptable-string? raw) "public_uuid is stored as ciphertext")(is (not= uuid raw) "public_uuid is not stored in plaintext")(is (= uuid (encryption/maybe-decrypt raw)) "and decrypts back to the uuid"))(is (= (subs uuid 0 public-sharing/public-uuid-prefix-length) (raw-public-uuid-prefix model id))@@ -142,26 +145,26 @@(is (nil? (raw-public-uuid-prefix model id)))(is (nil? (public-sharing/public-uuid->id model uuid)) "no longer resolves"))))-(deftest card-public-uuid-encryption-lifecycle-test- (encryption-test/with-secret-key encryption-test-secret-key+(deftest ^:synchronized card-public-uuid-encryption-lifecycle-test+ (encryption-tu/with-encrypted-app-db(mt/with-temporary-setting-values [enable-public-sharing true](mt/with-temp [:model/Card {id :id} {}](assert-public-uuid-lifecycle! :model/Card id)))))-(deftest dashboard-public-uuid-encryption-lifecycle-test- (encryption-test/with-secret-key encryption-test-secret-key+(deftest ^:synchronized dashboard-public-uuid-encryption-lifecycle-test+ (encryption-tu/with-encrypted-app-db(mt/with-temporary-setting-values [enable-public-sharing true](mt/with-temp [:model/Dashboard {id :id} {}](assert-public-uuid-lifecycle! :model/Dashboard id)))))-(deftest document-public-uuid-encryption-lifecycle-test- (encryption-test/with-secret-key encryption-test-secret-key+(deftest ^:synchronized document-public-uuid-encryption-lifecycle-test+ (encryption-tu/with-encrypted-app-db(mt/with-temporary-setting-values [enable-public-sharing true](mt/with-temp [:model/Document {id :id} {:name "Signature Doc"}](assert-public-uuid-lifecycle! :model/Document id)))))-(deftest action-public-uuid-encryption-lifecycle-test- (encryption-test/with-secret-key encryption-test-secret-key+(deftest ^:synchronized action-public-uuid-encryption-lifecycle-test+ (encryption-tu/with-encrypted-app-db(mt/with-actions-enabled(mt/with-temporary-setting-values [enable-public-sharing true](mt/with-actions [{action-id :action-id} {}]@@ -169,9 +172,9 @@(t2/update! :model/Action action-id {:public_uuid nil})(assert-public-uuid-lifecycle! :model/Action action-id))))))-(deftest public-uuid-resolves-via-endpoint-test+(deftest ^:synchronized public-uuid-resolves-via-endpoint-test(testing "GET /api/public/... resolves a shared entity by its uuid through the prefix lookup"- (encryption-test/with-secret-key encryption-test-secret-key+ (encryption-tu/with-encrypted-app-db(mt/with-temporary-setting-values [enable-public-sharing true](with-temp-public-card [{uuid :public_uuid, card-id :id}](is (= card-id (:id (mt/client :get 200 (str "public/card/" uuid))))))@@ -189,8 +192,8 @@"a plaintext public_uuid forged via raw SQL fails the strict read instead of resolving")(set-raw-public-uuid! model id nil))))-(deftest forged-plaintext-public-uuid-does-not-resolve-test- (encryption-test/with-secret-key encryption-test-secret-key+(deftest ^:synchronized forged-plaintext-public-uuid-does-not-resolve-test+ (encryption-tu/with-encrypted-app-db(mt/with-temporary-setting-values [enable-public-sharing true](mt/with-temp [:model/Card {card-id :id} {}](assert-forged-plaintext-does-not-resolve! :model/Card card-id))diff --git a/test/metabase/settings/models/setting_test.clj b/test/metabase/settings/models/setting_test.cljindex f9fe68b1238..8890d832bfe 100644--- a/test/metabase/settings/models/setting_test.clj+++ b/test/metabase/settings/models/setting_test.clj@@ -618,35 +618,42 @@(deftest encrypted-settings-test(testing "If encryption is *enabled*, make sure Settings get saved as encrypted!"- (encryption-test/with-secret-key "ABCDEFGH12345678"- (toucan-name! "Sad Can")- (is (u/base64-string? (actual-value-in-db :toucan-name)))- (testing "make sure it can be decrypted as well..."- (is (= "Sad Can"- (toucan-name)))))- (testing "But if encryption is not enabled, of course Settings shouldn't get saved as encrypted."- (encryption-test/with-secret-key nil+ ;; Setting an encryption key without running encrypt-db leaves the other encrypted settings in the shared app DB+ ;; stored plaintext, and restoring the whole-table settings cache strictly decrypts every one of them. Use an+ ;; isolated app DB so this test's key only meets settings it wrote itself.+ (mt/with-temp-empty-app-db [_conn :h2]+ (mdb/setup-db! :create-sample-content? false)+ (encryption-test/with-secret-key "ABCDEFGH12345678"(toucan-name! "Sad Can")- (is (= "Sad Can"- (actual-value-in-db :toucan-name)))))))+ (is (u/base64-string? (actual-value-in-db :toucan-name)))+ (testing "make sure it can be decrypted as well..."+ (is (= "Sad Can"+ (toucan-name)))))+ (testing "But if encryption is not enabled, of course Settings shouldn't get saved as encrypted."+ (encryption-test/with-secret-key nil+ (toucan-name! "Sad Can")+ (is (= "Sad Can"+ (actual-value-in-db :toucan-name))))))))(deftest previously-encrypted-settings-test(testing "Make sure settings that were encrypted don't cause `user-facing-info` to blow up if encyrption key changed"- (mt/discard-setting-changes [test-json-setting]- (encryption-test/with-secret-key "0B9cD6++AME+A7/oR7Y2xvPRHX3cHA2z7w+LbObd/9Y="- (test-json-setting! {:abc 123})- (is (not= "{\"abc\":123}"- (actual-value-in-db :test-json-setting))))- (testing (str "If fetching the Setting fails (e.g. because key changed) `user-facing-info` should return `nil` "- "rather than failing entirely")- (encryption-test/with-secret-key nil- (is (= {:key :test-json-setting- :value nil- :is_env_setting false- :env_name "MB_TEST_JSON_SETTING"- :description "Test setting - this only shows up in dev (4)"- :default nil}- (#'setting/user-facing-info (setting/resolve-setting :test-json-setting)))))))))+ (mt/with-temp-empty-app-db [_conn :h2]+ (mdb/setup-db! :create-sample-content? false)+ (mt/discard-setting-changes [test-json-setting]+ (encryption-test/with-secret-key "0B9cD6++AME+A7/oR7Y2xvPRHX3cHA2z7w+LbObd/9Y="+ (test-json-setting! {:abc 123})+ (is (not= "{\"abc\":123}"+ (actual-value-in-db :test-json-setting))))+ (testing (str "If fetching the Setting fails (e.g. because key changed) `user-facing-info` should return `nil` "+ "rather than failing entirely")+ (encryption-test/with-secret-key nil+ (is (= {:key :test-json-setting+ :value nil+ :is_env_setting false+ :env_name "MB_TEST_JSON_SETTING"+ :description "Test setting - this only shows up in dev (4)"+ :default nil}+ (#'setting/user-facing-info (setting/resolve-setting :test-json-setting))))))))));;; ----------------------------------------------- TIMESTAMP SETTINGS -----------------------------------------------diff --git a/test/metabase/setup/core_test.clj b/test/metabase/setup/core_test.cljindex 647146fd40c..36c0b1647f1 100644--- a/test/metabase/setup/core_test.clj+++ b/test/metabase/setup/core_test.clj@@ -4,11 +4,13 @@[clojure.test :refer :all][metabase.app-db.connection :as mdb.connection][metabase.app-db.core :as mdb]+ [metabase.app-db.encryption :as mdb.encryption][metabase.appearance.core :as appearance][metabase.config.core :as config][metabase.driver :as driver][metabase.models.interface :as mi][metabase.query-processor.middleware.cache-backend.interface :as i]+ [metabase.settings.models.setting :as setting][metabase.setup.core :as setup][metabase.test :as mt][metabase.test.fixtures :as fixtures]@@ -132,30 +134,114 @@(mdb/setup-db! :create-sample-content? true)(let [cache-backend (i/cache-backend :db)](i/save-results! cache-backend (codecs/to-bytes "cache-key") (codecs/to-bytes "cache-value"))+ ;; the v53 migration's legacy plaintext marker; new code never writes it and reads it as "no sentinel"(is (= "unencrypted" (t2/select-one-fn :value "setting" :key "encryption-check")))(is (not (encryption/possibly-encrypted-string? (t2/select-one-fn :details "metabase_database"))))(is (= 1 (t2/count :model/QueryCache)))- (testing "Adding encryption encrypts database on restart"+ (testing "Adding a key to an existing instance refuses to start rather than encrypting on its own"(encryption-test/with-secret-key "key1"(reset! (:status mdb.connection/*application-db*) ::setup-finished)- (mdb/setup-db! :create-sample-content? false)- (is (encryption/possibly-encrypted-string? (:value (t2/select-one "setting" :key "encryption-check"))))- (is (encryption/possibly-encrypted-string? (:details (t2/select-one "metabase_database"))))- (testing "Cache is cleared on encryption"- (is (= 0 (t2/count :model/QueryCache))))))))))+ (is (thrown-with-msg? Exception #"already contains data.*run `enable-encryption`"+ (mdb/setup-db! :create-sample-content? false)))+ (is (= "unencrypted" (t2/select-one-fn :value "setting" :key "encryption-check")))+ (is (not (encryption/possibly-encrypted-string? (t2/select-one-fn :details "metabase_database"))))+ (is (= 1 (t2/count :model/QueryCache)))+ (testing "after `enable-encryption` the database is encrypted and starts"+ (mdb/encrypt-db driver/*driver* (mdb/data-source) nil)+ (is (encryption/decryptable-string? (:value (t2/select-one "setting" :key "encryption-check"))))+ (is (encryption/decryptable-string? (:details (t2/select-one "metabase_database"))))+ (testing "Cache is cleared on encryption"+ (is (= 0 (t2/count :model/QueryCache))))+ (reset! (:status mdb.connection/*application-db*) ::setup-finished)+ (is (= :done (mdb/setup-db! :create-sample-content? false))))))))))(testing "Database created with encryption configured is encrypted"(encryption-test/with-secret-key "key2"(mt/with-temp-empty-app-db [_conn driver/*driver*](mdb/setup-db! :create-sample-content? true)- (is (encryption/possibly-encrypted-string? (t2/select-one-fn :value "setting" :key "encryption-check")))- (is (encryption/possibly-encrypted-string? (t2/select-one-fn :details "metabase_database")))+ (is (encryption/decryptable-string? (t2/select-one-fn :value "setting" :key "encryption-check")))+ (is (encryption/decryptable-string? (t2/select-one-fn :details "metabase_database")))(testing "Re-running server works"(reset! (:status mdb.connection/*application-db*) ::setup-finished)(mdb/setup-db! :create-sample-content? false)- (is (encryption/possibly-encrypted-string? (:value (t2/select-one "setting" :key "encryption-check")))))+ (is (encryption/decryptable-string? (:value (t2/select-one "setting" :key "encryption-check")))))+ (testing "A missing sentinel on a database whose content decrypts with the key is written back on startup"+ (t2/delete! :setting :key "encryption-check")+ (reset! (:status mdb.connection/*application-db*) ::setup-finished)+ (is (= :done (mdb/setup-db! :create-sample-content? false)))+ (is (string/valid-uuid? (encryption/maybe-decrypt (t2/select-one-fn :value "setting" :key "encryption-check")))))+ (testing "The legacy plaintext \"unencrypted\" marker on such a database is replaced the same way"+ (t2/delete! :setting :key "encryption-check")+ (t2/insert! :setting {:key "encryption-check", :value "unencrypted"})+ (reset! (:status mdb.connection/*application-db*) ::setup-finished)+ (is (= :done (mdb/setup-db! :create-sample-content? false)))+ (is (string/valid-uuid? (encryption/maybe-decrypt (t2/select-one-fn :value "setting" :key "encryption-check")))))+ (testing "Starting without the key throws"+ (encryption-test/with-secret-key nil+ (reset! (:status mdb.connection/*application-db*) ::setup-finished)+ (is (thrown-with-msg? Exception #"MB_ENCRYPTION_SECRET_KEY environment variable was NOT set"+ (mdb/setup-db! :create-sample-content? false)))))(testing "Different encryption key throws an error"(encryption-test/with-secret-key "different-key"(reset! (:status mdb.connection/*application-db*) ::setup-finished)(is (thrown-with-msg? Exception #"Database was encrypted with a different key than the MB_ENCRYPTION_SECRET_KEY environment contains" (mdb/setup-db! :create-sample-content? false)))(let [setting-value (:value (t2/select-one "setting" :key "site-uuid-for-version-info-fetching"))] ; need to select directly from "settings" to avoid auto-decryption(is (not (string/valid-uuid? setting-value)))))))))))++(defn- set-encryption-check-raw! [value]+ (t2/delete! :setting :key "encryption-check")+ (when value+ (t2/insert! :setting {:key "encryption-check", :value value})))++(deftest sentinel-state-never-triggers-encryption-test+ (testing "a restart must never encrypt a pre-existing plaintext row, whatever state the sentinel is in"+ (encryption-test/with-secret-key "sentinel-state-key-1"+ (mt/with-temp-empty-app-db [_conn :h2]+ (mdb/setup-db! :create-sample-content? true)+ (let [db-id (t2/select-one-fn :id :metabase_database)+ plaintext "{\"host\":\"example.com\"}"+ raw-detail #(t2/select-one-fn :details :metabase_database :id db-id)+ restart! (fn []+ (reset! (:status mdb.connection/*application-db*) ::setup-finished)+ (mdb/setup-db! :create-sample-content? false))]+ (is (encryption/decryptable-string? (raw-detail)))+ (t2/update! :metabase_database {:id db-id} {:details plaintext})+ (testing "with the sentinel deleted"+ (set-encryption-check-raw! nil)+ (is (thrown-with-msg? Exception #"not marked as encrypted and already contains data" (restart!)))+ (testing "the plaintext row is untouched and still rejected by the strict read"+ (is (= plaintext (raw-detail)))+ (is (thrown? Exception (:details (t2/select-one :model/Database :id db-id)))))+ (testing "no sentinel was written"+ (is (nil? (t2/select-one-fn :value :setting :key "encryption-check")))))+ (testing "\na plaintext random-uuid sentinel is rejected as a wrong key"+ (set-encryption-check-raw! (str (random-uuid)))+ (is (thrown-with-msg? Exception #"encrypted with a different key" (restart!)))+ (is (= plaintext (raw-detail))))+ (testing "\nthe legacy \"unencrypted\" marker reads as no sentinel: the plaintext row still refuses startup"+ (set-encryption-check-raw! "unencrypted")+ (is (thrown-with-msg? Exception #"not marked as encrypted and already contains data" (restart!)))+ (is (= plaintext (raw-detail)))+ (testing "and no sentinel was written"+ (is (= "unencrypted" (t2/select-one-fn :value :setting :key "encryption-check"))))))))))++(deftest fresh-install-with-encryption-key-test+ (testing "a database created with MB_ENCRYPTION_SECRET_KEY set stores every encrypted-at-rest value encrypted"+ (encryption-test/with-secret-key "fresh-install-test-key-1234"+ (mt/with-temp-empty-app-db [_conn :h2]+ (mdb/setup-db! :create-sample-content? true)+ (testing "encrypted-at-rest columns (read raw, so no model transform can hide a plaintext value)"+ (doseq [[table column] @#'mdb.encryption/encrypted-string-columns+ {:keys [id value]} (t2/select [table :id [column :value]] {:where [:!= column nil]})]+ (testing (format "%s.%s id %s" (name table) (name column) id)+ (is (encryption/decryptable-string? value)))))+ (testing "encrypted-at-rest bytes columns"+ (doseq [[table column] @#'mdb.encryption/encrypted-bytes-columns+ {:keys [id value]} (t2/select [table :id [column :value]] {:where [:!= column nil]})]+ (testing (format "%s.%s id %s" (name table) (name column) id)+ (is (encryption/decryptable-bytes? (#'mdb.encryption/maybe-blob->bytes value))))))+ (testing "settings that are encrypted at rest"+ (doseq [{k :key v :value} (t2/select :setting {:where [:!= :value nil]})+ :let [definition (get @setting/registered-settings (keyword k))]+ :when (and definition (not= :no (:encryption definition)))]+ (testing k+ (is (encryption/decryptable-string? v)))))))))diff --git a/test/metabase/slackbot/api_test.clj b/test/metabase/slackbot/api_test.cljindex 2a008d3b4fb..98a65515488 100644--- a/test/metabase/slackbot/api_test.clj+++ b/test/metabase/slackbot/api_test.clj@@ -3,6 +3,7 @@[clojure.string :as str][clojure.test :refer :all][metabase.analytics.prometheus-test :as prometheus-test]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.channel.settings :as channel.settings][metabase.metabot.agent.core :as agent][metabase.server.settings :as server.settings]@@ -22,7 +23,9 @@(set! *warn-on-reflection* true)-(use-fixtures :once (fixtures/initialize :test-users))+(use-fixtures :once+ (fixtures/initialize :test-users)+ (encryption-tu/with-encrypted-app-db-fixture tu/test-encryption-key))(deftest manifest-endpoint-test(testing "GET /api/slack/manifest with metabot-v3 feature"@@ -47,7 +50,7 @@(is (= "You must configure a site-url for Slack integration to work."(mt/user-http-request :crowberto :get 503 "slack/manifest"))))))))-(deftest events-endpoint-test+(deftest ^:synchronized events-endpoint-test(testing "POST /api/metabot/slack/events"(tu/with-slackbot-setup(testing "handles URL verification challenge"@@ -82,7 +85,7 @@{:type "url_verification":challenge "test"})))))))-(deftest feature-flag-test+(deftest ^:synchronized feature-flag-test(testing "POST /api/metabot/slack/events"(testing "ack events even when metabot-v3 feature is disabled to prevent Slack retries"(tu/with-slackbot-setup@@ -93,7 +96,7 @@body)](is (= "ok" response) "Should ACK the event with 200 OK")))))))-(deftest edited-message-ignored-test+(deftest ^:synchronized edited-message-ignored-test(testing "POST /events ignores edited messages"(tu/with-slackbot-setup(doseq [[desc event-mod] [["with :edited key" {:edited {:user "U123" :ts "123"}}]@@ -111,7 +114,7 @@(is (= 0 (count @post-calls)))(is (= 0 (count @ephemeral-calls))))))))))))-(deftest message-deleted-ignored-test+(deftest ^:synchronized message-deleted-ignored-test(testing "POST /events ignores message_deleted events"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge {:subtype "message_deleted"})@@ -128,7 +131,7 @@(is @ignored "Event should have been routed to ignore-event")(is (= 0 (count @post-calls)))))))))))-(deftest slackbot-disabled-setting-test+(deftest ^:synchronized slackbot-disabled-setting-test(testing "POST /events acks but does not process when slack-connect-enabled is false"(tu/with-slackbot-setup(mt/with-temporary-setting-values [sso-settings/slack-connect-enabled false]@@ -148,7 +151,7 @@(is (= 0 (count @delete-calls)) "No messages should be deleted")(is (= 0 (count @ephemeral-calls)) "No ephemeral messages should be sent"))))))))))-(deftest user-message-triggers-response-test+(deftest ^:synchronized user-message-triggers-response-test(testing "POST /events with user message triggers AI response via Slack streaming"(tu/with-slackbot-setup(let [mock-ai-text "Here is your answer"@@ -175,7 +178,7 @@(is (empty? @add-reaction-calls))(is (empty? @remove-reaction-calls))))))))))-(deftest app-mention-triggers-response-test+(deftest ^:synchronized app-mention-triggers-response-test(testing "POST /events with app_mention uses visible channel reply (not streaming)"(tu/with-slackbot-setup(let [mock-ai-text "Here is your answer"@@ -206,7 +209,7 @@(let [msg (t2/select-one :model/MetabotMessage :channel_id channel-id :role "assistant")](is (some? (:slack_msg_id msg))))))))))))-(deftest app-mention-long-answer-fits-slack-blocks-test+(deftest ^:synchronized app-mention-long-answer-fits-slack-blocks-test(testing "POST /events with app_mention truncates a long answer so Slack accepts it (BOT-1606)"(tu/with-slackbot-setup(let [channel-id "C-LONG-ANSWER-TEST"@@ -240,7 +243,7 @@"too long to post in Slack")))blocks)))))))))))-(deftest stream-start-failure-test+(deftest ^:synchronized stream-start-failure-test(testing "When start-stream fails, falls back to a regular message"(tu/with-slackbot-setup(let [event-body tu/base-dm-event]@@ -264,7 +267,7 @@(is (= 0 (count @stop-stream-calls))))))))))));; Not ^:parallel: `with-prometheus-system!` redefs a process-global var.-(deftest dm-response-undeliverable-metric-test+(deftest ^:synchronized dm-response-undeliverable-metric-test(testing "a DM whose stream and plain-text fallback both fail is counted as undeliverable"(tu/with-slackbot-setup(mt/with-prometheus-system! [_ system]@@ -286,7 +289,7 @@:timeout-ms 5000})(is (= 1.0 (mt/metric-value system :metabase-slackbot/responses-undeliverable))))))))))))-(deftest ai-request-error-stops-stream-test+(deftest ^:synchronized ai-request-error-stops-stream-test(testing "When the agent loop throws after the stream has started, the stream is stopped"(tu/with-slackbot-setup(let [event-body tu/base-dm-event]@@ -317,7 +320,7 @@(testing "stream was stopped during cleanup"(is (= 1 (count @stop-stream-calls))))))))))))-(deftest streaming-request-args-test+(deftest ^:synchronized streaming-request-args-test(testing "POST /events passes correct arguments to agent/run-agent-loop"(tu/with-slackbot-setup(doseq [[desc event-body]@@ -351,7 +354,7 @@(is (str/includes? content "Do not narrate the steps you took")"channel response-style suffix is appended"))))))))))))-(deftest slack-msg-id-stored-test+(deftest ^:synchronized slack-msg-id-stored-test(testing "User and bot messages are stored with their Slack ts as slack_msg_id"(tu/with-slackbot-setup(let [event-ts "1709567890.000001"@@ -379,7 +382,7 @@(is (= "C123" (:channel_id bot-msg)))(is (= (mt/user->id :rasta) (:user_id bot-msg)))))))))))))-(deftest user-message-with-visualizations-test+(deftest ^:synchronized user-message-with-visualizations-test(testing "POST /events with visualizations uploads images and finalizes them in stop-stream blocks"(tu/with-slackbot-setup(let [mock-ai-text "Here are your charts"@@ -425,7 +428,7 @@(mapv :type blocks)))(is (= "feedback_buttons" (get-in blocks [4 :elements 0 :type]))))))))))))-(deftest user-not-linked-sends-auth-message-test+(deftest ^:synchronized user-not-linked-sends-auth-message-test(testing "POST /events with unlinked user sends auth message (DM, no user mention prefix)"(tu/with-slackbot-setup(let [event-body (assoc-in tu/base-dm-event [:event :user] "U-UNKNOWN-USER")]@@ -446,7 +449,7 @@:text #"(?i).*connect.*slack.*metabase.*"}]@post-calls))))))))))-(deftest app-mention-unlinked-user-test+(deftest ^:synchronized app-mention-unlinked-user-test(testing "POST /events with app_mention from unlinked user sends ephemeral auth message"(tu/with-slackbot-setup(doseq [[desc thread-ts expected-thread-ts]@@ -547,7 +550,7 @@:provider_id slack-id}](is (nil? (#'slackbot/slack-id->user-id slack-id))))))))))-(deftest channel-message-without-mention-no-auth-test+(deftest ^:synchronized channel-message-without-mention-no-auth-test(testing "POST /events with channel message (no @mention) from unlinked user should NOT send auth message"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -572,7 +575,7 @@(testing "no ephemeral auth messages sent"(is (= 0 (count @ephemeral-calls)))))))))))-(deftest channel-message-without-mention-linked-user-test+(deftest ^:synchronized channel-message-without-mention-linked-user-test(testing "POST /events with channel message from linked user should be silently ignored"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -595,7 +598,7 @@(testing "no ephemeral messages"(is (= 0 (count @ephemeral-calls)))))))))))-(deftest channel-file-share-without-mention-ignored-test+(deftest ^:synchronized channel-file-share-without-mention-ignored-test(testing "POST /events with file_share in channel without @mention is ignored"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -652,7 +655,7 @@:request-user-id user-id}(#'slackbot/authorize-delete-request "U123" "C123" "ts123"))))))))-(deftest handle-delete-reaction-test+(deftest ^:synchronized handle-delete-reaction-test(testing "reaction_added with a delete emoji replaces the bot response with a removed notice"(tu/with-slackbot-setup(let [owner-id (mt/user->id :rasta)@@ -686,7 +689,7 @@(is (= message-ts (:ts (first @update-calls))))(is (str/includes? (:text (first @update-calls)) "removed")))))))))))-(deftest handle-delete-reaction-test-2+(deftest ^:synchronized handle-delete-reaction-test-2(testing "reaction_added with a non-delete emoji is ignored"(tu/with-slackbot-setup(let [event-body {:type "event_callback"@@ -706,7 +709,7 @@(Thread/sleep 200)(is (= 0 (count @update-calls)) "non-delete emoji should produce no update")))))))-(deftest handle-delete-reaction-test-3+(deftest ^:synchronized handle-delete-reaction-test-3(testing "reaction_added with delete emoji from non-owner is ignored"(tu/with-slackbot-setup(let [event-body {:type "event_callback"@@ -1132,7 +1135,7 @@;; -------------------------------- Visualization Integration Tests ---------------------------------(deftest adhoc-viz-execution-test+(deftest ^:synchronized adhoc-viz-execution-test(testing "POST /events with adhoc_viz executes query and uploads image"(tu/with-slackbot-setup(let [mock-ai-text "Here's your data"@@ -1161,7 +1164,7 @@(is (= ["section" "image" "context_actions"] (mapv :type blocks)))(is (re-matches #"FIMG-\d+" (get-in blocks [1 :slack_file :id])))))))))))-(deftest adhoc-viz-default-display-test+(deftest ^:synchronized adhoc-viz-default-display-test(testing "POST /events with adhoc_viz uses :table when display not specified"(tu/with-slackbot-setup(let [mock-query {:database 1 :type "query" :query {:source-table 2}}@@ -1177,7 +1180,7 @@(testing "display defaults to :table"(is (= :table (:display (first @generate-adhoc-output-calls)))))))))))-(deftest mixed-viz-types-test+(deftest ^:synchronized mixed-viz-types-test(testing "POST /events handles both static_viz and adhoc_viz in same response"(tu/with-slackbot-setup(let [mock-query {:database 1 :type "query" :query {:source-table 2}}@@ -1205,7 +1208,7 @@(is (= ["section" "image" "section" "image" "section" "image" "context_actions"](mapv :type blocks)))))))))))-(deftest viz-error-posts-error-message-test+(deftest ^:synchronized viz-error-posts-error-message-test(testing "posts error message when visualization generation fails"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -1225,7 +1228,7 @@:done? true? :timeout-ms 5000})(is (some #(= error-msg (:text %)) @post-calls))))))))))-(deftest viz-error-does-not-block-other-vizs-test+(deftest ^:synchronized viz-error-does-not-block-other-vizs-test(testing "a failing viz does not prevent subsequent vizs from rendering"(tu/with-slackbot-setup(let [fake-png (byte-array [0x89 0x50 0x4E 0x47])@@ -1251,7 +1254,7 @@(testing "second card still uploads"(is (= 1 (count @image-calls)))))))))))-(deftest viz-caption-and-link-on-image-test+(deftest ^:synchronized viz-caption-and-link-on-image-test(testing "image viz for static_viz uses the card name as caption, not the AI-provided caption"(tu/with-slackbot-setup(let [mock-data-parts [{:type "static_viz" :value {:entity_id 101 :title "AI-generated caption"}}]@@ -1277,7 +1280,7 @@(testing "image block references the uploaded Slack file"(is (= (:file-id img) (get-in blocks [1 :slack_file :id])))))))))))-(deftest table-viz-with-caption-test+(deftest ^:synchronized table-viz-with-caption-test(testing "table viz posts include caption block with link"(tu/with-slackbot-setup(let [mock-query {:database 1 :type "query" :query {:source-table 2}}@@ -1305,7 +1308,7 @@;; -------------------------------- Metrics Tests ---------------------------------(deftest dm-response-metrics-test+(deftest ^:synchronized dm-response-metrics-test(testing "Successful DM response increments prometheus counters and records duration"(mt/with-prometheus-system! [_ system](tu/with-slackbot-setup@@ -1322,7 +1325,7 @@(testing "response-duration-ms histogram is recorded"(is (pos? (:sum (mt/metric-value system :metabase-slackbot/response-duration-ms {:source "dm"}))))))))))))-(deftest channel-response-metrics-test+(deftest ^:synchronized channel-response-metrics-test(testing "Successful channel response increments prometheus counters"(mt/with-prometheus-system! [_ system](tu/with-slackbot-setup@@ -1338,7 +1341,7 @@(is (prometheus-test/approx= 1 (mt/metric-value system :metabase-slackbot/responses-generated{:source "channel" :result "success"})))))))))))-(deftest error-response-metrics-test+(deftest ^:synchronized error-response-metrics-test(testing "Failed response increments error counter and records duration"(mt/with-prometheus-system! [_ system](tu/with-slackbot-setup@@ -1358,7 +1361,7 @@(testing "response-duration-ms histogram is recorded even on error"(is (pos? (:sum (mt/metric-value system :metabase-slackbot/response-duration-ms {:source "dm"})))))))))))))-(deftest delete-response-metrics-test+(deftest ^:synchronized delete-response-metrics-test(testing "Deleting a response increments responses-deleted counter"(mt/with-prometheus-system! [_ system](tu/with-slackbot-setupdiff --git a/test/metabase/slackbot/config_test.clj b/test/metabase/slackbot/config_test.cljindex a85d17630ee..cd538e73e32 100644--- a/test/metabase/slackbot/config_test.clj+++ b/test/metabase/slackbot/config_test.clj@@ -1,6 +1,7 @@(ns metabase.slackbot.config-test(:require[clojure.test :refer :all]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.channel.settings :as channel.settings][metabase.slackbot.test-util :as tu][metabase.sso.settings :as sso-settings]@@ -10,9 +11,11 @@(set! *warn-on-reflection* true)-(use-fixtures :once (fixtures/initialize :test-users))+(use-fixtures :once+ (fixtures/initialize :test-users)+ (encryption-tu/with-encrypted-app-db-fixture tu/test-encryption-key))-(deftest setup-complete-test+(deftest ^:synchronized setup-complete-test(tu/with-slackbot-setup(let [request-body (assoc-in tu/base-dm-event [:event :text] "test")post-events #(mt/client :post %1 "metabot/slack/events"diff --git a/test/metabase/slackbot/streaming_test.clj b/test/metabase/slackbot/streaming_test.cljindex aabc8aa7f8a..3dbf8d0b3a6 100644--- a/test/metabase/slackbot/streaming_test.clj+++ b/test/metabase/slackbot/streaming_test.clj@@ -3,6 +3,7 @@[clojure.string :as str][clojure.test :refer :all][metabase.analytics.prometheus :as prometheus]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.channel.slack :as channel.slack][metabase.metabot.agent.core :as agent][metabase.metabot.persistence :as metabot.persistence]@@ -21,7 +22,9 @@(set! *warn-on-reflection* true)-(use-fixtures :once (fixtures/initialize :test-users))+(use-fixtures :once+ (fixtures/initialize :test-users)+ (encryption-tu/with-encrypted-app-db-fixture tu/test-encryption-key))(deftest ^:parallel slack-thread-conversation-id-test(testing "Same thread produces same conversation ID"@@ -271,7 +274,7 @@:positive false}(json/decode (get-in fb [:negative_button :value]) true)))))))))-(deftest streaming-response-includes-feedback-blocks-test+(deftest ^:synchronized streaming-response-includes-feedback-blocks-test(testing "send-response passes feedback blocks to stop-stream"(tu/with-slackbot-setup(let [event-body tu/base-dm-event]@@ -312,7 +315,7 @@:text "You've used all of your included AI service tokens. To keep using AI features, end your trial early and start your subscription, or add your own AI provider API key."}@posted-message))))))-(deftest slackbot-streaming-sets-ai-proxied-on-messages-test+(deftest ^:synchronized slackbot-streaming-sets-ai-proxied-on-messages-test(testing "start-turn! receives ai-proxy? = true (and writes it to both user and assistant rows)for metabase/ prefixed provider"(tu/with-slackbot-setup@@ -339,7 +342,7 @@(testing "start-turn! received ai-proxy? = true"(is (=? [{:ai-proxy? true}] @start-opts)))))))))-(deftest slackbot-streaming-seeds-state-from-db-test+(deftest ^:synchronized slackbot-streaming-seeds-state-from-db-test(testing "a turn seeds the agent loop with the state earlier turns in the thread persisted (BOT-522)"(tu/with-slackbot-setup(let [event-body tu/base-dm-event]@@ -377,7 +380,7 @@(is (= {:queries {:q1 {:database 1}}}(:state (last @ai-request-calls)))))))))))))-(deftest slackbot-streaming-records-streamed-error-test+(deftest ^:synchronized slackbot-streaming-records-streamed-error-test(testing "an :error part the agent loop emits instead of throwing is still recorded on the row,so conversation-state does not later replay a failed turn's partial state (BOT-522)"(tu/with-slackbot-setup@@ -402,7 +405,7 @@(is (not= ::timeout opts))(is (= {:message "boom"} (:error opts)))))))))))-(deftest slackbot-streaming-persists-failed-conversations-test+(deftest ^:synchronized slackbot-streaming-persists-failed-conversations-test(testing "User row is persisted even if setup throws after it (BOT-1279). With placeholders,start-turn! inserts user + placeholder atomically before any setup runs."(tu/with-slackbot-setup@@ -426,7 +429,7 @@(is (not= ::timeout opts))(is (some? (:slack-msg-id opts))))))))))))-(deftest slackbot-streaming-never-writes-pii-columns-test+(deftest ^:synchronized slackbot-streaming-never-writes-pii-columns-test(testing "Slack-originated rows leave ip_address/embedding_*/user_agent NULL regardless of analytics-pii-retention-enabled"(mt/with-premium-features #{:audit-app}(tu/with-slackbot-setup@@ -455,7 +458,7 @@(is (not (contains? opts :hostname)))(is (not (contains? opts :pii-info)))))))))))))))-(deftest slackbot-streaming-sets-ai-proxied-false-for-byok-test+(deftest ^:synchronized slackbot-streaming-sets-ai-proxied-false-for-byok-test(testing "start-turn! receives ai-proxy? = false (and writes it to both user and assistant rows)for direct BYOK provider"(tu/with-slackbot-setupdiff --git a/test/metabase/slackbot/test_util.clj b/test/metabase/slackbot/test_util.cljindex 7d564c06c28..3d62a047b30 100644--- a/test/metabase/slackbot/test_util.clj+++ b/test/metabase/slackbot/test_util.clj@@ -4,6 +4,7 @@[buddy.core.codecs :as codecs][buddy.core.mac :as mac][clojure.string :as str]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.channel.slack :as channel.slack][metabase.metabot.agent.core :as agent][metabase.slackbot.api :as slackbot]@@ -101,13 +102,14 @@(oversized-markdown-error blocks)))(defmacro with-ensure-encryption- "Use the existing encryption key if one is configured, otherwise set a test key.- Avoids conflicts with encrypted settings in the DB that were written with the real key."+ "Use the existing encryption key if one is configured, otherwise run against the namespace's prepared encrypted app+ DB (see [[metabase.app-db.encryption-test-util/with-encrypted-app-db]]). Reading a setting under an active key+ strictly decrypts it, so an isolated, already-encrypted app DB keeps that read from tripping over a plaintext+ setting left in the shared app DB by another namespace."[& body]`(if (encryption/default-encryption-enabled?)(do ~@body)- (with-redefs [encryption/default-secret-key test-encryption-key]- ~@body)))+ (encryption-tu/with-encrypted-app-db ~@body)))(defmacro with-slackbot-setup"Wrap body with all required settings for slackbot to be fully configured.diff --git a/test/metabase/slackbot/uploads_test.clj b/test/metabase/slackbot/uploads_test.cljindex f9fada4a165..6c43b7af9f7 100644--- a/test/metabase/slackbot/uploads_test.clj+++ b/test/metabase/slackbot/uploads_test.clj@@ -3,6 +3,7 @@[clojure.java.io :as io][clojure.string :as str][clojure.test :refer :all]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.channel.settings :as channel.settings][metabase.slackbot.client :as slackbot.client][metabase.slackbot.test-util :as tu]@@ -15,7 +16,9 @@(set! *warn-on-reflection* true)-(use-fixtures :once (fixtures/initialize :test-users))+(use-fixtures :once+ (fixtures/initialize :test-users)+ (encryption-tu/with-encrypted-app-db-fixture tu/test-encryption-key))(defn- with-upload-mocks!"Helper to set up common mocks for CSV upload tests.@@ -51,7 +54,7 @@(body-fn {:upload-calls upload-calls:download-calls download-calls}))))-(deftest csv-upload-disabled-test+(deftest ^:synchronized csv-upload-disabled-test(testing "POST /events with file upload when uploads are disabled"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -77,7 +80,7 @@(is (some #(= "CSV uploads are not enabled on this Metabase instance." %)@append-text-calls))))))))))))-(deftest csv-upload-success-test+(deftest ^:synchronized csv-upload-success-test(testing "POST /events with successful CSV upload"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -110,7 +113,7 @@(is (some #(= "Your CSV has been uploaded successfully as a model." %)@append-text-calls))))))))))))-(deftest csv-upload-non-csv-skipped-test+(deftest ^:synchronized csv-upload-non-csv-skipped-test(testing "POST /events with non-CSV file is skipped"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -142,7 +145,7 @@(is (some #(= "Only CSV files can be uploaded." %)@append-text-calls))))))))))))-(deftest csv-upload-non-csv-no-text-test+(deftest ^:synchronized csv-upload-non-csv-no-text-test(testing "POST /events with non-CSV file and no text responds directly without AI"(tu/with-slackbot-setup;; No :text field - user uploaded file without typing anything@@ -182,7 +185,7 @@(is (str/includes? message-text "can only process CSV and TSV"))(is (str/includes? message-text "query_result.xlsx")))))))))))))-(deftest csv-upload-mixed-files-test+(deftest ^:synchronized csv-upload-mixed-files-test(testing "POST /events with mix of CSV and non-CSV files"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event merge@@ -221,7 +224,7 @@(testing "only CSV/TSV files were uploaded"(is (= 2 (count @upload-calls)))))))))))))-(deftest csv-upload-file-too-large-test+(deftest ^:synchronized csv-upload-file-too-large-test(testing "POST /events with file exceeding size limit"(tu/with-slackbot-setup(let [too-large-size (inc (* 200 1024 1024)) ;; Just over 200MB@@ -250,7 +253,7 @@(testing "upload was not attempted"(is (= 0 (count @upload-calls)))))))))))))-(deftest csv-upload-no-permission-test+(deftest ^:synchronized csv-upload-no-permission-test(testing "POST /events with file upload when user lacks permission"(tu/with-slackbot-setup(let [event-body (update tu/base-dm-event :event mergediff --git a/test/metabase/sso/integrations/slack_connect_test.clj b/test/metabase/sso/integrations/slack_connect_test.cljindex 9c1afd044b9..9e0ba80c03d 100644--- a/test/metabase/sso/integrations/slack_connect_test.clj+++ b/test/metabase/sso/integrations/slack_connect_test.clj@@ -2,6 +2,7 @@(:require[clojure.string :as str][clojure.test :refer :all]+ [metabase.app-db.encryption-test-util :as encryption-tu][metabase.auth-identity.core :as auth-identity][metabase.sso.oidc.state :as oidc.state][metabase.sso.settings :as sso-settings]@@ -16,8 +17,6 @@(set! *warn-on-reflection* true)-(use-fixtures :once (fixtures/initialize :test-users))-(def test-encryption-key"Test encryption key for OIDC state encryption.""Orw0AAyzkO/kPTLJRxiyKoBHXa/d6ZcO+p+gpZO/wSQ=")@@ -26,11 +25,17 @@"Hashed test encryption key."(encryption/secret-key->hash test-encryption-key))+(use-fixtures :once+ (fixtures/initialize :test-users)+ (encryption-tu/with-encrypted-app-db-fixture test-secret))+(defmacro with-test-encryption!- "Wraps body with test encryption key enabled. Use for tests that involve OIDC state cookies."+ "Wraps body with test encryption key enabled, running against the namespace's isolated, already-encrypted app DB (see+ [[metabase.app-db.encryption-test-util/with-encrypted-app-db]]) so a strict decrypting read under the active key does+ not trip over a plaintext setting left in the shared app DB by another namespace. Use for tests that involve OIDC+ state cookies."[& body]- `(with-redefs [encryption/default-secret-key test-secret]- ~@body))+ `(encryption-tu/with-encrypted-app-db ~@body))(defn do-with-url-prefix-disabled"Test fixture that disables API URL prefix."@@ -64,7 +69,7 @@;;; -------------------------------------------------- Prerequisites Tests ---------------------------------------------------(deftest sso-prereqs-test+(deftest ^:synchronized sso-prereqs-test(with-test-encryption!(sso.test-helpers/do-with-other-sso-types-disabled!(fn []@@ -105,7 +110,7 @@;;; -------------------------------------------------- Redirect Tests ---------------------------------------------------(deftest redirect-test+(deftest ^:synchronized redirect-test(testing "with Slack Connect configured, a GET request should result in a redirect to Slack"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -145,7 +150,7 @@;;; -------------------------------------------------- Callback Tests ---------------------------------------------------(deftest callback-state-validation-test+(deftest ^:synchronized callback-state-validation-test(testing "callback should fail if state cookie is missing"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -156,7 +161,7 @@;; Without a state cookie, the callback fails with invalid/expired state error(is (str/includes? (:body response) "OIDC state cookie is invalid, expired, or missing")))))))-(deftest callback-state-validation-csrf-test+(deftest ^:synchronized callback-state-validation-csrf-test(testing "callback with mismatched state should indicate possible CSRF attack"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -178,7 +183,7 @@;; State mismatch should indicate possible CSRF attack(is (str/includes? (str (:body response)) "CSRF"))))))))-(deftest happy-path-callback-test+(deftest ^:synchronized happy-path-callback-test(testing "successful callback with valid code and state"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -202,7 +207,7 @@;;; -------------------------------------------------- Link-Only Mode Tests ---------------------------------------------------(deftest link-only-mode-requires-session-test+(deftest ^:synchronized link-only-mode-requires-session-test(testing "link-only mode should require authenticated session for initial request"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -213,7 +218,7 @@:redirect default-redirect-uri)](is (str/includes? (str (get response :body)) "authenticated session"))))))))-(deftest link-only-mode-with-session-test+(deftest ^:synchronized link-only-mode-with-session-test(testing "link-only mode should work with authenticated session"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -241,7 +246,7 @@;;; -------------------------------------------------- User Provisioning Tests ---------------------------------------------------(deftest create-new-account-test+(deftest ^:synchronized create-new-account-test(testing "A new account will be created for a Slack user we haven't seen before"(with-test-encryption!(sso.test-helpers/with-slack-default-setup!@@ -290,7 +295,7 @@(get-in (mt/latest-audit-log-entry :user-invited (:id new-user))[:details :email])))))))))))))-(deftest create-new-slack-user-no-user-provisioning-test+(deftest ^:synchronized create-new-slack-user-no-user-provisioning-test(testing "When user provisioning is disabled, throw an error if we attempt to create a new user."(with-test-encryption!(sso.test-helpers/with-slack-default-setup!diff --git a/test/metabase/util/encryption_test.clj b/test/metabase/util/encryption_test.cljindex df25264a9a0..ccaf4d84a8b 100644--- a/test/metabase/util/encryption_test.clj+++ b/test/metabase/util/encryption_test.clj@@ -96,16 +96,57 @@(is (empty? (messages))))))(def ^:private fake-ciphertext- "AES+CBC's block size is 16 bytes and the tag length is 32 bytes. This is a string of characters that is the same- length as would be expected for something that has been encrypted, but it is not encrypted, just unlucky enough to- have the same size"- (apply str (repeat 64 "a")))+ "AES+CBC's block size is 16 bytes and the tag length is 32 bytes, so the shortest possible ciphertext is 64 bytes+ (IV + one block + tag). This base64 string decodes to 64 bytes: the size of something that has been encrypted, but+ it is not encrypted, just unlucky enough to have the same size"+ (str (apply str (repeat 86 "a")) "=="))(deftest ^:parallel possibly-encrypted-test(testing "a value shaped like ciphertext but that cannot be decrypted with the current key throws rather than being returned as-is"(is (thrown? Throwable(encryption/maybe-decrypt-accepting-plaintext secret fake-ciphertext)))))+(deftest ^:parallel decryptable-string-test+ (testing "true only for ciphertext that decrypts with the given key"+ (let [ciphertext (encryption/encrypt secret "WOW")]+ (is (true? (encryption/decryptable-string? secret ciphertext)))+ (testing "false for ciphertext under another key"+ (is (false? (encryption/decryptable-string? secret-2 ciphertext))))+ (testing "false with no key, even for genuine ciphertext"+ (is (false? (encryption/decryptable-string? nil ciphertext))))))+ (testing "false for anything that is not ciphertext"+ (is (false? (encryption/decryptable-string? secret "WOW")))+ (is (false? (encryption/decryptable-string? secret "")))+ (is (false? (encryption/decryptable-string? secret nil)))+ (testing "including plaintext shaped like ciphertext, which `possibly-encrypted-string?` cannot tell apart"+ (is (true? (encryption/possibly-encrypted-string? fake-ciphertext)))+ (is (false? (encryption/decryptable-string? secret fake-ciphertext))))))++(deftest ^:parallel decryptable-bytes-test+ (let [plaintext (codecs/to-bytes "WOW")+ ciphertext (encryption/encrypt-bytes secret plaintext)+ fake-bytes (byte-array 64)]+ (testing "true only for ciphertext that decrypts with the given key"+ (is (true? (encryption/decryptable-bytes? secret ciphertext)))+ (is (false? (encryption/decryptable-bytes? secret-2 ciphertext)))+ (is (false? (encryption/decryptable-bytes? nil ciphertext))))+ (testing "false for anything that is not ciphertext"+ (is (false? (encryption/decryptable-bytes? secret plaintext)))+ (is (false? (encryption/decryptable-bytes? secret (byte-array 0))))+ (is (false? (encryption/decryptable-bytes? secret nil)))+ (testing "including bytes shaped like ciphertext"+ (is (true? (encryption/possibly-encrypted-bytes? fake-bytes)))+ (is (false? (encryption/decryptable-bytes? secret fake-bytes)))))))++(deftest ^:parallel possibly-encrypted-returns-booleans-test+ (testing "the shape checks never return nil"+ (is (false? (encryption/possibly-encrypted-string? nil)))+ (is (false? (encryption/possibly-encrypted-string? "")))+ (is (false? (encryption/possibly-encrypted-string? "not base64!")))+ (is (false? (encryption/possibly-encrypted-bytes? nil)))+ (is (false? (encryption/possibly-encrypted-bytes? (byte-array 3))))+ (is (true? (encryption/possibly-encrypted-string? (encryption/encrypt secret "WOW"))))))+(deftest ^:parallel maybe-decrypt-strict-test(testing "strict `maybe-decrypt`"(testing "decrypts a genuinely encrypted value"diff --git a/test_resources/error-migration.yaml b/test_resources/error-migration.yamlindex 14e8bb25947..dd1c45ea346 100644--- a/test_resources/error-migration.yaml+++ b/test_resources/error-migration.yaml@@ -3,6 +3,24 @@# env variable as it will still consider this migration as pending. Migrations that have an expected chance of# failure should generally use preConditions to detect these cases and mark the migration as ran.databaseChangeLog:+ # The startup encryption check reads the encryption-check setting and treats an unreadable one as invalid, so give+ # it a setting table to read (no row: the database is simply not encrypted).+ - changeSet:+ id: '0'+ author: noahmoss+ changes:+ - createTable:+ tableName: setting+ columns:+ - column:+ name: key+ type: varchar(254)+ constraints:+ primaryKey: true+ - column:+ name: value+ type: text+- changeSet:id: '1'author: noahmoss