Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Revert "remove transforms once again (#67809)" (#67945)

Alex Yarosh4a2fade2265b
diff --git a/docs/data-modeling/python-transforms.md b/docs/data-modeling/python-transforms.md
new file mode 100644
index 00000000000..b44760734ec
--- /dev/null
+++ b/docs/data-modeling/python-transforms.md
@@ -0,0 +1,85 @@
+---
+title: Python transforms
+summary: Use Python to wrangle your data in Metabase and write the results back to your database.
+---
+
+# Python transforms
+
+> Python transforms are only available on Metabase Cloud [Pro and Enterprise plans](https://www.metabase.com/pricing) with the **Python execution add-on**.
+
+You can write a Python script to transform the data from your database, and write the result back into the database as another table.
+
+For general information about Metabase transforms, see [Transforms](transforms.md).
+
+## Create a Python transform
+
+To write Python transforms, you'll need to have the Python execution add-on. Once you've enabled the add-on:
+
+1. Go to **Admin settings > Transforms** and click **Create a transform > Python script**.
+
+2. Select a database that has the data you want to transform. See [Databases that support transforms](transforms.md#databases-that-support-transforms).
+
+3. Select one or more tables with the data that you'd like to transform. Optionally, assign aliases to the tables.
+
+ The tables you select will be available as DataFrames in your Python code with your chosen aliases, and will be passed to the `transform()` function as parameters.
+
+4. Create a function `transform()` that does the data wrangling and returns a [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). The DataFrame returned by the function will be written back to your database when the transform is run.
+5. Click **Save** in the top right corner.
+6. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.
+
+ You can only write back to the same database as you chose for the transform source.
+
+7. Optionally, assign tags to your transforms. Tags are used by [jobs](transforms.md#jobs-and-tags) to run transforms on schedule.
+
+## Tips for Python transforms
+
+- A Python transform must have a function `transform()` that returns a single `pandas` DataFrame.
+- You can use aliases to include tables from your database as DataFrames inside the `transform()` function. The tables will _only_ be available in the transform function. Other functions won't have access to the tables.
+- Only `pandas` will be imported by default, but you can import [certain other packages](#available-python-packages). You can also use functions from the [common library](#common-python-library).
+- You'll see the output of `print()` statements in the transform's logs.
+- Metabase won't write DataFrame indexes to the database, including indexes created by `groupby()`. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function.
+
+## Run a Python transform
+
+See [Run a transform](transforms.md#run-a-transform). You'll see logs for a transform run (including the output of any `print()` statements you included in the `transform()` code) on the transform's page.
+
+## Available Python packages
+
+You can import any of the following Python packages in Python transforms:
+
+- [`pandas`](https://pandas.pydata.org/);
+- Any dependencies of `pandas`, e.g. [`numpy`](https://numpy.org/);
+- Any packages from the [Python standard library](https://docs.python.org/3/library/index.html), e.g. `json` or `datetime`.
+
+Due to security considerations for the Python execution environment, you won't be able to install or import any other packages.
+
+You can also write your own functions and add them to the [common library](#common-python-library) in Metabase, which will be available across all your Python transforms.
+
+## Common Python library
+
+If you have functions or classes you'd like to reuse across multiple transforms, you can add them to the common library that will be available across all Python transforms.
+
+To add things to the common Python library:
+
+1. Go to **Admin > Transforms > Python library**.
+
+2. Add a Python function or class.
+
+ Functions in this library can't access any data in your database.
+
+To use functions or classes from your Python library:
+
+1. When editing your Python transform script, check **Import common library** in the top right.
+
+ This will add an aptly named import `import common` to the transform's code.
+
+2. Reference functions or classes from the common library in your code like `common.manifest_kittens()`.
+
+## Current limitations of Python transforms
+
+- Currently, Python transforms are only available on Metabase Cloud.
+- The transform function must return a single `pandas` DataFrame. Other data manipulation and DataFrame libraries like `polars` or `pyspark` are not supported.
+- DataFrame indexes, including indexes created by `groupby()`, are ignored from writing back to the database. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function to make the index into a real column.
+- You can't preview the results of a Python transform.
+- Only a [limited set of packages](#available-python-packages) are available for import. You can't install additional packages.
+- Because Python transforms use `pandas`, all data manipulation is done in memory. The available memory is determined by the Python execution add-on. For large datasets, consider using [query-based transforms](transforms.md#query-based-transforms) that run in your database.
diff --git a/docs/data-modeling/transforms.md b/docs/data-modeling/transforms.md
new file mode 100644
index 00000000000..1ab45b70120
--- /dev/null
+++ b/docs/data-modeling/transforms.md
@@ -0,0 +1,176 @@
+---
+title: Transforms
+summary: Transforms allow you to wrangle your data in Metabase, write the query results back to your database, and reuse them in Metabase as sources for new queries.
+---
+
+# Transforms
+
+{% include plans-blockquote.html feature="Transforms" is_plural=true%}
+
+_Admin settings > Transforms_
+
+Transforms can be used to, well, transform your data - do stuff like preprocessing, cleaning, joining tables, pre-computing metrics. Transforms give you the ability to do the "T" of "ETL" within Metabase.
+
+You'll write a query or a Python script in Metabase, a transform will run this query or script, create a table in your target database containing the results, and sync that table to Metabase, so it can be used as a data source for questions or other transforms.
+
+## Transforms overview
+
+- **Transforms** are queries (either with SQL or the query builder) or Python scripts that write back to your database as a new, persistent table. Use transforms to clean, join, or pre-aggregate data.
+- Transforms are scheduled and organized using **tags** and **jobs**.
+ - You assign tags (e.g., daily, hourly) to group your transforms.
+ - A job runs on a schedule (e.g., every day at midnight) and executes all transforms that have been assigned a specific tag.
+- Each execution of a transform is a **run**. A run replaces the target table with fresh results. You can review the history of runs to monitor their success or failure.
+
+## Databases that support transforms
+
+Currently, Metabase can create transforms on the following databases:
+
+- BigQuery
+- ClickHouse (only ClickHouse Cloud)
+- MySQL/MariaDB
+- PostgreSQL
+- Redshift
+- Snowflake
+- SQL Server
+
+You can't create transforms on databases that have [Database routing](../permissions/database-routing.md) enabled, on Metabase [Cloud Storage](../cloud/storage.md), or on Metabase's Sample Database.
+
+Transforms will create tables in your database, so the database user you use for your connection must have appropriate privileges. See [Database users, roles, and privileges](../databases/users-roles-privileges.md).
+
+## Types of transforms
+
+Metabase supports two types of transforms: query-based transforms and Python transforms. Here's a high-level overview of how they work and the differences between them:
+
+### Query-based transforms
+
+- In Metabase, you create a `SELECT` query either using SQL or Metabase's [graphical query builder](../questions/query-builder/editor.md).
+- When the transform first runs, your _database_ executes the transform's query.
+- Your database writes the results of the query as a new table.
+- The new table is synced to Metabase.
+- On subsequent transform runs, your database will overwrite that table with the updated results (updates are not incremental).
+
+### Python-based transforms
+
+Python-based transforms require a dedicated Python execution environment, so you'll need to buy the Python execution add-on.
+
+- In your Metabase, you write a Python script that returns a `pandas` DataFrame and uses one or more tables from your database.
+- When Metabase runs the transform, a new Python execution environment is spun up. Python transforms run in a separate, isolated environment—not on your Metabase instance.
+- Metabase securely copies your source data to your Python environment and makes it available as pandas DataFrames.
+- The Python environment executes your Python script _in memory_.
+- The Python environment saves the resulting DataFrame as a file.
+- Your Metabase instance reads the file and writes the results to a new table in your database.
+- The new table is synced to Metabase.
+- On subsequent transform runs, your database will overwrite that table with the updated results (updates are not incremental).
+
+For more on Python transforms, see [Python transforms](python-transforms.md).
+
+## Create a transform
+
+Only admins can create transforms.
+
+To create a transform:
+
+1. Go to Admin settings > Transforms.
+2. Click **+ Create transform**.
+3. Select the source for your transform.
+
+ You can create transforms using the query builder, SQL editor, or by copying an existing question or model. On Metabase Cloud with the Python execution add-on, you can also [create transforms from a Python script](python-transforms.md). See [Types of transforms](#types-of-transforms).
+
+ If you select a question for the transform's source, Metabase will only _copy_ the question's query. Later edits to that question won't affect the transform's query.
+
+ Your transform can use the target tables of other transforms.
+
+4. Edit the query or the Python code for the transform.
+
+ In SQL transforms, you can reference other saved questions and use snippets, but you can't use SQL parameters.
+
+ When editing query-based transforms, you can preview results without writing data into your database.
+
+ For editing Python transforms, see [Python transforms](python-transforms.md).
+
+5. Click **Save** in the top right corner.
+6. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.
+
+ You can only transform data _within_ a database; you can't write from one database to another.
+
+7. Optionally, assign tags to your transforms. Tags are used by [jobs](#jobs-and-tags) to run transforms on schedule.
+
+## Edit a transform
+
+You can edit the transform's name and description, query or script, and target table. You can edit the transform even if it already ran or is scheduled to run.
+
+### Edit transform's query or script
+
+To edit the transform's query or script, click on "Edit query" or "Edit script" above the transform definition.
+
+Currently, you can't convert between different transform types (e.g. a query builder transform to a SQL-based transform, or a SQL transform into a Python transform). If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.
+
+Once you change the transform's query or script, the next transform run (manual or scheduled) will use the updated query and write the results into the target table. If you have questions that query the transform's target table, they might break. For example, if your new transform query no longer includes a column that a downstream question was relying on, that question will break.
+
+### Edit transform's target
+
+To edit transform's target table, i.e., the table where the query results are written, click on "Change target". You'll need to select whether you want to keep the old target table, or delete it. Deletion can't be undone.
+
+### Edit transform's name or description
+
+To edit a transform's name or description, click on the name or description at the top of the transform page.
+
+## Run a transform
+
+You can run a transform manually or on schedule (e.g., hourly).
+
+Running a transform for the first time will create and sync the table created by the transform, and you'll be able to edit the table's [metadata](./metadata-editing.md) and [permissions](../permissions/data.md). Subsequent runs will drop and recreate the table. Currently, Metabase doesn't support incremental transforms. Each transform run will recreate the target table using the entirety of the query results.
+
+To run a transform manually, visit the transform and click **Run transform**.
+
+To schedule a transform, you'll need to assign one or more tags to it, then create a [scheduled job](#jobs-and-tags) that picks up those tags.
+
+You can see the time and status of the latest transform run on the transform's page, or in the Runs view. The time of the run is given in the system's timezone.
+
+For Python transforms, you'll also see the transform's execution logs.
+
+## Transform dependencies
+
+Transform queries can use the data from other transforms. For example, you can have a transform that uses data from a `raw_events` table and writes to a `stg_events` table, and then create another transform that uses data from the `stg_events` table and writes to an `events` table.
+
+Metabase will track transform dependencies, and execute them in order. For each transform, you can see which transforms the current transform depends on.
+
+If a [job](#jobs-and-tags) includes a transform that depends on a table created by another transform, then the job will run all the tagged transforms and their dependencies, even if they lack tags.
+
+## Jobs and tags
+
+Jobs let you run multiple transforms on a schedule.
+
+You can see and filter the list of all jobs, their last and next run, and the status of the least run, in _Admin settings > Transforms > Jobs_
+
+To create a new job, go to _Admin settings > Transforms_, switch to the **Jobs** view, and click **Create a job**
+
+To edit a job, click on the job in the list of jobs.
+
+Jobs have two components: schedule and tags.
+
+Metabase uses tags to locate transforms that should be run by the job, and then schedules those transforms according to the cron schedule. The times are given in the system's timezone.
+
+### Tags
+
+Job can use multiple tags, in which case, the job will run all transforms that have _any_ of those tags. For example, you can have a job "Weekend job" that is scheduled run at noon on Saturdays and Sundays that picks up all transforms tagged either "Saturday", "Sunday", or "Weekend".
+
+By default, Metabase comes with hourly, daily, weekly, and monthly tags and jobs that are run on the corresponding schedules, but you can remove or rename those tags, or create new tags.
+
+To create a new tag, just type the new tag's name in "Tags" field (either when viewing a transform or when viewing a job) and select "Create a tag".
+
+### Jobs take dependencies into account
+
+Depended transforms will be scheduled and run intelligently: if Transform B depends on the output of Transform A, then a job will run Transform A before Transform B. A job will run all dependent transforms even if the dependencies aren't tagged. You will see the order of transform execution on the job's page.
+
+## Transforms vs models
+
+Transforms are similar to models with model persistence turned on, but there are a few crucial differences:
+
+- Transforms can only be created by admins. Models can be created by anyone with permissions to create queries on the data source (but only admins can enable model persistence).
+- You can choose the target schema and tables for transforms. Model persistence will create its own schema and tables.
+- You can't rename columns in a transform (but you can change column metadata once they're synced back to Metabase). You can rename columns in model metadata and then persist those column names using model persistence.
+- Transforms support more databases than model persistence
+- On Metabase Cloud with Python execution add-on, you can use Python to create transforms.
+
+Use models to enable non-admins to create their own datasets within Metabase, and to add context like field descriptions and semantic types. Use transforms to create persisted datasets in your database and reuse them across Metabase.
diff --git a/docs/databases/users-roles-privileges.md b/docs/databases/users-roles-privileges.md
index b576b33a7ef..2bb8d6f6c62 100644
--- a/docs/databases/users-roles-privileges.md
+++ b/docs/databases/users-roles-privileges.md
@@ -9,6 +9,7 @@ We recommend creating a `metabase` database user with the following database rol
- [`analytics` for read access](#minimum-database-privileges) to any schemas or tables used for analysis.
- Optional [`metabase_actions` for write access](#privileges-to-enable-actions-and-editable-table-data) to tables used for Metabase actions.
- Optional [`metabase_model_persistence` for write access](#privileges-to-enable-model-persistence) to the schema used for Metabase model persistence.
+- Optional [`metabase_transforms` for write access](#privileges-to-enable-transforms) to the schema used for Metabase transforms.
Bundling your privileges into roles based on use cases makes it easier to manage privileges in the future (especially in [multi-tenant situations](#multi-tenant-permissions)). For example, you could:
@@ -134,6 +135,19 @@ GRANT INSERT, UPDATE, DELETE ON "your_model's_table" IN SCHEMA "your_schema" TO
GRANT metabase_model_persistence TO metabase;
```
+## Privileges to enable transforms
+
+[Transforms](../data-modeling/transforms.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables.
+
+In addition to the [minimum database privileges](#minimum-database-privileges):
+
+- Create a new role called `metabase_transforms`.
+- Create a dedicated schema for transforms.
+- Give the `metabase_transforms` role `CREATE TABLE` / `ALTER` / `DROP` privileges for the specified schema
+- Give the `metabase_transforms` role to the `metabase` user.
+
+In certain cases, Metabase will give you an option to create new schemas for transforms, in which case you'll also need `CREATE SCHEMA` privileges for your metabase users (or `CREATE DATABASE` for ClickHouse).
+
## Privileges to enable uploads
You can [upload CSVs](../databases/uploads.md) to supported databases. Metabase's database user should have write access (`INSERT`, `UPDATE`, `DELETE`) to the schema where you want to store the uploads.
diff --git a/docs/installation-and-operation/serialization.md b/docs/installation-and-operation/serialization.md
index a16a8e3fa9c..3b4083bb9d6 100644
--- a/docs/installation-and-operation/serialization.md
+++ b/docs/installation-and-operation/serialization.md
@@ -50,6 +50,7 @@ Metabase will only export the following entities:
- Collections (but personal collections don't get exported unless explicitly specified them through [export options](#customize-what-gets-exported))
- Dashboards
- Saved questions
+- Transforms (including jobs)
- Documents (without comments)
- Actions
- Models
@@ -631,53 +632,53 @@ tar -xvf metabase_data.tgz
1. Send a `curl` request to export data:
- ```sh
- curl \
- -H 'x-api-key: YOUR_API_KEY' \
- -X POST 'https://your-metabase-url/api/ee/serialization/export' \
- -o metabase_data.tgz
- ```
+ ```sh
+ curl \
+ -H 'x-api-key: YOUR_API_KEY' \
+ -X POST 'https://your-metabase-url/api/ee/serialization/export' \
+ -o metabase_data.tgz
+ ```
- substituting `YOUR_API_KEY` with your API key and `your-metabase-url` with the URL of your Metabase instance.
+ substituting `YOUR_API_KEY` with your API key and `your-metabase-url` with the URL of your Metabase instance.
- > We use `POST`, not `GET`, for the `/export` endpoint.
-
- This command will download the files as a GZIP-compressed Tar file named `metabase_data.tgz`.
+ > We use `POST`, not `GET`, for the `/export` endpoint.
+
+ This command will download the files as a GZIP-compressed Tar file named `metabase_data.tgz`.
2. Unzip the compressed file:
- ```sh
- tar -xvf metabase_data.tgz
- ```
-
- The extracted directory will be called something like `metabase-yyyy-MM-dd_HH-mm`, with the date and time of the export.
+ ```sh
+ tar -xvf metabase_data.tgz
+ ```
+
+ The extracted directory will be called something like `metabase-yyyy-MM-dd_HH-mm`, with the date and time of the export.
### Step 3: Import
1. Compress the directory containing serialized Metabase application data.
- Let's say you have your YAML files with Metabase application data in a directory called `metabase_data`. Before importing those files to your target Metabase, you'll need to compress those files.
-
- ```sh
- tar -czf metabase_data.tgz metabase_data
- ```
+ Let's say you have your YAML files with Metabase application data in a directory called `metabase_data`. Before importing those files to your target Metabase, you'll need to compress those files.
+
+ ```sh
+ tar -czf metabase_data.tgz metabase_data
+ ```
2. POST to `/api/ee/serialization/import`.
- From the directory where you've stored your GZIP-compressed file, run:
-
- ```sh
- curl -X POST \
- -H 'x-api-key: YOUR_API_KEY' \
- -F 'file=@metabase_data.tgz' \
- 'https://your-metabase-url/api/ee/serialization/import' \
- -o -
- ```
+ From the directory where you've stored your GZIP-compressed file, run:
+
+ ```sh
+ curl -X POST \
+ -H 'x-api-key: YOUR_API_KEY' \
+ -F 'file=@metabase_data.tgz' \
+ 'https://your-metabase-url/api/ee/serialization/import' \
+ -o -
+ ```
- substituting `YOUR_API_KEY` with your API key and `your-metabase-url` with your Metabase instance URL.
- The `-o -` option will output logs in the terminal.
+ substituting `YOUR_API_KEY` with your API key and `your-metabase-url` with your Metabase instance URL.
+ The `-o -` option will output logs in the terminal.
- > If you import Metabase data into the same Metabase as you exported it from, you will overwrite your existing questions, dashboards, etc. See [How import works](#how-import-works).
+ > If you import Metabase data into the same Metabase as you exported it from, you will overwrite your existing questions, dashboards, etc. See [How import works](#how-import-works).
## Other uses of serialization
diff --git a/docs/permissions/row-and-column-security.md b/docs/permissions/row-and-column-security.md
index 5573b8637f5..6f4aaebde57 100644
--- a/docs/permissions/row-and-column-security.md
+++ b/docs/permissions/row-and-column-security.md
@@ -195,7 +195,7 @@ Learn more about [SQL parameters](../questions/native-editor/sql-parameters.md)
For example, say have a table like this:
| User_ID | Value |
-| ------- | ----- |
+|---------|-------|
| 1 | 10 |
| 1 | 50 |
| 2 | 5 |
@@ -203,7 +203,7 @@ For example, say have a table like this:
| 3 | 5 |
| 3 | 5 |
-If you want to give someone access to multiple user IDs (e.g., the person should see rows for both `User_ID` 1 and 2.), you can set up a user attribute, like `user_id` that can handle comma-separated values like "1,2".
+If you want to give someone access to multiple user IDs (e.g., the person should see rows for both `User_ID` 1 and 2.), you can set up a user attribute, like `user_id` that can handle comma-separated values like "1,2".
1. Create a SQL question that parses the comma-separated string and filters the table:
@@ -313,6 +313,10 @@ Create a SQL question that casts the advanced data type column to a basic data t
If you can't use SQL casting in Metabase, create a view in your database that converts the advanced data type to a basic type, then set up row and column security on that view instead of the original table. You'll also need to block the original table.
+#### Option 3: Use transforms
+
+ Use a [transform](../data-modeling/transforms.md) to create a table that casts the advanced data type to a basic type. Then set up row and column security on the transformed table instead. You'll also need to block the original table.
+
## Further reading
- [Row and column security examples](./row-and-column-security-examples.md)