Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

docs: python transforms (#64518)

Alex Yarosh371b2c2081f1
diff --git a/docs/data-modeling/python-transforms.md b/docs/data-modeling/python-transforms.md
new file mode 100644
index 00000000000..b44760734ec
--- /dev/null
+++ b/docs/data-modeling/python-transforms.md
@@ -0,0 +1,85 @@
+---
+title: Python transforms
+summary: Use Python to wrangle your data in Metabase and write the results back to your database.
+---
+
+# Python transforms
+
+> Python transforms are only available on Metabase Cloud [Pro and Enterprise plans](https://www.metabase.com/pricing) with the **Python execution add-on**.
+
+You can write a Python script to transform the data from your database, and write the result back into the database as another table.
+
+For general information about Metabase transforms, see [Transforms](transforms.md).
+
+## Create a Python transform
+
+To write Python transforms, you'll need to have the Python execution add-on. Once you've enabled the add-on:
+
+1. Go to **Admin settings > Transforms** and click **Create a transform > Python script**.
+
+2. Select a database that has the data you want to transform. See [Databases that support transforms](transforms.md#databases-that-support-transforms).
+
+3. Select one or more tables with the data that you'd like to transform. Optionally, assign aliases to the tables.
+
+ The tables you select will be available as DataFrames in your Python code with your chosen aliases, and will be passed to the `transform()` function as parameters.
+
+4. Create a function `transform()` that does the data wrangling and returns a [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). The DataFrame returned by the function will be written back to your database when the transform is run.
+5. Click **Save** in the top right corner.
+6. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.
+
+ You can only write back to the same database as you chose for the transform source.
+
+7. Optionally, assign tags to your transforms. Tags are used by [jobs](transforms.md#jobs-and-tags) to run transforms on schedule.
+
+## Tips for Python transforms
+
+- A Python transform must have a function `transform()` that returns a single `pandas` DataFrame.
+- You can use aliases to include tables from your database as DataFrames inside the `transform()` function. The tables will _only_ be available in the transform function. Other functions won't have access to the tables.
+- Only `pandas` will be imported by default, but you can import [certain other packages](#available-python-packages). You can also use functions from the [common library](#common-python-library).
+- You'll see the output of `print()` statements in the transform's logs.
+- Metabase won't write DataFrame indexes to the database, including indexes created by `groupby()`. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function.
+
+## Run a Python transform
+
+See [Run a transform](transforms.md#run-a-transform). You'll see logs for a transform run (including the output of any `print()` statements you included in the `transform()` code) on the transform's page.
+
+## Available Python packages
+
+You can import any of the following Python packages in Python transforms:
+
+- [`pandas`](https://pandas.pydata.org/);
+- Any dependencies of `pandas`, e.g. [`numpy`](https://numpy.org/);
+- Any packages from the [Python standard library](https://docs.python.org/3/library/index.html), e.g. `json` or `datetime`.
+
+Due to security considerations for the Python execution environment, you won't be able to install or import any other packages.
+
+You can also write your own functions and add them to the [common library](#common-python-library) in Metabase, which will be available across all your Python transforms.
+
+## Common Python library
+
+If you have functions or classes you'd like to reuse across multiple transforms, you can add them to the common library that will be available across all Python transforms.
+
+To add things to the common Python library:
+
+1. Go to **Admin > Transforms > Python library**.
+
+2. Add a Python function or class.
+
+ Functions in this library can't access any data in your database.
+
+To use functions or classes from your Python library:
+
+1. When editing your Python transform script, check **Import common library** in the top right.
+
+ This will add an aptly named import `import common` to the transform's code.
+
+2. Reference functions or classes from the common library in your code like `common.manifest_kittens()`.
+
+## Current limitations of Python transforms
+
+- Currently, Python transforms are only available on Metabase Cloud.
+- The transform function must return a single `pandas` DataFrame. Other data manipulation and DataFrame libraries like `polars` or `pyspark` are not supported.
+- DataFrame indexes, including indexes created by `groupby()`, are ignored from writing back to the database. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function to make the index into a real column.
+- You can't preview the results of a Python transform.
+- Only a [limited set of packages](#available-python-packages) are available for import. You can't install additional packages.
+- Because Python transforms use `pandas`, all data manipulation is done in memory. The available memory is determined by the Python execution add-on. For large datasets, consider using [query-based transforms](transforms.md#query-based-transforms) that run in your database.
diff --git a/docs/data-modeling/transforms.md b/docs/data-modeling/transforms.md
index 5bacf9d04ad..42b452a098c 100644
--- a/docs/data-modeling/transforms.md
+++ b/docs/data-modeling/transforms.md
@@ -11,11 +11,11 @@ _Admin settings > Transforms_
Transforms can be used to, well, transform your data - do stuff like preprocessing, cleaning, joining tables, pre-computing metrics. Transforms give you the ability to do the "T" of "ETL" within Metabase.
-You'll write a query in Metabase, a transform will run this query, create a table in your target database containing the results, and sync that table to Metabase, so it can be used as a data source for questions or other transforms.
+You'll write a query or a Python script in Metabase, a transform will run this query or script, create a table in your target database containing the results, and sync that table to Metabase, so it can be used as a data source for questions or other transforms.
## Transforms overview
-- **Transforms** are queries that write back to your database as a new, persistent table. Use transforms to clean, join, or pre-aggregate data.
+- **Transforms** are queries (either with SQL or the query builder) or Python scripts that write back to your database as a new, persistent table. Use transforms to clean, join, or pre-aggregate data.
- Transforms are scheduled and organized using **tags** and **jobs**.
- You assign tags (e.g., daily, hourly) to group your transforms.
- A job runs on a schedule (e.g., every day at midnight) and executes all transforms that have been assigned a specific tag.
@@ -37,6 +37,33 @@ You can't create transforms on databases that have [Database routing](../permiss
Transforms will create tables in your database, so the database user you use for your connection must have appropriate privileges. See [Database users, roles, and privileges](../databases/users-roles-privileges.md).
+## Types of transforms
+
+Metabase supports two types of transforms: query-based transforms and Python transforms. Here's a high-level overview of how they work and the differences between them:
+
+### Query-based transforms
+
+- In Metabase, you create a `SELECT` query either using SQL or Metabase's [graphical query builder](../questions/query-builder/editor.md).
+- When the transform first runs, your _database_ executes the transform's query.
+- Your database writes the results of the query as a new table.
+- The new table is synced to Metabase.
+- On subsequent transform runs, your database will overwrite that table with the updated results (updates are not incremental).
+
+### Python-based transforms
+
+Python-based transforms require a dedicated Python execution environment, so you'll need to buy the Python execution add-on.
+
+- In your Metabase, you write a Python script that returns a `pandas` DataFrame and uses one or more tables from your database.
+- When Metabase runs the transform, a new Python execution environment is spun up. Python transforms run in a separate, isolated environment—not on your Metabase instance.
+- Metabase securely copies your source data to your Python environment and makes it available as pandas DataFrames.
+- The Python environment executes your Python script _in memory_.
+- The Python environment saves the resulting DataFrame as a file.
+- Your Metabase instance reads the file and writes the results to a new table in your database.
+- The new table is synced to Metabase.
+- On subsequent transform runs, your database will overwrite that table with the updated results (updates are not incremental).
+
+For more on Python transforms, see [Python transforms](python-transforms.md).
+
## Create a transform
Only admins can create transforms.
@@ -45,20 +72,24 @@ To create a transform:
1. Go to Admin settings > Transforms.
2. Click **+ Create transform**.
-3. Select the source for your transform. You can create transforms using the query builder, SQL editor, or by copying an existing question or model.
+3. Select the source for your transform.
+
+ You can create transforms using the query builder, SQL editor, or by copying an existing question or model. On Metabase Cloud with the Python execution add-on, you can also [create transforms from a Python script](python-transforms.md). See [Types of transforms](#types-of-transforms).
- If you select a question for the transform's query, Metabase will only copy the question's query. Later edits to that question won't affect the transform's query.
+ If you select a question for the transform's source, Metabase will only _copy_ the question's query. Later edits to that question won't affect the transform's query.
- Your transform can query the target tables of other transforms.
+ Your transform can use the target tables of other transforms.
-4. Edit the query in the query builder or the SQL editor.
+4. Edit the query or the Python code for the transform.
In SQL transforms, you can reference other saved questions and use snippets, but you can't use SQL parameters.
- When editing the query, you can run the query to preview the results. Previewing query results won't write data into your database.
+ When editing query-based transforms, you can preview results without writing data into your database.
+
+ For editing Python transforms, see [Python transforms](python-transforms.md).
5. Click **Save** in the top right corner.
-6. Select a target schema for your transform and enter a name for the target table. Metabase will write create the table the results of the transform query into this table.
+6. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.
You can only transform data _within_ a database; you can't write from one database to another.
@@ -66,15 +97,15 @@ To create a transform:
## Edit a transform
-You can edit the transform's name and description, query, and target table. You can edit the transform even if it already ran or is scheduled to run.
+You can edit the transform's name and description, query or script, and target table. You can edit the transform even if it already ran or is scheduled to run.
-### Edit transform's query
+### Edit transform's query or script
-To edit the transform's query, click on "Edit query" above the query definition.
+To edit the transform's query or script, click on "Edit query" or "Edit script" above the transform definition.
-Currently, you can't convert a transform built with the query builder into a SQL-based transform. If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.
+Currently, you can't convert between different transform types (e.g. a query builder transform to a SQL-based transform, or a SQL transform into a Python transform). If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.
-Once you change the transform's query, the next transform run (manual or scheduled) will use the updated query and write the results into the target table. If you have questions that query the transform's target table, they might break. For example, if your new transform query no longer includes a column that a downstream question was relying on, that question will break.
+Once you change the transform's query or script, the next transform run (manual or scheduled) will use the updated query and write the results into the target table. If you have questions that query the transform's target table, they might break. For example, if your new transform query no longer includes a column that a downstream question was relying on, that question will break.
### Edit transform's target
@@ -96,6 +127,8 @@ To schedule a transform, you'll need to assign one or more tags to it, then crea
You can see the time and status of the latest transform run on the transform's page, or in the Runs view. The time of the run is given in the system's timezone.
+For Python transforms, you'll also see the transform's execution logs.
+
## Transform dependencies
Transform queries can use the data from other transforms. For example, you can have a transform that uses data from a `raw_events` table and writes to a `stg_events` table, and then create another transform that uses data from the `stg_events` table and writes to an `events` table.
@@ -138,5 +171,6 @@ Transforms are similar to models with model persistence turned on, but there are
- You can choose the target schema and tables for transforms. Model persistence will create its own schema and tables.
- You can't rename columns in a transform (but you can change column metadata once they're synced back to Metabase). You can rename columns in model metadata and then persist those column names using model persistence.
- Transforms support more databases than model persistence
+- On Metabase Cloud with Python execution add-on, you can use Python to create transforms.
Use models to enable non-admins to create their own datasets within Metabase, and to add context like field descriptions and semantic types. Use transforms to create persisted datasets in your database and reuse them across Metabase.