Skip to content

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Edits on docs about how to run python transforms in self hosted deployments (#78098)

da5854e3e83ff4
diff --git a/docs/data-studio/transforms/addons.md b/docs/data-studio/transforms/addons.md
index 906a217c852..00e8a3bf614 100644
--- a/docs/data-studio/transforms/addons.md
+++ b/docs/data-studio/transforms/addons.md
@@ -37,7 +37,7 @@ With basic transforms, you can:
- **Metabase Cloud**: Basic transform functionality on Metabase Cloud - Starter, Pro, or Enterprise - comes with an additional small fee per successful transform run, see [Pricing](https://www.metabase.com/pricing).
- Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).
+ Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).
### Cancel basic transforms
@@ -65,6 +65,8 @@ The Advanced transforms add-on comes with an additional charge per successful tr
To enable Advanced transforms functionality, you need to have [Basic transforms](#basic-transforms) already, see [Enable basic transforms](#enable-basic-transforms).
+For the full self-hosted setup see [Set up transforms on a self-hosted Metabase](transforms-overview.md#set-up-transforms-on-a-self-hosted-metabase).
+
There are two ways to enable Advanced transforms:
- **From your Metabase instance**: you can navigate to a feature requiring advanced transforms (like Python transforms or transform inspector), and follow the prompts to upgrade.
diff --git a/docs/data-studio/transforms/python-runner.md b/docs/data-studio/transforms/python-runner.md
index 0bde1cdc1e0..69f7d04f7e2 100644
--- a/docs/data-studio/transforms/python-runner.md
+++ b/docs/data-studio/transforms/python-runner.md
@@ -11,7 +11,7 @@ To run Python transforms from a self-hosted Metabase, you'll need to configure a
## Prerequisites
-- Docker installed and running.
+- Docker installed and running, or an infrastructure that can run containers.
- **Self-hosted Metabase Pro or Enterprise license** (Python Runner requires a Pro/Enterprise plan).
- For production: an S3-compatible storage bucket (AWS S3, MinIO, etc.).
@@ -69,13 +69,13 @@ docker network create metabase-network
docker run -d \
--network metabase-network \
-e AUTH_TOKEN=your-secure-token-here \
- --name python-runner metabase/python-runner:latest
+ --name python-runner --hostname python-runner metabase/python-runner:latest
docker run -d \
--network metabase-network \
-p 3000:3000 \
-e MB_PYTHON_RUNNER_URL=http://python-runner:5000 \
- -e MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here \
+ -e MB_PYTHON_RUNNER_API_TOKEN=<your-secure-token-here> \
-e MB_PYTHON_STORAGE_S_3_ENDPOINT=https://s3.amazonaws.com \
-e MB_PYTHON_STORAGE_S_3_BUCKET=your-bucket-name \
-e MB_PYTHON_STORAGE_S_3_REGION=us-east-1 \
@@ -97,24 +97,27 @@ docker run -d \
These settings can also be configured in the Metabase UI at **Admin** > **Settings** > **Python Runner**. Note that environment variables take precedence over UI settings.
-| Variable | Description |
-| ----------------------------------------- | ---------------------------------------------------------------------------------- |
-| `MB_PYTHON_RUNNER_URL` | URL where Metabase can reach the Python Runner (e.g., `http://python-runner:5000`) |
-| `MB_PYTHON_RUNNER_API_TOKEN` | Authentication token. Must match `AUTH_TOKEN` in the Python Runner. |
-| `MB_PYTHON_STORAGE_S_3_ENDPOINT` | S3 endpoint URL. |
-| `MB_PYTHON_STORAGE_S_3_BUCKET` | S3 bucket name for storing Python artifacts |
-| `MB_PYTHON_STORAGE_S_3_REGION` | AWS region (e.g., `us-east-1`) |
-| `MB_PYTHON_STORAGE_S_3_ACCESS_KEY` | S3 access key |
-| `MB_PYTHON_STORAGE_S_3_SECRET_KEY` | S3 secret key |
-| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |
+| Variable | Description |
+| ------------------------------------------ | ---------------------------------------------------------------------------------- |
+| `MB_PYTHON_RUNNER_URL` | URL where Metabase can reach the Python Runner (e.g., `http://python-runner:5000`) |
+| `MB_PYTHON_RUNNER_API_TOKEN` | Authentication token. Must match `AUTH_TOKEN` in the Python Runner. |
+| `MB_PYTHON_STORAGE_S_3_ENDPOINT` | S3 endpoint URL. |
+| `MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` | S3 endpoint seen from the runner container, if different from the main endpoint. |
+| `MB_PYTHON_STORAGE_S_3_BUCKET` | S3 bucket name for storing Python artifacts |
+| `MB_PYTHON_STORAGE_S_3_REGION` | AWS region (e.g., `us-east-1`) |
+| `MB_PYTHON_STORAGE_S_3_ACCESS_KEY` | S3 access key |
+| `MB_PYTHON_STORAGE_S_3_SECRET_KEY` | S3 secret key |
+| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |
+
+`MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` is the host Metabase signs into the presigned URLs the runner uploads to and downloads from. Set it when the runner resolves storage by a different hostname than Metabase does.
## Using Docker Compose
-Docker Compose simplifies managing multiple containers. Below are example configurations for different scenarios.
+Below are example configurations for different scenarios.
-### Simple setup with LocalStack
+### Self-hosted storage with MinIO
-For simple setup and local testing, you can use LocalStack to simulate S3:
+[MinIO](https://min.io/) is an S3-compatible server you can run yourself, which makes it a common pick for self-hosted Metabase. This Compose file runs Metabase, the Python runner, MinIO, and a short-lived container that creates the bucket Metabase will use — neither MinIO nor Metabase creates that bucket for you.
```yml
name: metabase-python-runner
@@ -125,15 +128,16 @@ services:
- "3000:3000"
environment:
- MB_PYTHON_RUNNER_URL=http://python-runner:5000
- - MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here
- - MB_PYTHON_STORAGE_S_3_ENDPOINT=http://localstack:4566
+ - MB_PYTHON_RUNNER_API_TOKEN=${AUTH_TOKEN}
+ - MB_PYTHON_STORAGE_S_3_ENDPOINT=http://minio:9000
+ - MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT=http://minio:9000
- MB_PYTHON_STORAGE_S_3_BUCKET=metabase-python-runner
- MB_PYTHON_STORAGE_S_3_REGION=us-east-1
- MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS=true
- - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=test
- - MB_PYTHON_STORAGE_S_3_SECRET_KEY=test
+ - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=${MINIO_ROOT_USER}
+ - MB_PYTHON_STORAGE_S_3_SECRET_KEY=${MINIO_ROOT_PASSWORD}
depends_on:
- localstack-init:
+ minio-init:
condition: service_completed_successfully
python-runner:
condition: service_started
@@ -141,77 +145,59 @@ services:
python-runner:
image: metabase/python-runner:latest
environment:
- - AUTH_TOKEN=your-secure-token-here
+ - AUTH_TOKEN=${AUTH_TOKEN}
- localstack:
- image: localstack/localstack:latest
+ minio:
+ image: quay.io/minio/minio:latest
+ command: server /data --console-address ":9001"
environment:
- - SERVICES=s3
- healthcheck:
- test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]
- interval: 5s
- timeout: 5s
- retries: 10
-
- localstack-init:
- image: localstack/localstack:latest
- entrypoint:
- ["/bin/sh", "-c", "awslocal s3 mb s3://metabase-python-runner || true"]
- environment:
- - AWS_ENDPOINT_URL=http://localstack:4566
+ - MINIO_ROOT_USER=${MINIO_ROOT_USER}
+ - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}
+ volumes:
+ - minio-data:/data
+
+ minio-init:
+ image: quay.io/minio/mc:latest
depends_on:
- localstack:
- condition: service_healthy
+ minio:
+ condition: service_started
+ entrypoint:
+ - /bin/sh
+ - -c
+ - |
+ until mc alias set local http://minio:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}" >/dev/null 2>&1; do
+ echo "waiting for minio..."; sleep 2;
+ done
+ mc mb --ignore-existing local/metabase-python-runner
+
+volumes:
+ minio-data: {}
```
-Then run:
+Both S3 endpoint variables point at `http://minio:9000` because Metabase and the runner share the Compose network and reach MinIO by the same hostname. `MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` sets the host Metabase signs into the presigned URLs the runner uploads to and downloads from — if your runner resolves storage by a different hostname than Metabase does, set this to the runner's view of it.
+
+Create a `.env` file:
```bash
-docker compose up -d
+AUTH_TOKEN=your-secure-token-here
+MINIO_ROOT_USER=minioadmin
+MINIO_ROOT_PASSWORD=minioadmin
```
-### Production setup
+Then bring everything up and try a transform:
-For production, use an external S3-compatible storage service:
+1. Put a strong shared secret in `AUTH_TOKEN` (the runner and Metabase have to agree on it):
-```yml
-name: metabase-python-runner
-services:
- metabase:
- image: metabase/metabase-enterprise:latest
- ports:
- - "3000:3000"
- environment:
- - MB_PYTHON_RUNNER_URL=http://python-runner:5000
- - MB_PYTHON_RUNNER_API_TOKEN=${AUTH_TOKEN}
- - MB_PYTHON_STORAGE_S_3_ENDPOINT=${S3_ENDPOINT}
- - MB_PYTHON_STORAGE_S_3_BUCKET=${S3_BUCKET}
- - MB_PYTHON_STORAGE_S_3_REGION=${S3_REGION:-us-east-1}
- - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=${S3_ACCESS_KEY}
- - MB_PYTHON_STORAGE_S_3_SECRET_KEY=${S3_SECRET_KEY}
- depends_on:
- python-runner:
- condition: service_started
-
- python-runner:
- image: metabase/python-runner:latest
- environment:
- - AUTH_TOKEN=${AUTH_TOKEN}
-```
+ ```bash
+ openssl rand -hex 32
+ ```
-Create a `.env` file with your configuration:
+2. Start the stack:
-```bash
-AUTH_TOKEN=your-secure-token-here
-S3_ENDPOINT=https://s3.amazonaws.com
-S3_BUCKET=metabase-python-runner
-S3_REGION=us-east-1
-S3_ACCESS_KEY=your-access-key
-S3_SECRET_KEY=your-secret-key
-```
+ ```bash
+ docker compose up -d
+ ```
-Then run:
+3. You'll need a Pro or Enterprise license with the [Advanced transforms add-on](addons.md). In Metabase, [enable transforms](transforms-overview.md#enable-transforms), then [create a Python transform](python-transforms.md#create-a-python-transform) and click **Run**.
-```bash
-docker compose up -d
-```
+4. Open **Data Studio > Jobs > Runs** and find your run. If it failed, the run's logs will usually tell you whether Metabase couldn't reach the runner, or the runner couldn't reach MinIO.
\ No newline at end of file
diff --git a/docs/data-studio/transforms/transforms-overview.md b/docs/data-studio/transforms/transforms-overview.md
index e750334aaa7..85bb0f05a07 100644
--- a/docs/data-studio/transforms/transforms-overview.md
+++ b/docs/data-studio/transforms/transforms-overview.md
@@ -56,6 +56,16 @@ Permission configuration for transform depends on your plan.
- To **see** the list of transforms on your instance, people need to be able to access Data Studio, so they need to be either an Admin or a member of the special [Data Analyst group](../../people-and-groups/managing.md).
- To **execute** transforms on a database, people need to be either Admins on belong to the special [Data Analyst group](../../people-and-groups/managing.md). Additionally people need to have the [Transform permissions](../../permissions/data.md) for that database.
+## Set up transforms on a self-hosted Metabase
+
+If you run Metabase yourself, here's the path to working transforms:
+
+1. **Check your plan.** [Basic transforms](addons.md#basic-transforms) (query-based) are included on self-hosted Metabase. [Advanced transforms](addons.md#advanced-transforms) — Python transforms, the [transform inspector](transform-inspector.md), and [writable connections](../../databases/writable-connection.md) — need a self-hosted Pro or Enterprise plan with the Advanced transforms add-on.
+2. **Connect a database that can write.** Transforms create and replace tables in your database, so the database user needs create, drop, and write privileges. See [Database users, roles, and privileges](../../databases/users-roles-privileges.md). For cleaner isolation, point the transform at a [writable connection](../../databases/writable-connection.md). Only some databases [support transforms](#databases-that-support-transforms).
+3. **For Python transforms, set up a runner.** Query-based transforms need nothing extra — they run inside your database. Python transforms run in a separate execution environment, so you'll point Metabase at a [self-hosted Python runner](python-runner.md) backed by S3-compatible storage (AWS S3, MinIO, and so on).
+4. **[Enable transforms](#enable-transforms)** in Data Studio.
+5. **Create and run a transform.** [Create a query-based transform](query-transforms.md#create-a-query-based-transform) or a [Python transform](python-transforms.md#create-a-python-transform), run it manually, and check the result under **Runs**. Once it works, [schedule it with jobs](jobs-and-runs.md).
+
## Enable transforms
Before you can start writing transforms, you'll need to enable transforms in your Metabase instance.