The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Edits on docs about how to run python transforms in self hosted deployments (#78098)
da5854e3e83ff4
diff --git a/docs/data-studio/transforms/addons.md b/docs/data-studio/transforms/addons.mdindex 906a217c852..00e8a3bf614 100644--- a/docs/data-studio/transforms/addons.md+++ b/docs/data-studio/transforms/addons.md@@ -37,7 +37,7 @@ With basic transforms, you can:- **Metabase Cloud**: Basic transform functionality on Metabase Cloud - Starter, Pro, or Enterprise - comes with an additional small fee per successful transform run, see [Pricing](https://www.metabase.com/pricing).- Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).+ Only people logged in with an email of a [Metabase Store admin](../../cloud/accounts-and-billing.md#add-managers) (not just Metabase _instance_ admins) can [enable basic transforms](./transforms-overview.md#enable-transforms).### Cancel basic transforms@@ -65,6 +65,8 @@ The Advanced transforms add-on comes with an additional charge per successful trTo enable Advanced transforms functionality, you need to have [Basic transforms](#basic-transforms) already, see [Enable basic transforms](#enable-basic-transforms).+For the full self-hosted setup see [Set up transforms on a self-hosted Metabase](transforms-overview.md#set-up-transforms-on-a-self-hosted-metabase).+There are two ways to enable Advanced transforms:- **From your Metabase instance**: you can navigate to a feature requiring advanced transforms (like Python transforms or transform inspector), and follow the prompts to upgrade.diff --git a/docs/data-studio/transforms/python-runner.md b/docs/data-studio/transforms/python-runner.mdindex 0bde1cdc1e0..69f7d04f7e2 100644--- a/docs/data-studio/transforms/python-runner.md+++ b/docs/data-studio/transforms/python-runner.md@@ -11,7 +11,7 @@ To run Python transforms from a self-hosted Metabase, you'll need to configure a## Prerequisites-- Docker installed and running.+- Docker installed and running, or an infrastructure that can run containers.- **Self-hosted Metabase Pro or Enterprise license** (Python Runner requires a Pro/Enterprise plan).- For production: an S3-compatible storage bucket (AWS S3, MinIO, etc.).@@ -69,13 +69,13 @@ docker network create metabase-networkdocker run -d \--network metabase-network \-e AUTH_TOKEN=your-secure-token-here \- --name python-runner metabase/python-runner:latest+ --name python-runner --hostname python-runner metabase/python-runner:latestdocker run -d \--network metabase-network \-p 3000:3000 \-e MB_PYTHON_RUNNER_URL=http://python-runner:5000 \- -e MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here \+ -e MB_PYTHON_RUNNER_API_TOKEN=<your-secure-token-here> \-e MB_PYTHON_STORAGE_S_3_ENDPOINT=https://s3.amazonaws.com \-e MB_PYTHON_STORAGE_S_3_BUCKET=your-bucket-name \-e MB_PYTHON_STORAGE_S_3_REGION=us-east-1 \@@ -97,24 +97,27 @@ docker run -d \These settings can also be configured in the Metabase UI at **Admin** > **Settings** > **Python Runner**. Note that environment variables take precedence over UI settings.-| Variable | Description |-| ----------------------------------------- | ---------------------------------------------------------------------------------- |-| `MB_PYTHON_RUNNER_URL` | URL where Metabase can reach the Python Runner (e.g., `http://python-runner:5000`) |-| `MB_PYTHON_RUNNER_API_TOKEN` | Authentication token. Must match `AUTH_TOKEN` in the Python Runner. |-| `MB_PYTHON_STORAGE_S_3_ENDPOINT` | S3 endpoint URL. |-| `MB_PYTHON_STORAGE_S_3_BUCKET` | S3 bucket name for storing Python artifacts |-| `MB_PYTHON_STORAGE_S_3_REGION` | AWS region (e.g., `us-east-1`) |-| `MB_PYTHON_STORAGE_S_3_ACCESS_KEY` | S3 access key |-| `MB_PYTHON_STORAGE_S_3_SECRET_KEY` | S3 secret key |-| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |+| Variable | Description |+| ------------------------------------------ | ---------------------------------------------------------------------------------- |+| `MB_PYTHON_RUNNER_URL` | URL where Metabase can reach the Python Runner (e.g., `http://python-runner:5000`) |+| `MB_PYTHON_RUNNER_API_TOKEN` | Authentication token. Must match `AUTH_TOKEN` in the Python Runner. |+| `MB_PYTHON_STORAGE_S_3_ENDPOINT` | S3 endpoint URL. |+| `MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` | S3 endpoint seen from the runner container, if different from the main endpoint. |+| `MB_PYTHON_STORAGE_S_3_BUCKET` | S3 bucket name for storing Python artifacts |+| `MB_PYTHON_STORAGE_S_3_REGION` | AWS region (e.g., `us-east-1`) |+| `MB_PYTHON_STORAGE_S_3_ACCESS_KEY` | S3 access key |+| `MB_PYTHON_STORAGE_S_3_SECRET_KEY` | S3 secret key |+| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |++`MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` is the host Metabase signs into the presigned URLs the runner uploads to and downloads from. Set it when the runner resolves storage by a different hostname than Metabase does.## Using Docker Compose-Docker Compose simplifies managing multiple containers. Below are example configurations for different scenarios.+Below are example configurations for different scenarios.-### Simple setup with LocalStack+### Self-hosted storage with MinIO-For simple setup and local testing, you can use LocalStack to simulate S3:+[MinIO](https://min.io/) is an S3-compatible server you can run yourself, which makes it a common pick for self-hosted Metabase. This Compose file runs Metabase, the Python runner, MinIO, and a short-lived container that creates the bucket Metabase will use — neither MinIO nor Metabase creates that bucket for you.```ymlname: metabase-python-runner@@ -125,15 +128,16 @@ services:- "3000:3000"environment:- MB_PYTHON_RUNNER_URL=http://python-runner:5000- - MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here- - MB_PYTHON_STORAGE_S_3_ENDPOINT=http://localstack:4566+ - MB_PYTHON_RUNNER_API_TOKEN=${AUTH_TOKEN}+ - MB_PYTHON_STORAGE_S_3_ENDPOINT=http://minio:9000+ - MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT=http://minio:9000- MB_PYTHON_STORAGE_S_3_BUCKET=metabase-python-runner- MB_PYTHON_STORAGE_S_3_REGION=us-east-1- MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS=true- - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=test- - MB_PYTHON_STORAGE_S_3_SECRET_KEY=test+ - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=${MINIO_ROOT_USER}+ - MB_PYTHON_STORAGE_S_3_SECRET_KEY=${MINIO_ROOT_PASSWORD}depends_on:- localstack-init:+ minio-init:condition: service_completed_successfullypython-runner:condition: service_started@@ -141,77 +145,59 @@ services:python-runner:image: metabase/python-runner:latestenvironment:- - AUTH_TOKEN=your-secure-token-here+ - AUTH_TOKEN=${AUTH_TOKEN}- localstack:- image: localstack/localstack:latest+ minio:+ image: quay.io/minio/minio:latest+ command: server /data --console-address ":9001"environment:- - SERVICES=s3- healthcheck:- test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]- interval: 5s- timeout: 5s- retries: 10-- localstack-init:- image: localstack/localstack:latest- entrypoint:- ["/bin/sh", "-c", "awslocal s3 mb s3://metabase-python-runner || true"]- environment:- - AWS_ENDPOINT_URL=http://localstack:4566+ - MINIO_ROOT_USER=${MINIO_ROOT_USER}+ - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}+ volumes:+ - minio-data:/data++ minio-init:+ image: quay.io/minio/mc:latestdepends_on:- localstack:- condition: service_healthy+ minio:+ condition: service_started+ entrypoint:+ - /bin/sh+ - -c+ - |+ until mc alias set local http://minio:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}" >/dev/null 2>&1; do+ echo "waiting for minio..."; sleep 2;+ done+ mc mb --ignore-existing local/metabase-python-runner++volumes:+ minio-data: {}```-Then run:+Both S3 endpoint variables point at `http://minio:9000` because Metabase and the runner share the Compose network and reach MinIO by the same hostname. `MB_PYTHON_STORAGE_S_3_CONTAINER_ENDPOINT` sets the host Metabase signs into the presigned URLs the runner uploads to and downloads from — if your runner resolves storage by a different hostname than Metabase does, set this to the runner's view of it.++Create a `.env` file:```bash-docker compose up -d+AUTH_TOKEN=your-secure-token-here+MINIO_ROOT_USER=minioadmin+MINIO_ROOT_PASSWORD=minioadmin```-### Production setup+Then bring everything up and try a transform:-For production, use an external S3-compatible storage service:+1. Put a strong shared secret in `AUTH_TOKEN` (the runner and Metabase have to agree on it):-```yml-name: metabase-python-runner-services:- metabase:- image: metabase/metabase-enterprise:latest- ports:- - "3000:3000"- environment:- - MB_PYTHON_RUNNER_URL=http://python-runner:5000- - MB_PYTHON_RUNNER_API_TOKEN=${AUTH_TOKEN}- - MB_PYTHON_STORAGE_S_3_ENDPOINT=${S3_ENDPOINT}- - MB_PYTHON_STORAGE_S_3_BUCKET=${S3_BUCKET}- - MB_PYTHON_STORAGE_S_3_REGION=${S3_REGION:-us-east-1}- - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=${S3_ACCESS_KEY}- - MB_PYTHON_STORAGE_S_3_SECRET_KEY=${S3_SECRET_KEY}- depends_on:- python-runner:- condition: service_started-- python-runner:- image: metabase/python-runner:latest- environment:- - AUTH_TOKEN=${AUTH_TOKEN}-```+ ```bash+ openssl rand -hex 32+ ```-Create a `.env` file with your configuration:+2. Start the stack:-```bash-AUTH_TOKEN=your-secure-token-here-S3_ENDPOINT=https://s3.amazonaws.com-S3_BUCKET=metabase-python-runner-S3_REGION=us-east-1-S3_ACCESS_KEY=your-access-key-S3_SECRET_KEY=your-secret-key-```+ ```bash+ docker compose up -d+ ```-Then run:+3. You'll need a Pro or Enterprise license with the [Advanced transforms add-on](addons.md). In Metabase, [enable transforms](transforms-overview.md#enable-transforms), then [create a Python transform](python-transforms.md#create-a-python-transform) and click **Run**.-```bash-docker compose up -d-```+4. Open **Data Studio > Jobs > Runs** and find your run. If it failed, the run's logs will usually tell you whether Metabase couldn't reach the runner, or the runner couldn't reach MinIO.\ No newline at end of filediff --git a/docs/data-studio/transforms/transforms-overview.md b/docs/data-studio/transforms/transforms-overview.mdindex e750334aaa7..85bb0f05a07 100644--- a/docs/data-studio/transforms/transforms-overview.md+++ b/docs/data-studio/transforms/transforms-overview.md@@ -56,6 +56,16 @@ Permission configuration for transform depends on your plan.- To **see** the list of transforms on your instance, people need to be able to access Data Studio, so they need to be either an Admin or a member of the special [Data Analyst group](../../people-and-groups/managing.md).- To **execute** transforms on a database, people need to be either Admins on belong to the special [Data Analyst group](../../people-and-groups/managing.md). Additionally people need to have the [Transform permissions](../../permissions/data.md) for that database.+## Set up transforms on a self-hosted Metabase++If you run Metabase yourself, here's the path to working transforms:++1. **Check your plan.** [Basic transforms](addons.md#basic-transforms) (query-based) are included on self-hosted Metabase. [Advanced transforms](addons.md#advanced-transforms) — Python transforms, the [transform inspector](transform-inspector.md), and [writable connections](../../databases/writable-connection.md) — need a self-hosted Pro or Enterprise plan with the Advanced transforms add-on.+2. **Connect a database that can write.** Transforms create and replace tables in your database, so the database user needs create, drop, and write privileges. See [Database users, roles, and privileges](../../databases/users-roles-privileges.md). For cleaner isolation, point the transform at a [writable connection](../../databases/writable-connection.md). Only some databases [support transforms](#databases-that-support-transforms).+3. **For Python transforms, set up a runner.** Query-based transforms need nothing extra — they run inside your database. Python transforms run in a separate execution environment, so you'll point Metabase at a [self-hosted Python runner](python-runner.md) backed by S3-compatible storage (AWS S3, MinIO, and so on).+4. **[Enable transforms](#enable-transforms)** in Data Studio.+5. **Create and run a transform.** [Create a query-based transform](query-transforms.md#create-a-query-based-transform) or a [Python transform](python-transforms.md#create-a-python-transform), run it manually, and check the result under **Runs**. Once it works, [schedule it with jobs](jobs-and-runs.md).+## Enable transformsBefore you can start writing transforms, you'll need to enable transforms in your Metabase instance.