The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
docs: transform screenshots (#70225)
Alex Yaroshb6b44f3cd68e
diff --git a/docs/data-studio/dependency-diagnostics.md b/docs/data-studio/dependency-diagnostics.mdindex 70ce6bffd14..8e27ad99bd1 100644--- a/docs/data-studio/dependency-diagnostics.md+++ b/docs/data-studio/dependency-diagnostics.md@@ -5,6 +5,8 @@ summary: Find broken content and unused items so you can fix problems and clean# Dependency diagnostics+{% include plans-blockquote.html feature="Dependency disagnistics" %}+Dependency diagnostics shows you content with broken dependencies, and content that isn't referenced by anything else.To open Dependency diagnostics:diff --git a/docs/data-studio/dependency-graph.md b/docs/data-studio/dependency-graph.mdindex ef2803ddac5..608a04d04ba 100644--- a/docs/data-studio/dependency-graph.md+++ b/docs/data-studio/dependency-graph.md@@ -5,6 +5,8 @@ summary: Visualize how your content connects and what depends on what in Metabas# Dependency graph+{% include plans-blockquote.html feature="Dependency graph" %}+The dependency graph provides a visual representation of how your Metabase content is connected. You can use it to follow how data flows through your Metabase to better understand the impact of any changes you make.diff --git a/docs/data-studio/images/create-transform.png b/docs/data-studio/images/create-transform.pngnew file mode 100644index 00000000000..c6bd7e7887dBinary files /dev/null and b/docs/data-studio/images/create-transform.png differdiff --git a/docs/data-studio/images/jobs.png b/docs/data-studio/images/jobs.pngnew file mode 100644index 00000000000..f0a27edfecaBinary files /dev/null and b/docs/data-studio/images/jobs.png differdiff --git a/docs/data-studio/images/new_transform.png b/docs/data-studio/images/new_transform.pngnew file mode 100644index 00000000000..6b19e847ccfBinary files /dev/null and b/docs/data-studio/images/new_transform.png differdiff --git a/docs/data-studio/images/runs.png b/docs/data-studio/images/runs.pngnew file mode 100644index 00000000000..d661c62f9b0Binary files /dev/null and b/docs/data-studio/images/runs.png differdiff --git a/docs/data-studio/images/sql-transform.png b/docs/data-studio/images/sql-transform.pngnew file mode 100644index 00000000000..e8a1ec4738eBinary files /dev/null and b/docs/data-studio/images/sql-transform.png differdiff --git a/docs/data-studio/images/tags.png b/docs/data-studio/images/tags.pngnew file mode 100644index 00000000000..be0764c2fe0Binary files /dev/null and b/docs/data-studio/images/tags.png differdiff --git a/docs/data-studio/images/transform-codegen.png b/docs/data-studio/images/transform-codegen.pngnew file mode 100644index 00000000000..2750d2f64e0Binary files /dev/null and b/docs/data-studio/images/transform-codegen.png differdiff --git a/docs/data-studio/images/transform-dependencies.png b/docs/data-studio/images/transform-dependencies.pngnew file mode 100644index 00000000000..9a7a09aa513Binary files /dev/null and b/docs/data-studio/images/transform-dependencies.png differdiff --git a/docs/data-studio/images/transform-edit-settings.png b/docs/data-studio/images/transform-edit-settings.pngnew file mode 100644index 00000000000..3dd0816a1d1Binary files /dev/null and b/docs/data-studio/images/transform-edit-settings.png differdiff --git a/docs/data-studio/images/transform-settings.png b/docs/data-studio/images/transform-settings.pngnew file mode 100644index 00000000000..af60d72c879Binary files /dev/null and b/docs/data-studio/images/transform-settings.png differdiff --git a/docs/data-studio/images/transforms.png b/docs/data-studio/images/transforms.pngnew file mode 100644index 00000000000..b7d8d4a100fBinary files /dev/null and b/docs/data-studio/images/transforms.png differdiff --git a/docs/data-studio/transforms/jobs-and-runs.md b/docs/data-studio/transforms/jobs-and-runs.mdindex cd1a0f3c0e8..5de28bba303 100644--- a/docs/data-studio/transforms/jobs-and-runs.md+++ b/docs/data-studio/transforms/jobs-and-runs.md@@ -11,6 +11,8 @@ Jobs are scheduled runs of transforms based on the transform's tags.## Transform tags++Add tags to transforms so that you can use jobs to run the transforms on a schedule. For example, you could add a "Nightly" tag to a transform, and have a job that runs all the transforms with the "Nightly" tag at midnight every day.To add a tag to a transform:@@ -23,12 +25,12 @@ By default, Metabase comes with hourly, daily, weekly, and monthly tags and jobsOnce you've tagged a transform, you can create a job that uses that tag to run the transform on the job's schedule.-Job can use multiple tags, and will run all transforms that have _any_ of those tags. For example, you can have a job "Weekend job" that is scheduled run at noon on Saturdays and Sundays that picks up all transforms tagged either "Saturday", "Sunday", or "Weekend".-## Jobs_Data Studio > Jobs_++Jobs run one or more transforms on schedule based on transform tags.To see all jobs, go to **Data Studio** and click on the **Jobs** at the bottom of the left sidebar.@@ -40,6 +42,8 @@ Jobs have two components: schedule and tags.- **Schedule** determines when the job will be executed: daily, hourly, etc. You can specify a custom cron schedule (e.g. "Every weekday at 9:05 AM"). The times are given in your Metabase's system timezone.- **Tags** determine _which_ transforms a job runs, not when the job runs. For example, you can create a `Weekdays` tag, add that tag to a few transforms, then create a job that runs all the transforms with the `Weekdays` tag every weekday at 9:05AM.+Job can use multiple tags, and will run all transforms that have _any_ of those tags. For example, you can have a job "Weekend job" that is scheduled run at noon on Saturdays and Sundays that picks up all transforms tagged either "Saturday", "Sunday", or "Weekend".+## Jobs will run all dependent transformsIf one transform depends on another, Metabase will run the dependency first, even if that transform isn't tagged in the job. So if transform B depends on A, Metabase will first run A, even if A doesn't have a tag.@@ -50,6 +54,8 @@ You can see which transforms a job will run (and in which order) on the job's pa## Runs++You can see all past and current transform runs (both manual and scheduled) by going to **Data Studio** and clicking on **Runs** at the bottom of the left sidebar. The transform run times will be given in Greenwich Mean Time (GMT).You can click on any transform run to see more details about the run, like the error logs. To go to the transform definition from the transform run page, click on the icon next the transform name in the right sidebar.diff --git a/docs/data-studio/transforms/python-runner.md b/docs/data-studio/transforms/python-runner.mdindex 64c53931771..c0024289ada 100644--- a/docs/data-studio/transforms/python-runner.md+++ b/docs/data-studio/transforms/python-runner.md@@ -5,14 +5,14 @@ summary: Configure self-hosted Python execution environment to run Python transf# Python runner-> Self-hosted Python transforms require self-hosted Pro or Enterprise plan with the Transforms add-on.+> Self-hosted Python transforms require self-hosted Pro or Enterprise plan with the **Advanced transforms** add-on.To run Python transforms from a self-hosted Metabase, you'll need to configure a separate self-hosted execution environment. If you're using Metabase Cloud, you'll need to buy the Transforms add-on.## Prerequisites- Docker installed and running.-- **Metabase Pro or Enterprise license** (Python Runner requires a Pro/Enterprise plan).+- **Self-hosted Metabase Pro or Enterprise license** (Python Runner requires a Pro/Enterprise plan).- For production: an S3-compatible storage bucket (AWS S3, MinIO, etc.).## Quickstartdiff --git a/docs/data-studio/transforms/python-transforms.md b/docs/data-studio/transforms/python-transforms.mdindex d9d63fbbc43..b9396ef2751 100644--- a/docs/data-studio/transforms/python-transforms.md+++ b/docs/data-studio/transforms/python-transforms.md@@ -24,9 +24,12 @@ Python-based transforms require a dedicated [Python execution environment](#set-To execute Python transforms, you'll need a _Python runner_ - a dedicated environment for running Python code.-If you're on Metabase Cloud, get the **Transforms add-on**, and you're good to go.+- If you're on Metabase Cloud, get the **Transforms add-on** with **SQL + Python** option, and you're good to go.-If you're self-hosting Metabase, you'll need to set up a self-hosted Python runner, see [Python runner](python-runner.md).+- If you're self-hosting Metabase, you'll need to:++ 1. Get the **Advanced transforms** add-on for your subscription in Metabase Store [store.metabase.com](https://store.metabase.com).+ 2. Set up a self-hosted Python runner, see [Python runner](python-runner.md).## Create a Python transformdiff --git a/docs/data-studio/transforms/query-transforms.md b/docs/data-studio/transforms/query-transforms.mdindex 86c5184169d..f00a9850e95 100644--- a/docs/data-studio/transforms/query-transforms.md+++ b/docs/data-studio/transforms/query-transforms.md@@ -5,10 +5,12 @@ summary: Create Metabase questions and SQL queries to transform your data and wr# Query-based transforms-> On Metabase Cloud, you need the Transforms add-on to run query-based transforms.+> On Metabase Cloud, you need the **Transforms** add-on to run query-based transforms.With query-based transforms, you can write a query in SQL or Metabase's query builder, and then write the results of the query back into the database on schedule.++For general information about Metabase transforms, see [Transforms](transforms-overview.md).## How query-based transforms work@@ -21,12 +23,14 @@ For general information about Metabase transforms, see [Transforms](transforms-o## Create a query-based transform-Currently, you can't convert between different transform types (like converting a query builder transform to a SQL-based transform, or a SQL transform into a Python transform). If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.+On Metabase Cloud, you need the **Transforms** add-on to create query-based transforms.1. Go to **Data studio > Transforms**.2. Click **+ New** and pick "Query builder", "SQL", or "Copy of existing question".+ Currently, you can't convert between different transform types (like converting a query builder transform to a SQL-based transform, or a SQL transform into a Python transform). If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.+3. Write your transform query as you would normally write a query in Metabase. See [Query builder](../../questions/query-builder/editor.md) and [SQL editor](../../questions/native-editor/writing-sql.md) documentation for more information.Not all databases support transforms, see [Databases that support transforms](transforms-overview.md#databases-that-support-transforms).diff --git a/docs/data-studio/transforms/transforms-overview.md b/docs/data-studio/transforms/transforms-overview.mdindex 7650c97d34b..e6900d2a73c 100644--- a/docs/data-studio/transforms/transforms-overview.md+++ b/docs/data-studio/transforms/transforms-overview.md@@ -11,6 +11,8 @@ Transforms can be used to, well, transform your data - do stuff like preprocessiYou'll write a query or a Python script in Metabase, a transform will run this query or script, create a table in your target database containing the results, and sync that table to Metabase, so it can be used as a data source for questions or other transforms.++## Transforms overview- **Transforms** are queries (created either with SQL or the query builder) or Python scripts that write back to your database and create a new, persistent table. Use transforms to clean, join, or pre-aggregate data.@@ -62,6 +64,8 @@ You can see all your Metabase's transforms:2. Click the **grid** icon on top right and go to **Data Studio**.3. In the left sidebar, select **Transforms**.++## Create a transform_Data Studio > Transforms_@@ -75,10 +79,14 @@ To create a transform:3. In the left sidebar, select **Transforms**.4. Click **+ New** and select a source for your transform.+ +You can create your transform using Metabase's [graphical query builder](../../questions/query-builder/editor.md), SQL, or Python.For more information on transforms built with the query builder or SQL, see [query-based transforms](query-transforms.md). For more information on Python transforms, see [Python transforms](python-transforms.md).+ +If you select **Copy of a saved question** for the transform's source, you can copy the query of an existing Metabase question (either a SQL question or a query builder question) into your transform query. Metabase will only _copy_ the question's query. Later edits to that original question won't affect the transform's query.5. Create the query or script for your transform.@@ -97,6 +105,8 @@ To create a transform:- **Folder** (optional): The folder where the transform should live. Click on the field to pick a different folder or create a new one.- **Incremental transformation** (optional): see [Incremental query-based transforms](query-transforms.md#incremental-query-transforms) or [Incremental Python transforms](python-transforms.md)+ +7. Optionally, once the transform is saved, assign tags to your transform. Tags are used by [jobs](./jobs-and-runs.md) to run transforms on schedule.## Use Metabot to generate code for transforms@@ -105,13 +115,19 @@ To create a transform:You can ask Metabot to generate a new SQL or Python-based transform, or edit an existing transform.-1. Go to **Data studio > Transforms**.+1. Go to **Data studio > Transforms**. If you want Metabot to edit an existing transform, navigate to the transform.2. While in Transforms view, click on the Metabot icon in top right.3. Describe the transform that you'd like Metabot to write.You can specify which kind of transform you want (Python or SQL) and @-mention specific data sources to help Metabot understand your request.-Metabot will create a to-do list for itself that will show its thinking process, and then work through the list.+ ++4. Metabot will create a to-do list for itself that will show its thinking process, and then work through the list.++5. If you asked Metabot to create a new transform, review the code that Metabot offers in the chat window and click on "Create" under the code snippet.++ You can continue working with Metabot to refine your code. Metabot will suggest code changes, and give you the option of accepting or rejecting the changes.@@ -123,6 +139,8 @@ You can edit the transform's name and description, query/script, target table, aIf you're using remote sync, you won't be able to edit transforms if your instance is in ["read-only" sync mode](../../installation-and-operation/remote-sync.md).++### Edit transform's query or script_Data Studio > Transforms > Definition_@@ -162,10 +180,12 @@ For Python transforms, you'll also see the transform's execution logs.## Transform dependencies-> Transform dependency graph is only available on Pro and Enterprise plans (both self-hosted and Metabase Cloud)+{% include plans-blockquote.html feature="Transform dependencies" is_plural=true%}_Data Studio > Transforms > Dependencies_++Transform queries can use the data from other transforms, and query-based transforms can also reference Metabase questions and models. For example, you can have a transform that uses data from a `raw_events` table and writes to a `stg_events` table, and then create another transform that uses data from the `stg_events` table and writes to an `events` table.Metabase will track transform dependencies and execute transforms in a reasonable order. So for example, if transform B relies on a table created by transform A, Metabase will run transform A first, then run transform B.