The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
Data Studio docs (#66299)
Robert Kosara0ae4c70ba805
diff --git a/docs/data-modeling/model-persistence.md b/docs/data-modeling/model-persistence.mdindex 971a7408b37..f528657cecb 100644--- a/docs/data-modeling/model-persistence.md+++ b/docs/data-modeling/model-persistence.md@@ -4,6 +4,8 @@ title: Model persistence# Model persistence+> Prefer [Transforms](../data-studio/transforms/transforms-overview.md) instead of model persistence. Model persistence remains supported for now, but it will be deprecated in future versions of Metabase.+Metabase can persist the results of your models so that your models (and the questions based on those models) load faster.Metabase will store model results in tables in a bespoke schema in your data warehouse (not the Metabase application database). When people ask questions based on your models, Metabase will use the tables with the stored results instead of re-running the model's query.diff --git a/docs/data-modeling/python-transforms.md b/docs/data-modeling/python-transforms.mddeleted file mode 100644index b44760734ec..00000000000--- a/docs/data-modeling/python-transforms.md+++ /dev/null@@ -1,85 +0,0 @@-----title: Python transforms-summary: Use Python to wrangle your data in Metabase and write the results back to your database.------# Python transforms--> Python transforms are only available on Metabase Cloud [Pro and Enterprise plans](https://www.metabase.com/pricing) with the **Python execution add-on**.--You can write a Python script to transform the data from your database, and write the result back into the database as another table.--For general information about Metabase transforms, see [Transforms](transforms.md).--## Create a Python transform--To write Python transforms, you'll need to have the Python execution add-on. Once you've enabled the add-on:--1. Go to **Admin settings > Transforms** and click **Create a transform > Python script**.--2. Select a database that has the data you want to transform. See [Databases that support transforms](transforms.md#databases-that-support-transforms).--3. Select one or more tables with the data that you'd like to transform. Optionally, assign aliases to the tables.-- The tables you select will be available as DataFrames in your Python code with your chosen aliases, and will be passed to the `transform()` function as parameters.--4. Create a function `transform()` that does the data wrangling and returns a [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html). The DataFrame returned by the function will be written back to your database when the transform is run.-5. Click **Save** in the top right corner.-6. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.-- You can only write back to the same database as you chose for the transform source.--7. Optionally, assign tags to your transforms. Tags are used by [jobs](transforms.md#jobs-and-tags) to run transforms on schedule.--## Tips for Python transforms--- A Python transform must have a function `transform()` that returns a single `pandas` DataFrame.-- You can use aliases to include tables from your database as DataFrames inside the `transform()` function. The tables will _only_ be available in the transform function. Other functions won't have access to the tables.-- Only `pandas` will be imported by default, but you can import [certain other packages](#available-python-packages). You can also use functions from the [common library](#common-python-library).-- You'll see the output of `print()` statements in the transform's logs.-- Metabase won't write DataFrame indexes to the database, including indexes created by `groupby()`. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function.--## Run a Python transform--See [Run a transform](transforms.md#run-a-transform). You'll see logs for a transform run (including the output of any `print()` statements you included in the `transform()` code) on the transform's page.--## Available Python packages--You can import any of the following Python packages in Python transforms:--- [`pandas`](https://pandas.pydata.org/);-- Any dependencies of `pandas`, e.g. [`numpy`](https://numpy.org/);-- Any packages from the [Python standard library](https://docs.python.org/3/library/index.html), e.g. `json` or `datetime`.--Due to security considerations for the Python execution environment, you won't be able to install or import any other packages.--You can also write your own functions and add them to the [common library](#common-python-library) in Metabase, which will be available across all your Python transforms.--## Common Python library--If you have functions or classes you'd like to reuse across multiple transforms, you can add them to the common library that will be available across all Python transforms.--To add things to the common Python library:--1. Go to **Admin > Transforms > Python library**.--2. Add a Python function or class.-- Functions in this library can't access any data in your database.--To use functions or classes from your Python library:--1. When editing your Python transform script, check **Import common library** in the top right.-- This will add an aptly named import `import common` to the transform's code.--2. Reference functions or classes from the common library in your code like `common.manifest_kittens()`.--## Current limitations of Python transforms--- Currently, Python transforms are only available on Metabase Cloud.-- The transform function must return a single `pandas` DataFrame. Other data manipulation and DataFrame libraries like `polars` or `pyspark` are not supported.-- DataFrame indexes, including indexes created by `groupby()`, are ignored from writing back to the database. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function to make the index into a real column.-- You can't preview the results of a Python transform.-- Only a [limited set of packages](#available-python-packages) are available for import. You can't install additional packages.-- Because Python transforms use `pandas`, all data manipulation is done in memory. The available memory is determined by the Python execution add-on. For large datasets, consider using [query-based transforms](transforms.md#query-based-transforms) that run in your database.diff --git a/docs/data-modeling/transforms.md b/docs/data-modeling/transforms.mddeleted file mode 100644index 1ab45b70120..00000000000--- a/docs/data-modeling/transforms.md+++ /dev/null@@ -1,176 +0,0 @@-----title: Transforms-summary: Transforms allow you to wrangle your data in Metabase, write the query results back to your database, and reuse them in Metabase as sources for new queries.------# Transforms--{% include plans-blockquote.html feature="Transforms" is_plural=true%}--_Admin settings > Transforms_--Transforms can be used to, well, transform your data - do stuff like preprocessing, cleaning, joining tables, pre-computing metrics. Transforms give you the ability to do the "T" of "ETL" within Metabase.--You'll write a query or a Python script in Metabase, a transform will run this query or script, create a table in your target database containing the results, and sync that table to Metabase, so it can be used as a data source for questions or other transforms.--## Transforms overview--- **Transforms** are queries (either with SQL or the query builder) or Python scripts that write back to your database as a new, persistent table. Use transforms to clean, join, or pre-aggregate data.-- Transforms are scheduled and organized using **tags** and **jobs**.- - You assign tags (e.g., daily, hourly) to group your transforms.- - A job runs on a schedule (e.g., every day at midnight) and executes all transforms that have been assigned a specific tag.-- Each execution of a transform is a **run**. A run replaces the target table with fresh results. You can review the history of runs to monitor their success or failure.--## Databases that support transforms--Currently, Metabase can create transforms on the following databases:--- BigQuery-- ClickHouse (only ClickHouse Cloud)-- MySQL/MariaDB-- PostgreSQL-- Redshift-- Snowflake-- SQL Server--You can't create transforms on databases that have [Database routing](../permissions/database-routing.md) enabled, on Metabase [Cloud Storage](../cloud/storage.md), or on Metabase's Sample Database.--Transforms will create tables in your database, so the database user you use for your connection must have appropriate privileges. See [Database users, roles, and privileges](../databases/users-roles-privileges.md).--## Types of transforms--Metabase supports two types of transforms: query-based transforms and Python transforms. Here's a high-level overview of how they work and the differences between them:--### Query-based transforms--- In Metabase, you create a `SELECT` query either using SQL or Metabase's [graphical query builder](../questions/query-builder/editor.md).-- When the transform first runs, your _database_ executes the transform's query.-- Your database writes the results of the query as a new table.-- The new table is synced to Metabase.-- On subsequent transform runs, your database will overwrite that table with the updated results (updates are not incremental).--### Python-based transforms--Python-based transforms require a dedicated Python execution environment, so you'll need to buy the Python execution add-on.--- In your Metabase, you write a Python script that returns a `pandas` DataFrame and uses one or more tables from your database.-- When Metabase runs the transform, a new Python execution environment is spun up. Python transforms run in a separate, isolated environment—not on your Metabase instance.-- Metabase securely copies your source data to your Python environment and makes it available as pandas DataFrames.-- The Python environment executes your Python script _in memory_.-- The Python environment saves the resulting DataFrame as a file.-- Your Metabase instance reads the file and writes the results to a new table in your database.-- The new table is synced to Metabase.-- On subsequent transform runs, your database will overwrite that table with the updated results (updates are not incremental).--For more on Python transforms, see [Python transforms](python-transforms.md).--## Create a transform--Only admins can create transforms.--To create a transform:--1. Go to Admin settings > Transforms.-2. Click **+ Create transform**.-3. Select the source for your transform.-- You can create transforms using the query builder, SQL editor, or by copying an existing question or model. On Metabase Cloud with the Python execution add-on, you can also [create transforms from a Python script](python-transforms.md). See [Types of transforms](#types-of-transforms).-- If you select a question for the transform's source, Metabase will only _copy_ the question's query. Later edits to that question won't affect the transform's query.-- Your transform can use the target tables of other transforms.--4. Edit the query or the Python code for the transform.-- In SQL transforms, you can reference other saved questions and use snippets, but you can't use SQL parameters.-- When editing query-based transforms, you can preview results without writing data into your database.-- For editing Python transforms, see [Python transforms](python-transforms.md).--5. Click **Save** in the top right corner.-6. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.-- You can only transform data _within_ a database; you can't write from one database to another.--7. Optionally, assign tags to your transforms. Tags are used by [jobs](#jobs-and-tags) to run transforms on schedule.--## Edit a transform--You can edit the transform's name and description, query or script, and target table. You can edit the transform even if it already ran or is scheduled to run.--### Edit transform's query or script--To edit the transform's query or script, click on "Edit query" or "Edit script" above the transform definition.--Currently, you can't convert between different transform types (e.g. a query builder transform to a SQL-based transform, or a SQL transform into a Python transform). If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.--Once you change the transform's query or script, the next transform run (manual or scheduled) will use the updated query and write the results into the target table. If you have questions that query the transform's target table, they might break. For example, if your new transform query no longer includes a column that a downstream question was relying on, that question will break.--### Edit transform's target--To edit transform's target table, i.e., the table where the query results are written, click on "Change target". You'll need to select whether you want to keep the old target table, or delete it. Deletion can't be undone.--### Edit transform's name or description--To edit a transform's name or description, click on the name or description at the top of the transform page.--## Run a transform--You can run a transform manually or on schedule (e.g., hourly).--Running a transform for the first time will create and sync the table created by the transform, and you'll be able to edit the table's [metadata](./metadata-editing.md) and [permissions](../permissions/data.md). Subsequent runs will drop and recreate the table. Currently, Metabase doesn't support incremental transforms. Each transform run will recreate the target table using the entirety of the query results.--To run a transform manually, visit the transform and click **Run transform**.--To schedule a transform, you'll need to assign one or more tags to it, then create a [scheduled job](#jobs-and-tags) that picks up those tags.--You can see the time and status of the latest transform run on the transform's page, or in the Runs view. The time of the run is given in the system's timezone.--For Python transforms, you'll also see the transform's execution logs.--## Transform dependencies--Transform queries can use the data from other transforms. For example, you can have a transform that uses data from a `raw_events` table and writes to a `stg_events` table, and then create another transform that uses data from the `stg_events` table and writes to an `events` table.--Metabase will track transform dependencies, and execute them in order. For each transform, you can see which transforms the current transform depends on.--If a [job](#jobs-and-tags) includes a transform that depends on a table created by another transform, then the job will run all the tagged transforms and their dependencies, even if they lack tags.--## Jobs and tags--Jobs let you run multiple transforms on a schedule.--You can see and filter the list of all jobs, their last and next run, and the status of the least run, in _Admin settings > Transforms > Jobs_--To create a new job, go to _Admin settings > Transforms_, switch to the **Jobs** view, and click **Create a job**--To edit a job, click on the job in the list of jobs.--Jobs have two components: schedule and tags.--Metabase uses tags to locate transforms that should be run by the job, and then schedules those transforms according to the cron schedule. The times are given in the system's timezone.--### Tags--Job can use multiple tags, in which case, the job will run all transforms that have _any_ of those tags. For example, you can have a job "Weekend job" that is scheduled run at noon on Saturdays and Sundays that picks up all transforms tagged either "Saturday", "Sunday", or "Weekend".--By default, Metabase comes with hourly, daily, weekly, and monthly tags and jobs that are run on the corresponding schedules, but you can remove or rename those tags, or create new tags.--To create a new tag, just type the new tag's name in "Tags" field (either when viewing a transform or when viewing a job) and select "Create a tag".--### Jobs take dependencies into account--Depended transforms will be scheduled and run intelligently: if Transform B depends on the output of Transform A, then a job will run Transform A before Transform B. A job will run all dependent transforms even if the dependencies aren't tagged. You will see the order of transform execution on the job's page.--## Transforms vs models--Transforms are similar to models with model persistence turned on, but there are a few crucial differences:--- Transforms can only be created by admins. Models can be created by anyone with permissions to create queries on the data source (but only admins can enable model persistence).-- You can choose the target schema and tables for transforms. Model persistence will create its own schema and tables.-- You can't rename columns in a transform (but you can change column metadata once they're synced back to Metabase). You can rename columns in model metadata and then persist those column names using model persistence.-- Transforms support more databases than model persistence-- On Metabase Cloud with Python execution add-on, you can use Python to create transforms.--Use models to enable non-admins to create their own datasets within Metabase, and to add context like field descriptions and semantic types. Use transforms to create persisted datasets in your database and reuse them across Metabase.diff --git a/docs/data-studio/data-structure.md b/docs/data-studio/data-structure.mdnew file mode 100644index 00000000000..1aae4c960a2--- /dev/null+++ b/docs/data-studio/data-structure.md@@ -0,0 +1,150 @@+---+title: Data structure+summary: See all the tables in your Metabase, edit and sync metadata, edit column types, set visibility, publish tables, and more.+---++## Data structure++_Data studio > Data structure_++The **Data structure** area of [Data Studio](overview.md) is where you can manage table metadata to make it easier for people to work with your data.++You can do things like:++- Browse all the tables in your Metabase+- Publish tables to the Library+- Sync and scan tables+- Assign owners and other attributes to tables+- Set table and field visibility+- Edit table and column names and types+- Set data formatting settings+- Create measures and segments++## Permissions for Data Structure++To access the Data Structure area of Data Studio, you need to be a members of the Admin group or the Data Analyst group (Data Analyst group is only available on Pro/Enterprise plans).++People in the Data Analyst group will have table metadata and data structure access to _all_ tables in your Metabase, even if they have limited View Data permissions for those tables. If you only want to give someone access to table metadata for some - but not all - tables, use the table metadata permissions and access the Table Metadata through Admin > Table metadata instead of Data Studio.++## Browse tables++The Data Structure tab gives you an overview of all tables in all databases connected to your Metabase, together with their owner, row count, and published state. For now, Metabase only displays row counts for PostgreSQL tables.++You can search for table names, but the search will only match beginnings of words in table names. So for example, if you search for “base”, results will include names like “Baseball stats” and “All your base are belong to us”, but the results won't include tables like “Metabase secrets”.++You can also filter tables by attributes like owners, visibility, or source - for example, if you wanted to find all hidden tables, or all tables created from CSV uploads.++You can set [table attributes](#table-attributes), [edit metadata](#table-and-field-metadata), [publish the table](#publishing-and-unpublishing-tables) or create [segments](#segments) or [measures](#measures) on the table. You can also select tables in bulk to publish or assign attributes (including visibility) to multiple tables at once.++## Publishing and unpublishing tables++{% include plans-blockquote.html feature="Publishing tables to the Library" %}++Once you select a table in Data Structure, you can publish the table to add it to the Library. The Library is a special collection that helps you create a source of truth for analytics by providing a centrally managed set of curated content.++See [Publishing tables](./library.md#publishing-tables) in the [Library docs](./library.md).++## Sync settings++You can trigger manual re-sync of the table schema. Re-syncing can be useful if you have added or removed columns from the table, and you don't see those changes reflected in Metabase.++You can also re-scan field values for the table or discard cached field values, which is useful if you need to retrieve updated values for dropdown filters.++See [syncs and scans](../databases/sync-scan.md) for more information.++## Table attributes++### Owner++Table **owner** can be a Metabase user or an arbitrary email address. Use this owner attribute to let people know who on your team is responsible for which table.++Table owner attribute is not exposed to people outside Data Studio.++### Visibility layer++The **Visibility layer** attribute controls whether people in your Metabase can see the table when building new queries. You can also use it to tag tables according to how ready for end-user consumption they are (for example, if you're using medallion architecture).++The options are:++- **Hidden**: the table isn't available in the query builder and isn't [synced](../databases/sync-scan.md). People with SQL access can still query the table.+- **Internal**: table visible in the query builder and synced.+- **Final**: table is visible in the query builder and synced.++Table visibility layer can be used as a tool to improve user experience of people working in you Metabase, and it should _not be used to control access_ - use [data permissions](../permissions/data.md) to control access instead.++### Entity type++You can use **entity type** to tell people what kind of information is contained in the table, for example "Transaction" or "Event". Metabase will try to detect and automatically assign appropriate entity types (like entity type "Person" for a `Users` table"), but you can always change it later.++Outside Data Studio, entity type determines the record icons in the [details view](../exploration-and-organization/exploration.md#view-details-of-a-record).++### Source++**Source** describes where the data comes from. It can be useful when you want to identify tables that are, for example, ingested tables, or tables coming from CSV uploads.++Metabase will automatically assign the source "Metabase transforms” to tables created by [Metabase transforms](./transforms/transforms-overview.md), and source "Uploaded data" to tables created by new [CSV uploads](../databases/uploads.md) (CSV uploads that you created before getting Data Studio will not be automatically tagged).++## Table and field metadata++You can edit field descriptions, types, visibility settings, and formatting. For example, you can choose to display a filter on a field as a dropdown, or display days as `21.03.2026` instead of `03/21/2026`.++See [Table metadata editing](../data-modeling/metadata-editing.md).++## Segments++Segments are saved filters on tables. You can use segments to create an official definition of a subset of customers, users, or products that everyone on your team can refer to consistently (for example what constitutes an "active user").++People will see segments as options in the Filter block of the [query builder](../questions/query-builder/editor.md).++For now, in addition to Data Studio, segments can also be managed through **Admin settings > Table Metadata**, see [Segments in table metadata](../data-modeling/segments.md).++To see all segments on a table, select the table in **Data Structure** and switch to **Segments** tab.++### Create a segment++1. Select the table in **Data structure**.+2. Scroll down and switch to **Segments**.+3. Click on **+New Segment**. You'll see an abridged version of Metabase's query builder.+4. Add the filters describing the segment (e.g. `Active = true`) and name the segment.+5. To preview the records in the segment, click on the **three dots** icon next to the segment's name and select **Preview**.+6. Save.++### Use a segment in the query builder++To use a segment in the query builder, start a new question from the table that the segment is based on, and select the segment in the **Filter** block. The segment's filters will be applied behind the scenes.++### Delete a segment++Deleting a segment will not break questions using it. The questions that use the segment will keep using the same filters as before.++1. Select the table in **Data structure**.+2. Scroll down and switch to **Segments** and choose your segment.+3. On the segment's page, click on the **three dots** icon next to the segment's name and select **Remove segment**.++## Measures++Measures are saved aggregations on tables. You can use measures to create an official saved calculation, like what "Net Promoter Score" is calculated.++People will see measures as options in the Summarize block of the [query builder](../questions/query-builder/editor.md).++To see all measures on a table, select the table in **Data structure** and switch to **Measures** tab.++### Create a measure++1. Select the table in **Data structure**.+2. Scroll down and switch to **Measures**.+3. Click on **+New measure**. You'll see an abridged version of Metabase's query builder that focuses on aggregation.+4. Add your aggregation. For more complex aggregations, check out [custom expressions](../questions/query-builder/expressions.md).+5. To preview the results of the measure, click on the **three dots** icon next to the measure's name and select **Preview**.++### Use a measure in the query builder++To use a measure in the query builder, start a new question from the table that the measure is based on, and select the measure in the **Summarize** block. Measure's saved aggregation will be applied behind the scenes. You can use breakouts with a saved measure.++### Delete a measure++Deleting a measure will not break questions using it. The questions that use the measure will keep using the same aggregations as before.++1. Select the table in **Data structure**.+2. Scroll down and switch to **Measures** and choose your measure.+3. On the measure's page, click on the **three dots** icon next to the measure's name and select **Remove measure**.diff --git a/docs/data-studio/dependency-diagnostics.md b/docs/data-studio/dependency-diagnostics.mdnew file mode 100644index 00000000000..244b6167510--- /dev/null+++ b/docs/data-studio/dependency-diagnostics.md@@ -0,0 +1,98 @@+---+title: Dependency diagnostics+summary: Find broken content and unused items so you can fix problems and clean up safely.+---++# Dependency diagnostics++Dependency diagnostics shows you content with broken dependencies, and content that isn't referenced by anything else.++To open Dependency diagnostics:++1. Open [Data Studio](./overview.md).+2. Click **Dependency diagnostics** in the left sidebar.++Dependency diagnostics has two tabs: **Broken dependencies** and **Unreferenced entities**.++## Broken dependencies++> Note: Broken dependency detection currently covers only questions built with the query builder (MBQL). SQL questions and SQL models aren't included.++Broken dependencies lists items that other content depends on, where a dependent item references a column or field the item no longer provides. This can happen when a column is removed or renamed, or when a field is hidden from the query builder by setting Visibility to [**Do not include**](../data-modeling/metadata-editing.md).++The broken dependencies table includes:++- Tables+- [Questions](../questions/introduction.md)+- [Models](../data-modeling/models.md)++For each item, the list shows:++- **Dependency**: The upstream item causing the issue+- **Location**: Where the item lives in your collections+- **Problems**: Missing columns detected for this item+- **Broken dependents**: Downstream items that break because they reference those missing columns++Metabase only shows items when you have access to both the source item and at least one broken dependent, and neither item is archived.++### View details for a broken item++Selecting an item opens a details panel with:++- The item's name, owner, and creation date+- A list of missing or invalid columns, with copy buttons to help fix references+- Downstream items that are broken because of this specific issue. You can filter this list by entity type+- Quick links to open items or view them in the [dependency graph](./dependency-graph.md)++For tables created by transforms, the sidebar also shows a link to the source transform.++### Why an item can still run but show up here++Some items show up as broken even though they still return results. The query runs, but parts of it no longer line up cleanly with the current data structure.++This usually happens for one of the following reasons.++**Unknown field after a data source change**++Older queries may reference columns that Metabase can no longer clearly resolve, often because the data source, table, or schema changed. In the query builder, these appear as Unknown field.++To fix this, open the item and replace the Unknown field with the correct column from the current table or model.++**Downstream breakage after SQL output changes**++Another common case is when a SQL question or SQL model changes the columns it returns. Items built on top of it using the notebook editor may still run, but they rely on column names that no longer match the SQL output and are therefore considered broken.++To fix this, either restore the expected column names in the SQL query or update the affected items to use the new column names. You can use the dependency graph to find everything impacted.++## Unreferenced entities++Unreferenced entities shows items that aren't used by any other non-archived content. Use this view to clean up unused content, especially reusable items like models, metrics, and snippets. Start by checking why an item was created, when it was last edited, and whether it was meant to be reused.++An unreferenced item isn't broken and isn't automatically safe to delete. It simply means nothing else depends on it. Check [Usage analytics](../usage-and-performance-tools/usage-analytics.md) for view activity before you archive or delete an item.++The Unreferenced entities list is limited to what you can access, so an item can show up as unreferenced even if something you can’t see depends on it.++Unreferenced entities includes:++- Tables+- [Questions](../questions/introduction.md)+- [Models](../data-modeling/models.md)+- [Metrics](../data-modeling/metrics.md)+- Segments+- Measures+- [Snippets](../questions/native-editor/snippets.md)++For each item, the list shows:++- **Name**: The item's title+- **Location**: Where the item lives in your collections++### Viewing unreferenced item details++Selecting an item opens a details panel with:++- Who created the item and when+- When it was last edited+- Links to open the item or view it in the [dependency graph](./dependency-graph.md)++For tables, the sidebar also shows owners, descriptions, and links to open the schema or database. For tables created by transforms, the sidebar shows a link to the source transform.diff --git a/docs/data-studio/dependency-graph.md b/docs/data-studio/dependency-graph.mdnew file mode 100644index 00000000000..841cab5064c--- /dev/null+++ b/docs/data-studio/dependency-graph.md@@ -0,0 +1,91 @@+---+title: Dependency graph+summary: Visualize how your content connects and what depends on what in Metabase.+---++# Dependency graph++The dependency graph provides a visual representation of how your Metabase content is connected. You can use it to follow how data flows through your Metabase to better understand the impact of any changes you make.++## What the dependency graph tracks++The dependency graph tracks the relationships between:++- Tables+- [Questions](../questions/introduction.md)+- [Models](../data-modeling/models.md)+- [Snippets](../questions/native-editor/snippets.md)+- [Transforms](./transforms/transforms-overview.md)+- [Metrics](../data-modeling/metrics.md)+- [Dashboards](../dashboards/introduction.md)+- [Documents](../documents/introduction.md)++## Viewing the dependency graph++To open the full dependency graph:++1. Open [Data Studio](./overview.md).+2. Click **Dependency graph** in the left sidebar.++This opens a canvas view where you can search for and visualize the dependencies of any item across your entire Metabase instance.++You can also view the dependency graph from any item in Data Studio by switching to the Dependencies tab.++## What the dependency graph shows++The dependency graph shows both the content an item relies on and the content that relies on it, displayed as connected items you can explore.++Each item in the graph shows:++- Type (table, model, question, etc.)+- Name+- Dependent counts showing how many other items rely on it++These counts show how important an item is. Items with many dependents usually need extra care before making changes.++### Viewing dependent details++You can click any dependent count to open a details panel on the right, where you can:++**Search**++- Find items by name+- See each item’s title and collection location++**Filter**++- Verified items+- Items in dashboards+- Items in official collections+- Items not in personal collections++**Sort**++- Name+- Location+- View count++Sorting by view count helps you see which dependents are used most often.++## How Metabase finds dependencies++Metabase determines dependencies by analyzing how items are constructed and reused, not just by tracing their immediate data source.+It inspects things like:++- Query structure and joins+- References to other questions or models+- How items are used in dashboards, filters, and other content++Because of this, dependencies can exist even when no data is directly queried. For example:++- A SQL question may depend on another GUI question it queries+- A question used only to populate a dashboard filter dropdown still counts as a dependency+- Questions that are joined together may appear as separate dependent items++Some dependencies are inferred rather than explicitly defined. SQL queries can introduce implicit joins, and complex questions may have multiple internal dependency paths instead of a single source to result flow. As a result, a dependency may appear in the graph even if it’s not obvious from the question editor.++If a dependency looks unexpected, it usually reflects an indirect or inferred relationship. In those cases, review how the item is used before changing or removing it.++## Further reading++- [Library](./library.md)diff --git a/docs/data-studio/images/data-studio.png b/docs/data-studio/images/data-studio.pngnew file mode 100644index 00000000000..9c1b1de30fcBinary files /dev/null and b/docs/data-studio/images/data-studio.png differdiff --git a/docs/data-studio/images/library-in-data-studio.png b/docs/data-studio/images/library-in-data-studio.pngnew file mode 100644index 00000000000..f4532b414f2Binary files /dev/null and b/docs/data-studio/images/library-in-data-studio.png differdiff --git a/docs/data-studio/images/library-in-sidebar.png b/docs/data-studio/images/library-in-sidebar.pngnew file mode 100644index 00000000000..5c48981df11Binary files /dev/null and b/docs/data-studio/images/library-in-sidebar.png differdiff --git a/docs/data-studio/images/library-permissions.png b/docs/data-studio/images/library-permissions.pngnew file mode 100644index 00000000000..9b09b78b06eBinary files /dev/null and b/docs/data-studio/images/library-permissions.png differdiff --git a/docs/data-studio/images/starting-data.png b/docs/data-studio/images/starting-data.pngnew file mode 100644index 00000000000..c050ebc5362Binary files /dev/null and b/docs/data-studio/images/starting-data.png differdiff --git a/docs/data-studio/library.md b/docs/data-studio/library.mdnew file mode 100644index 00000000000..10ca9fe78cc--- /dev/null+++ b/docs/data-studio/library.md@@ -0,0 +1,102 @@+---+title: Library+summary: Create a source of truth for analytics with curated tables, metrics, and SQL snippets that your team can trust.+---++# Library++{% include plans-blockquote.html feature="Library" %}++"I have always imagined that Paradise will be a kind of library."++― Jorge Luis Borges++++The Library helps you create a source of truth for analytics by providing a centrally managed set of curated content. Use the Library to separate authoritative, reusable components from ad-hoc analyses.++## How the Library works++++The Library is a special section in the navigation sidebar of the main Metabase app that you curate in [Data Studio](./overview.md) (and only in Data Studio). Unlike regular collections, each folder in the Library restricts which type of content it contains (which helps keep the Library tidy).++## Adding items to the Library++1. Click the grid icon in the upper right.+2. Select **Data Studio**.+3. In the **Library** tab, click **+ New**.++You can:++- [Publish a table](#publishing-tables)+- [Create a metric](#metrics)+- [Create a SQL snippet or folder](#sql-snippets)++## Publishing tables++++Tables published to the Library appear first in the Data section when people choose data sources, nudging them toward trusted data.++You must explicitly publish tables to the Library. We use the word "publish" to suggest that the tables you include in your Library are meant to be finished, polished tables. If your tables need to be cleaned or combined before they're ready for analytical queries, check out [transforms](./transforms/transforms-overview.md).++Tables published to the Library remain available via the data browser as well.++### Managing tables++Once a table is published, you can view and manage its metadata, and more.++- Overview+- [Segments](../data-modeling/segments.md)+- Measures+- [Dependencies](./dependency-graph.md)++To query a table from the Library in Data Studio:++1. Click the table.+2. Click the three-dot menu.+3. Select **View**.++### Published tables can't have dependencies outside of the Library++Tables published to the Library can't depend on any tables outside of the Library. If, for example, you want to publish a table that includes data from another table, such as a [foreign-key remapping](../questions/visualizations/table.md#foreign-key-remapping), Metabase will publish those tables as well.++### Unpublishing tables++To unpublish a table from the Library:++1. Visit the table in Data Studio in the Library tab.+2. Click on the three-dot menu next to the table's name.+3. Click **Unpublish**.++If other tables depend on the table you want to unpublish, Metabase will unpublish those tables as well. You'll get a confirmation message explaining which tables Metabase would unpublish.++Unpublishing a table just removes the table from the Library. That table will still be available via the data browser and data pickers.++## Metrics++[Metrics](../data-modeling/metrics.md) are standardized calculations that people can trust. Only the metrics you save directly to the Library are included.++## SQL snippets++[SQL snippets](../questions/native-editor/snippets.md) are reusable bits of code. All snippets in your Metabase are available to the library, including snippets created elsewhere in your Metabase. You can also create snippet folders in the Library.++## Versioning the Library++You can [sync Library content to version control](../installation-and-operation/remote-sync.md), giving you change history and the ability to publish content across environments.++## Library permissions++++Metabase uses the standard [collection permissions](../permissions/collections.md) to determine who can view and edit items in the Library. One difference to note is that people in the Data Analyst group always have Curate access to the Library. Like with the Admin group, you can't change the Data Analyst's Curate access to the Library.++[Data permissions](../permissions/data.md) work the same way in the Library as everywhere else in Metabase. One thing to call out: like with models, if you publish a table to the Library, it will grant query access to a group with view access to the database, even if their group has Create Queries set to No in [data permissions](../permissions/data.md) for that particular table.++## Further reading++- [Dependency graph](./dependency-graph.md)+- [Remote sync](../installation-and-operation/remote-sync.md)+- [Models](../data-modeling/models.md)+- [Metrics](../data-modeling/metrics.md)diff --git a/docs/data-studio/overview.md b/docs/data-studio/overview.mdnew file mode 100644index 00000000000..de9fed1c7ad--- /dev/null+++ b/docs/data-studio/overview.md@@ -0,0 +1,25 @@+---+title: Data studio+summary: Data Studio provides tools to shape and track your data so everyone can trust the numbers.+---++# Data Studio++++Data Studio provides tools to shape and track your data so everyone can trust the numbers.++- **Create an easy-to-understand semantic layer** to match how people think about your business.+- **Speed up queries** by transforming tables to anticipate usage patterns.+- **View dependency graphs** to identify and fix problems before they impact reports.++## What's in Data Studio++- **[Library](./library.md)**\*: A curated space for your organization's most trusted analytics content—tables, metrics, and SQL snippets that your data team recommends.+- Data structures. Add table metadata to make tables easier to work with.+- Glossary. Define terms relevant to your business, both for people and agents trying to understand your data.+- **[Dependency graph](./dependency-graph.md)**\*: A visual map of how your content connects, so you can understand the impact of changes before you make them.+- Dependency diagnostics\*. See which items have broken dependencies, or that aren't used.+- Transforms++\* Available on [Pro and Enterprise plans](https://www.metabase.com/pricing/).diff --git a/docs/data-studio/transforms/jobs-and-runs.md b/docs/data-studio/transforms/jobs-and-runs.mdnew file mode 100644index 00000000000..cd1a0f3c0e8--- /dev/null+++ b/docs/data-studio/transforms/jobs-and-runs.md@@ -0,0 +1,57 @@+---+title: Transform jobs+summary: Run transforms on schedule.+---++# Transform jobs++_Data Studio > Jobs_++Jobs are scheduled runs of transforms based on the transform's tags.++## Transform tags++Add tags to transforms so that you can use jobs to run the transforms on a schedule. For example, you could add a "Nightly" tag to a transform, and have a job that runs all the transforms with the "Nightly" tag at midnight every day.++To add a tag to a transform:++1. Make sure you have [permissions to edit transforms](./transforms-overview.md#permissions-for-transforms).+2. Visit the transform in **Data studio > Transforms**.+3. On the **Settings** page for a transform, add transform tags.++By default, Metabase comes with hourly, daily, weekly, and monthly tags and jobs that are run on the corresponding schedules, but you can remove or rename those tags, or create new tags. To create a new tag, just type the new tag's name in **Tags** field (either when viewing a transform or when viewing a job) and select **Create a tag**.++Once you've tagged a transform, you can create a job that uses that tag to run the transform on the job's schedule.++Job can use multiple tags, and will run all transforms that have _any_ of those tags. For example, you can have a job "Weekend job" that is scheduled run at noon on Saturdays and Sundays that picks up all transforms tagged either "Saturday", "Sunday", or "Weekend".++## Jobs++_Data Studio > Jobs_++Jobs run one or more transforms on schedule based on transform tags.++To see all jobs, go to **Data Studio** and click on the **Jobs** at the bottom of the left sidebar.++To create a new job, go to **Data Studio > Jobs**, and click on the **+ New** button in the top right.++Jobs have two components: schedule and tags.++- **Schedule** determines when the job will be executed: daily, hourly, etc. You can specify a custom cron schedule (e.g. "Every weekday at 9:05 AM"). The times are given in your Metabase's system timezone.+- **Tags** determine _which_ transforms a job runs, not when the job runs. For example, you can create a `Weekdays` tag, add that tag to a few transforms, then create a job that runs all the transforms with the `Weekdays` tag every weekday at 9:05AM.++## Jobs will run all dependent transforms++If one transform depends on another, Metabase will run the dependency first, even if that transform isn't tagged in the job. So if transform B depends on A, Metabase will first run A, even if A doesn't have a tag.++This means that you can explicitly tag transform A to run daily, and transform B hourly, but because transform B depends on transform A, transform A will _also_ run hourly (in addition to daily), despite not having the tag.++You can see which transforms a job will run (and in which order) on the job's page in **Data Studio > Jobs**.++## Runs++You can see all past and current transform runs (both manual and scheduled) by going to **Data Studio** and clicking on **Runs** at the bottom of the left sidebar. The transform run times will be given in Greenwich Mean Time (GMT).++You can click on any transform run to see more details about the run, like the error logs. To go to the transform definition from the transform run page, click on the icon next the transform name in the right sidebar.++The "Tags" column in the **Runs** table will only show the transform's specific tags. But the run might not have anything to do with those tags. Another job with different tags could have run the transform because [jobs will run _all_ dependent transforms](#jobs-will-run-all-dependent-transforms).diff --git a/docs/data-studio/transforms/python-runner.md b/docs/data-studio/transforms/python-runner.mdnew file mode 100644index 00000000000..70f44c25957--- /dev/null+++ b/docs/data-studio/transforms/python-runner.md@@ -0,0 +1,219 @@+---+title: Python runner+summary: Configure self-hosted Python execution environment to run Python transforms in Metabase.+---++# Python runner++> Self-hosted Python transforms require self-hosted Pro or Enterprise plan with the Transforms add-on.++To run Python transforms from a self-hosted Metabase, you'll need to configure a separate self-hosted execution environment. If you're using Metabase Cloud, you'll need to buy the Transforms add-on.++## Prerequisites++- Docker installed and running.+- **Metabase Pro or Enterprise license** (Python Runner requires a Pro/Enterprise plan).+- For production: an S3-compatible storage bucket (AWS S3, MinIO, etc.).++## Quickstart++The simplest way to try the Python Runner is to use the built-in S3 server (not for production):++```bash+# Create a Docker network for the containers to communicate+docker network create metabase-network++# Start the Python Runner with internal S3+docker run -d \+ --network metabase-network \+ -e AUTH_TOKEN=your-secure-token-here \+ -e ENABLE_INTERNAL_S3=true \+ --name python-runner metabase/python-runner:latest++# Start Metabase Enterprise+docker run -d \+ --network metabase-network \+ -p 3000:3000 \+ -e MB_PYTHON_RUNNER_URL=http://python-runner:5000 \+ -e MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here \+ -e MB_PYTHON_STORAGE_S_3_ENDPOINT=http://python-runner:4566 \+ -e MB_PYTHON_STORAGE_S_3_BUCKET=metabase-python-runner \+ -e MB_PYTHON_STORAGE_S_3_REGION=us-east-1 \+ -e MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS=true \+ -e MB_PYTHON_STORAGE_S_3_ACCESS_KEY=test \+ -e MB_PYTHON_STORAGE_S_3_SECRET_KEY=test \+ --name metabase metabase/metabase-enterprise:latest+```++## Production setup++For production, use an external S3-compatible storage service:++### Step 1: Generate a secure authentication token++```bash+openssl rand -hex 32+```++Save this token securely - you'll need it for both containers.++### Step 2: Set up S3 storage++Create a bucket in your S3-compatible storage service (AWS S3, MinIO, etc.) and note your access credentials.++### Step 3: Start the containers++```bash+docker network create metabase-network++docker run -d \+ --network metabase-network \+ -e AUTH_TOKEN=your-secure-token-here \+ --name python-runner metabase/python-runner:latest++docker run -d \+ --network metabase-network \+ -p 3000:3000 \+ -e MB_PYTHON_RUNNER_URL=http://python-runner:5000 \+ -e MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here \+ -e MB_PYTHON_STORAGE_S_3_ENDPOINT=https://s3.amazonaws.com \+ -e MB_PYTHON_STORAGE_S_3_BUCKET=your-bucket-name \+ -e MB_PYTHON_STORAGE_S_3_REGION=us-east-1 \+ -e MB_PYTHON_STORAGE_S_3_ACCESS_KEY=your-access-key \+ -e MB_PYTHON_STORAGE_S_3_SECRET_KEY=your-secret-key \+ --name metabase metabase/metabase-enterprise:latest+```++## Configuration reference++### Python Runner environment variables++| Variable | Description |+| -------------------- | ------------------------------------------------------------------------------------------- |+| `AUTH_TOKEN` | Authentication token for API requests. Must match `MB_PYTHON_RUNNER_API_TOKEN` in Metabase. |+| `ENABLE_INTERNAL_S3` | Set to `true` to enable built-in S3 server on port 4566. Not for production. |++### Metabase environment variables++These settings can also be configured in the Metabase UI at **Admin Settings** > **Settings** > **Python Runner** (`/admin/settings/python-runner`). Note that environment variables take precedence over UI settings.++| Variable | Description |+| ----------------------------------------- | ---------------------------------------------------------------------------------- |+| `MB_PYTHON_RUNNER_URL` | URL where Metabase can reach the Python Runner (e.g., `http://python-runner:5000`) |+| `MB_PYTHON_RUNNER_API_TOKEN` | Authentication token. Must match `AUTH_TOKEN` in the Python Runner. |+| `MB_PYTHON_STORAGE_S_3_ENDPOINT` | S3 endpoint URL. |+| `MB_PYTHON_STORAGE_S_3_BUCKET` | S3 bucket name for storing Python artifacts |+| `MB_PYTHON_STORAGE_S_3_REGION` | AWS region (e.g., `us-east-1`) |+| `MB_PYTHON_STORAGE_S_3_ACCESS_KEY` | S3 access key |+| `MB_PYTHON_STORAGE_S_3_SECRET_KEY` | S3 secret key |+| `MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS` | (Optional) Set to `true` for S3-compatible services like MinIO or LocalStack |++---++## Using Docker Compose++Docker Compose simplifies managing multiple containers. Below are example configurations for different scenarios.++### Simple setup with LocalStack++For simple setup and local testing, you can use LocalStack to simulate S3:++```yml+name: metabase-python-runner+services:+ metabase:+ image: metabase/metabase-enterprise:latest+ ports:+ - "3000:3000"+ environment:+ - MB_PYTHON_RUNNER_URL=http://python-runner:5000+ - MB_PYTHON_RUNNER_API_TOKEN=your-secure-token-here+ - MB_PYTHON_STORAGE_S_3_ENDPOINT=http://localstack:4566+ - MB_PYTHON_STORAGE_S_3_BUCKET=metabase-python-runner+ - MB_PYTHON_STORAGE_S_3_REGION=us-east-1+ - MB_PYTHON_STORAGE_S_3_PATH_STYLE_ACCESS=true+ - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=test+ - MB_PYTHON_STORAGE_S_3_SECRET_KEY=test+ depends_on:+ localstack-init:+ condition: service_completed_successfully+ python-runner:+ condition: service_started++ python-runner:+ image: metabase/python-runner:latest+ environment:+ - AUTH_TOKEN=your-secure-token-here++ localstack:+ image: localstack/localstack:latest+ environment:+ - SERVICES=s3+ healthcheck:+ test: ["CMD", "curl", "-f", "http://localhost:4566/_localstack/health"]+ interval: 5s+ timeout: 5s+ retries: 10++ localstack-init:+ image: localstack/localstack:latest+ entrypoint:+ ["/bin/sh", "-c", "awslocal s3 mb s3://metabase-python-runner || true"]+ environment:+ - AWS_ENDPOINT_URL=http://localstack:4566+ depends_on:+ localstack:+ condition: service_healthy+```++Then run:++```bash+docker compose up -d+```++### Production setup++For production, use an external S3-compatible storage service:++```yml+name: metabase-python-runner+services:+ metabase:+ image: metabase/metabase-enterprise:latest+ ports:+ - "3000:3000"+ environment:+ - MB_PYTHON_RUNNER_URL=http://python-runner:5000+ - MB_PYTHON_RUNNER_API_TOKEN=${AUTH_TOKEN}+ - MB_PYTHON_STORAGE_S_3_ENDPOINT=${S3_ENDPOINT}+ - MB_PYTHON_STORAGE_S_3_BUCKET=${S3_BUCKET}+ - MB_PYTHON_STORAGE_S_3_REGION=${S3_REGION:-us-east-1}+ - MB_PYTHON_STORAGE_S_3_ACCESS_KEY=${S3_ACCESS_KEY}+ - MB_PYTHON_STORAGE_S_3_SECRET_KEY=${S3_SECRET_KEY}+ depends_on:+ python-runner:+ condition: service_started++ python-runner:+ image: metabase/python-runner:latest+ environment:+ - AUTH_TOKEN=${AUTH_TOKEN}+```++Create a `.env` file with your configuration:++```bash+AUTH_TOKEN=your-secure-token-here+S3_ENDPOINT=https://s3.amazonaws.com+S3_BUCKET=metabase-python-runner+S3_REGION=us-east-1+S3_ACCESS_KEY=your-access-key+S3_SECRET_KEY=your-secret-key+```++Then run:++```bash+docker compose up -d+```diff --git a/docs/data-studio/transforms/python-transforms.md b/docs/data-studio/transforms/python-transforms.mdnew file mode 100644index 00000000000..d9d63fbbc43--- /dev/null+++ b/docs/data-studio/transforms/python-transforms.md@@ -0,0 +1,131 @@+---+title: Python transforms+summary: Use Python to wrangle your data in Metabase and write the results back to your database.+---++# Python transforms++> Python transforms require the **Transforms add-on**.++Use Python to write [transforms](transforms-overview.md).++## How Python transforms work++Python-based transforms require a dedicated [Python execution environment](#set-up-a-python-runner).++- In your Metabase, you write a Python script that returns a `pandas` DataFrame and uses one or more tables from your database.+- Metabase will spin up a new Python execution environment and run the transform there (not on your Metabase instance).+- Metabase securely copies your source data to your Python environment and makes it available as pandas DataFrames.+- The Python environment executes your Python script _in memory_ and saves the resulting DataFrame as a file.+- Your Metabase reads the DataFrame file, writes the results to a new table in your database, and syncs the table to your Metabase.+- On subsequent transform runs, your database will overwrite that table with the updated results unless you mark the transform as [incremental](#incremental-python-transforms).++## Set up a Python runner++To execute Python transforms, you'll need a _Python runner_ - a dedicated environment for running Python code.++If you're on Metabase Cloud, get the **Transforms add-on**, and you're good to go.++If you're self-hosting Metabase, you'll need to set up a self-hosted Python runner, see [Python runner](python-runner.md).++## Create a Python transform++Once you've [set up the Python runner](#set-up-a-python-runner):++1. Go to **Data studio > Transforms**.+2. Click on **+ New** and select **Python script**.++3. Select a database that has the data you want to transform. See [Databases that support transforms](transforms-overview.md#databases-that-support-transforms).++4. Select one or more tables with the data that you'd like to transform. Optionally, assign aliases to the tables.++ The tables you select will be available as DataFrames in your Python code with your chosen aliases, and will be passed to the `transform()` function as parameters. For now, all the tables you pick must be in the same database.++5. Create a function `transform()` that does the data wrangling and returns a [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html).++ See [Tips for writing Python transforms](#tips-for-writing-python-transforms). The DataFrame returned by the function will be written back to your database when the transform is run.++6. To test your transform, press the **Run Python script** button at the bottom right of the editor.++ Metabase will pull 100 rows from each input table and run your transform on those rows. You'll be able to see the result in the **Results preview** tab below the editor. You can see any other output (e.g. outputs of any print statements in your code) in the **Output** tab.++ The transform preview will **only use first 100 rows from each input table**. This means that you might not see the real results of your transforms in preview: for example, if your transform renames the values of `Doohickey` to `Widget`, and the first 100 records of the input table happen to not contain any doohickeys, you won't be able to preview the result.++7. Once you're done with your code, click **Save** in the top right corner.++8. Select a target schema for your transform and enter a name for the target table. Metabase will write the results of the transform into this table.++ You can only write back to the same database as you chose for the transform source.++9. Optionally, make your transform incremental. See [Incremental Python transforms](#incremental-python-transforms)+10. Optionally, assign tags to your transforms. Tags are used by [jobs](jobs-and-runs.md) to run transforms on schedule.++## Tips for writing Python transforms++- Metabase will automatically add `import common` to the code of your Python transform. This imports the [Python library](#common-python-library). You can use Python library for reusable functions and classes.+- A Python transform must have a function `transform()` that returns a single `pandas` DataFrame.+- You can use aliases to include tables from your database as DataFrames inside the `transform()` function. The tables will _only_ be available in the transform function. Other functions won't have access to the tables.+- Only `pandas` will be imported by default, but you can import [certain other packages](#available-python-packages). You can also use functions from the [common library](#common-python-library).+- Metabase won't write DataFrame indexes to the database, including indexes created by `groupby()`. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function.++## Run a Python transform++See [Run a transform](transforms-overview.md#run-a-transform). You'll see logs for a transform run (including the output of any `print()` statements you included in the `transform()` code) on the transform's page.++## Available Python packages++You can import any of the following Python packages in Python transforms:++- [`pandas`](https://pandas.pydata.org/).+- Any dependencies of `pandas`, e.g. [`numpy`](https://numpy.org/).+- Any packages from the [Python standard library](https://docs.python.org/3/library/index.html), e.g. `json` or `datetime`.++Due to security considerations for the Python execution environment, you won't be able to install or import any other packages.++You can also write your own functions and add them to the [common Python library](#common-python-library) in Metabase, which will be available across all your Python transforms.++## Common Python library++If you have functions or classes you'd like to reuse across multiple transforms, you can add them to the common library that will be available across all Python transforms.++To add things to the common Python library:++1. Go to **Data studio > Transforms**.+2. Scroll to the very bottom of the transforms list and click on **Python library**.++3. Add a Python function or class.++ Functions in this library can't access any data in your database.++To use functions or classes from your Python library:++1. Metabase will automatically add `import common` to your transform's code. The `common` here refers to the Python library. You can click on `common` in the editor to navigate to the library.++2. You can reference functions or classes from the common library in your code like `common.manifest_kittens()`.++## Incremental Python transforms++By default, Metabase will process all the data in all input tables, drop the existing target table (if one exists), and create a new table with the processed data. You can tell Metabase to only write **new** data to your target table by marking your transform as incremental.++### Prerequisites for incremental transforms++To update a table incrementally, your data has to have a certain structure. See [Prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).++### Make a Python transform incremental++To make a Python transform incremental:++1. Go to the transform's page in **Data studio > Transforms**.+2. Switch to **Settings** tab.+3. In **Column to check for new values**, select the column that Metabase should check to determine which values are new. See [Prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms) for more information on the requirements for that column.++ Unlike [Query transforms](./query-transforms.md), where you select an _output_ column as the column to check for new values, with Python transforms, you have to select a column from the _input_ tables as the column to check for new values.++## Current limitations of Python transforms++- The transform function must return a single `pandas` DataFrame. Other data manipulation and DataFrame libraries like `polars` or `pyspark` are not supported.+- Transform preview only uses 100 input rows from each input table.+- DataFrame indexes, including indexes created by `groupby()`, are ignored from writing back to the database. If you're using a custom index that you'd like to include in the target table, you'll need to [reset index](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html) on your DataFrame inside the `transform()` function to make the index into a real column.+- Only a [limited set of packages](#available-python-packages) are available for import. You can't install additional packages.+- Because Python transforms use `pandas`, all data manipulation is done in memory. The available memory is determined by the Python execution add-on. For large datasets, consider using [query-based transforms](./query-transforms.md) that run in your database.+- Only one Python transform can be run at any given time.diff --git a/docs/data-studio/transforms/query-transforms.md b/docs/data-studio/transforms/query-transforms.mdnew file mode 100644index 00000000000..87fd8ff578c--- /dev/null+++ b/docs/data-studio/transforms/query-transforms.md@@ -0,0 +1,96 @@+---+title: Query-based transforms+summary: Create Metabase questions and SQL queries to transform your data and write the results back into your data warehouse.+---++# Query-based transforms++> On Metabase Cloud, you need the Transforms add-on to run query-based transforms.++With query-based transforms, you can write a query in SQL or Metabase's query builder, and then write the results of the query back into the database on schedule.++For general information about Metabase transforms, see [Transforms](transforms-overview.md).++## How query-based transforms work++- In Metabase, you create a `SELECT` query either using SQL or Metabase's [graphical query builder](../../questions/query-builder/editor.md).+- When the transform first runs, your _database_ executes the transform's query.+- Your database writes the results of the query to a new table.+- The new table is synced to Metabase.+- On subsequent transform runs, your database will overwrite that table with the updated results unless you [configure your transform to be incremental](#incremental-query-transforms).++## Create a query-based transform++Currently, you can't convert between different transform types (like converting a query builder transform to a SQL-based transform, or a SQL transform into a Python transform). If you want to change your transform built with the query builder into a SQL transform, you'll need to create a new transform with the same target and tags, and delete the old transform.++1. Go to **Data studio > Transforms**.++2. Click **+ New** and pick "Query builder", "SQL", or "Copy of existing question".++3. Write your transform query as you would normally write a query in Metabase. See [Query builder](../../questions/query-builder/editor.md) and [SQL editor](../../questions/native-editor/writing-sql.md) documentation for more information.++ Not all databases support transforms, see [Databases that support transforms](transforms-overview.md#databases-that-support-transforms).++4. To test your transform, press the **Run** button at the bottom of the editor.++ Previewing a query transform in the editor will _not_ write the result of the transform back to the database.++5. Click **Save** in the top right corner and fill out the transform information:++ - **Name** (required): The name of the transform.+ - **Schema** (required): Target schema for your transform. This schema can be different from the schema of the source table(s). You create a new schema by typing its name in this field. You can only transform data _within_ a database; you can't write from one database to another.+ - **Table name** (required): Name of the target table. Metabase will write the results of the transform into this table, and then sync the table in Metabase.+ - **Folder** (optional): The folder where the transform should live. Click on the field to pick a different folder or create a new one.+ - **Incremental transformation** (optional): see [Incremental query transforms](#incremental-query-transforms)++6. Optionally, assign tags to your transforms. Tags are used by [jobs](jobs-and-runs.md) to run transforms on schedule.++## Run a query transform++See [Run a transform](transforms-overview.md#run-a-transform). You'll see logs for a transform run on the transform's page.++## Incremental query transforms++By default, on every transform run after the first one, Metabase will process all the data in all input tables, then drop the existing target table, and create a new table with the processed data. You can tell Metabase to only write **new** data to your target table by marking your transform as incremental.++### Prerequisites for incremental transforms++Your data has to have certain structure for incremental transforms to work. See [Prerequisites for incremental transforms](transforms-overview.md#prerequisites-for-incremental-transforms).++### How incremental query transforms work++For a transform to run incrementally, you'll need to pick a column ("checkpoint") that Metabase needs to check for new values. Then, behind the scenes, Metabase will add a filter around your transform query that will filter the results of the query for values greater than the last written checkpoint value.++### Make a query transform incremental++To make a query transform incremental:++1. Go to the transform's page in **Data studio > Transforms**.+2. Switch to **Settings** tab.+3. In **Column to check for new values**, select the column that Metabase should check to determine which values are new. See [Prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms) for more information on the requirements for that column.++ You have to select the column from the list of the columns of the _output_ tables. Note: this is different from [Python transforms](python-transforms.md), where you select an _input_ column as column to check for new values.++ If you're using SQL, Metabase might tell you that your query is too complicated to automatically make the transform incremental. In this case, you need to add the filter for new values manually. For example, let's say you have a transform query:++ ```sql+ SELECT id, total FROM orders;+ ```++ (This query is actually simple enough for Metabase to handle it automatically, we're just using it as an example)++ If you want use the `id` column to check for new values, i.e. only write back the records with `id` greater than already existing `id`, you can add a manual filter like this:++ ```sql+ SELECT id, total FROM orders+ {%raw%}[[WHERE id > {{checkpoint}}]]{% endraw %}+ ```++ and then select `id` as the **Column to check for new values** in the incremental transform settings.++ If you're using a timestamp column as a checkpoint, you'll need to explicitly cast it to timestamp:++ ```sql+ SELECT created_at, total FROM orders+ {%raw%}[[WHERE created_at > {{checkpoint}}::timestamp]]{% endraw %}+ ```diff --git a/docs/data-studio/transforms/transforms-overview.md b/docs/data-studio/transforms/transforms-overview.mdnew file mode 100644index 00000000000..e6a68811b3a--- /dev/null+++ b/docs/data-studio/transforms/transforms-overview.md@@ -0,0 +1,215 @@+---+title: Transforms overview+summary: Transforms allow you to wrangle your data in Metabase, write the query results back to your database, and reuse them in Metabase as sources for new queries.+---++# Transforms++_Data Studio > Transforms_++Transforms can be used to, well, transform your data - do stuff like preprocessing, cleaning, joining tables, pre-computing metrics. Transforms give you the ability to do the "T" of "ETL" within Metabase.++You'll write a query or a Python script in Metabase, a transform will run this query or script, create a table in your target database containing the results, and sync that table to Metabase, so it can be used as a data source for questions or other transforms.++## Transforms overview++- **Transforms** are queries (created either with SQL or the query builder) or Python scripts that write back to your database and create a new, persistent table. Use transforms to clean, join, or pre-aggregate data.+- Transforms are scheduled and organized using **tags** and **jobs**.+ - You assign tags (e.g., daily, hourly) to group your transforms.+ - A job runs on a schedule (e.g., every day at midnight) and executes all transforms that have been assigned a specific tag.+- Each execution of a transform is a **run**. A run replaces the target table with fresh results. You can review the history of runs to monitor their success or failure.++## Databases that support transforms++Currently, Metabase can create transforms on the following databases:++- BigQuery+- ClickHouse (only ClickHouse Cloud)+- MySQL/MariaDB+- PostgreSQL+- Redshift+- Snowflake+- SQL Server++You can't create transforms on databases that have [Database routing](../../permissions/database-routing.md) enabled, or on Metabase's Sample Database.++Transforms will create tables in your database, so the database user you use for your connection must have appropriate privileges. See [Database users, roles, and privileges](../../databases/users-roles-privileges.md).++## Types of transforms++Metabase supports two types of transforms: query-based transforms and Python transforms. You can write query-based transforms in SQL or Metabase Query builder, and they will run in your database. Python transforms are written in (unsurprisingly) Python and will run in a dedicated execution environment. For more details:++- [How query-based transforms work](query-transforms.md#how-query-based-transforms-work)+- [How Python transforms work](python-transforms.md#how-python-transforms-work)++## Permissions for transforms++If you are running Metabase Open Source/Starter, Admins (and only Admins) can see and run transforms.++Metabase Pro/Enterprise comes with additional permission controls for transforms, see [Transform permissions](../../permissions/data.md).++To **see** the list of transforms on your instance, people need to be able to access Data Studio, so they need to be either an Admin or a member of the special [Data Analyst group](../../people-and-groups/managing.md).++To **execute** transforms on a database, people additionally need to have the [Transform permissions](../../permissions/data.md) for that database.++## See all transforms++_Data Studio > Transforms_++You can see all your Metabase's transforms:++1. Make sure you have [appropriate permissions to see transforms](#permissions-for-transforms).+2. Click on the **grid** icon on top right and go to **Data Studio**.+3. In the left sidebar, select **Transforms**.++## Create a transform++_Data Studio > Transforms_++If you're using remote sync, you won't be able to create transforms if your instance is in ["read-only" sync mode](../../installation-and-operation/remote-sync.md).++To create a transform:++1. Make sure you have [appropriate permissions for creating transforms](#permissions-for-transforms).+2. Click on the **grid** icon on top right and go to **Data Studio**.+3. In the left sidebar, select **Transforms**.+4. Click **+ New** and select a source for your transform.++ You can create your transform using Metabase's [graphical query builder](../../questions/query-builder/editor.md), SQL, or Python.++ For more information on transforms built with the query builder or SQL, see [query-based transforms](query-transforms.md). For more information on Python transforms, see [Python transforms](python-transforms.md).++ If you select **Copy of a saved question** for the transform's source, you can copy the query of an existing Metabase question (either a SQL question or a query builder question) into your transform query. Metabase will only _copy_ the question's query. Later edits to that original question won't affect the transform's query.++5. Create the query or script for your transform.++ See [query-based transforms](query-transforms.md) and [Python transforms](python-transforms.md) for more information. You can reference target tables of other transforms when writing your transform.++ If you have the Metabot AI add-on, you can [use Metabot](#use-metabot-to-generate-code-for-transforms) to generate code for your transform.++6. Click **Save** in the top right corner and fill out the transform information:++ - **Name** (required): The name of the transform.+ - **Schema** (required): Target schema for your transform. This schema can be different from the schema of the source table(s). You create a new schema by typing its name in this field. You can only transform data _within_ a database; you can't write from one database to another.+ - **Table name** (required): Name of the target table. Metabase will write the results of the transform into this table, and then sync the table in Metabase.+ - **Folder** (optional): The folder where the transform should live. Click on the field to pick a different folder or create a new one.+ - **Incremental transformation** (optional): see [Incremental query-based transforms](query-transforms.md#incremental-query-transforms) or [Incremental Python transforms](python-transforms.md)++7. Optionally, once the transform is saved, assign tags to your transform. Tags are used by [jobs](./jobs-and-runs.md) to run transforms on schedule.++## Use Metabot to generate code for transforms++> Code generation for transforms requires the **Metabot AI** and **Transforms** add-ons.++You can ask Metabot to generate a new SQL or Python-based transform, or edit an existing transform.++1. Go to **Data studio > Transforms**.+2. While in Transforms view, click on the Metabot icon in top right.+3. Describe the transform that you'd like Metabot to write.++ You can specify which kind of transform you want (Python or SQL) and @-mention specific data sources to help Metabot understand your request.++Metabot will create a to-do list for itself that will show its thinking process, and then work through the list.++You can continue working with Metabot to refine your code. Metabot will suggest code changes, and give you the option of accepting or rejecting the changes.++## Edit a transform++_Data Studio > Transforms_++You can edit the transform's name and description, query/script, target table, and incremental settings. You can edit the transform even if it already ran or is scheduled to run.++If you're using remote sync, you won't be able to edit transforms if your instance is in ["read-only" sync mode](../../installation-and-operation/remote-sync.md).++### Edit transform's query or script++_Data Studio > Transforms > Definition_++To edit the transform's query or script:++1. Make sure you have [permissions to edit transforms](../../permissions/data.md).+2. Go to **Data Studio > Transforms**.+3. Find the transform you'd like to edit and click on **Edit definition** above the transform definition.+4. Edit the query or script.++ See [query-based transforms](query-transforms.md) and [Python transforms](python-transforms.md) for more information. You can [use Metabot](#use-metabot-to-generate-code-for-transforms) to help edit your transform.++Once you change the transform's query or script, the next transform run (manual or scheduled) will use the updated query and write the results into the target table. If you've changed the table's columns, and you have questions that query the table, they might break. For example, if your new transform query no longer includes a column that a downstream question was relying on, that question will break.++### Edit transform's target++_Data Studio > Transforms > Settings_++To edit transform's target table, i.e., the table where the query results are written, go the transforms **Settings** tab and click on **Change target**. You'll need to select whether you want to keep the old target table, or delete it. Deletion can't be undone.++**Questions built on the old target will _not_ be transferred to the new target table.** If you delete the old target table, any questions using the old transform target table will break. If you keep the old target around, the questions built on it won't break but they will _not_ use the new target table, and so will become outdated.++## Run a transform++You can run a transform manually or schedule the transform using tags and jobs.++Running a transform for the first time will create and sync the table created by the transform, and you'll be able to edit the table's [metadata](../../data-modeling/metadata-editing.md) and [permissions](../../permissions/data.md). Subsequent runs will drop and recreate the table, unless you use [Incremental transforms](#incremental-transforms).++To run a transform manually, visit the transform in **Data Studio > Transforms > Runs** and click **Run**.++To schedule a transform, you'll need to assign one or more tags to it, then create a [scheduled job](./jobs-and-runs.md) that picks up those tags.++You can see the time and status of the latest transform run on the transform's page, or in the [Runs view](./jobs-and-runs.md). The time of the run is given in the system's timezone.++For Python transforms, you'll also see the transform's execution logs.++## Transform dependencies++> Transform dependency graph is only available on Pro and Enterprise plans (both self-hosted and Metabase Cloud)++_Data Studio > Transforms > Dependencies_++Transform queries can use the data from other transforms, and query-based transforms can also reference Metabase questions and models. For example, you can have a transform that uses data from a `raw_events` table and writes to a `stg_events` table, and then create another transform that uses data from the `stg_events` table and writes to an `events` table.++Metabase will track transform dependencies and execute transforms in a reasonable order. So for example, if transform B relies on a table created by transform A, Metabase will run transform A first, then run transform B.++On Metabase Pro or Enterprise plans, you can see the transform dependencies graph by going to **Data Studio > Transforms** and go to the **Dependencies** tab.++If a job includes a transform that depends on a table created by another transform, then the job will run all the tagged transforms and their dependencies, even if they lack tags, see [Jobs and runs](jobs-and-runs.md) for more information.++### Incremental transforms++_Data Studio > Transforms > Settings_++Incremental transforms only append new data since the previous transform run. For example, you might have new transaction data coming in every day, and run the transform nightly. With each run, the incremental transform would only write the rows added after the previous run the night before.++### Prerequisites for incremental transforms++- There is a column in your data that Metabase can check for new values to determine which data is new. We'll refer to this as a "Checkpoint" column.+- The checkpoint column has to have increasing values, like a sequential ID or timestamp column. Metabase will determine what "new" data is by looking for values that are _greater than_ already-written checkpoint values.+- The checkpoint column should be present in both input and output table.+- Your schema is stable, meaning that the structure of the tables is not going to change from run to run.++### Make a transform incremental++Incremental transforms work differently for query-based transforms and Python transforms, so see [incremental query transforms](query-transforms.md#incremental-query-transforms) and [incremental Python transforms](./python-transforms.md#incremental-python-transforms) for more information.++## Versioning transforms++_Admin settings > General > Remote sync_++You can check your transforms into git with [Remote Sync](../../installation-and-operation/remote-sync.md). If you enable transform sync, Metabase will serialize transforms as YAML files and push them to your specified GitHub repo branch.++To enable git sync of transforms:++1. Go to Admin settings by click on the **grid** icon in top right and select **Admin**.+2. On the **General** tab, pick **Remote sync** in the left sidebar.+3. Follow the steps to [Set up Remote Sync](../../installation-and-operation/remote-sync.md), and toggle "Transforms" sync on.++Keep in mind that this setting only controls whether transforms are checked _into_ the git repo. The transform sync setting does _not_ affect how the instance behaves in read-only mode. If your instance is in read-only mode, you will not be able to create or edit transforms.++## Transforms vs models++Transforms are similar to models with model persistence turned on, but there are a few crucial differences:++- Transforms can only be created by [analysts with transform permissions](../../permissions/data.md). Models can be created by anyone with permissions to create queries on the data source (but only admins can enable model persistence on an instance).+- You can choose the target schema and tables for transforms. Model persistence will create its own schema and tables.+- Transforms support more databases than model persistence.+- You can use Python to create transforms.++Use models to enable non-admins to create their own datasets within Metabase, and to add context like field descriptions and semantic types. Use transforms to create persisted datasets in your database and reuse them across Metabase. In future versions of Metabase, model persistence will be deprecated in favor of transforms.diff --git a/docs/databases/users-roles-privileges.md b/docs/databases/users-roles-privileges.mdindex 2bb8d6f6c62..27f04d55649 100644--- a/docs/databases/users-roles-privileges.md+++ b/docs/databases/users-roles-privileges.md@@ -137,13 +137,13 @@ GRANT metabase_model_persistence TO metabase;## Privileges to enable transforms-[Transforms](../data-modeling/transforms.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables.+[Transforms](../data-studio/transforms/transforms-overview.md) let Metabase write query results back to your database. We suggest that you create a dedicated schema for your transforms. Metabase's database user will need to be able to create and drop transform tables.In addition to the [minimum database privileges](#minimum-database-privileges):- Create a new role called `metabase_transforms`.- Create a dedicated schema for transforms.-- Give the `metabase_transforms` role `CREATE TABLE` / `ALTER` / `DROP` privileges for the specified schema+- Give the `metabase_transforms` role `CREATE TABLE` / `ALTER` / `DROP` privileges for the specified schema.- Give the `metabase_transforms` role to the `metabase` user.In certain cases, Metabase will give you an option to create new schemas for transforms, in which case you'll also need `CREATE SCHEMA` privileges for your metabase users (or `CREATE DATABASE` for ClickHouse).diff --git a/docs/installation-and-operation/remote-sync.md b/docs/installation-and-operation/remote-sync.mdindex fb0da0f887e..34a7e37ef5a 100644--- a/docs/installation-and-operation/remote-sync.md+++ b/docs/installation-and-operation/remote-sync.md@@ -164,7 +164,7 @@ At this point, you should be all set up. Exit Admin settings, then reload your b### 8. Configure transforms syncing (optional)-To version control your data transformation logic, you can sync your [Transforms](../data-modeling/transforms.md) including all your tags and jobs. Transform syncing is all or nothing: Metabase will sync your entire transforms namespace. You can't selectively sync specific transform folders.+To version control your data transformation logic, you can sync your [Transforms](../data-studio/transforms/transforms-overview.md) including all your tags and jobs. Transform syncing is all or nothing: Metabase will sync your entire transforms namespace. You can't selectively sync specific transform folders.## An example dev-to-production workflow@@ -416,4 +416,9 @@ If you want to switch fully to Remote Sync, we recommend starting with a new rep3. Move the content you want to sync into a synced collection.4. Push up your changes to the new repo.-Remote Sync doesn't sync table metadata, so if you're importing and exporting your [table metadata](../data-modeling/metadata-editing.md), you should stick with serialization.+Remote Sync does NOT sync table metadata, so if you're importing and exporting your [table metadata](../data-modeling/metadata-editing.md), you should stick with serialization.++## Further reading++- [Library](../data-studio/library.md)+- [Serialization](./serialization.md)diff --git a/docs/permissions/row-and-column-security.md b/docs/permissions/row-and-column-security.mdindex 24a397265c9..96d4814937b 100644--- a/docs/permissions/row-and-column-security.md+++ b/docs/permissions/row-and-column-security.md@@ -195,7 +195,7 @@ Learn more about [SQL parameters](../questions/native-editor/sql-parameters.md)For example, say have a table like this:| User_ID | Value |-|---------|-------|+| ------- | ----- || 1 | 10 || 1 | 50 || 2 | 5 |@@ -315,7 +315,7 @@ If you can't use SQL casting in Metabase, create a view in your database that co#### Option 3: Use transforms- Use a [transform](../data-modeling/transforms.md) to create a table that casts the advanced data type to a basic type. Then set up row and column security on the transformed table instead. You'll also need to block the original table.+Use a [transform](../data-studio/transforms/transforms-overview.md) to create a table that casts the advanced data type to a basic type. Then set up row and column security on the transformed table instead. You'll also need to block the original table.### People with row and column security can't create Slack subscriptions or alerts