The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
TypeScript analyticsbibusiness-intelligencebusinessintelligenceclojuredashboarddatadata-analysisdata-visualizationdatabasemetabasemysqlpostgrespostgresqlreportingslacksql-editorvisualization
docs: new incremental transforms (#71080)
Alex Yarosh2f7a83ddc513
diff --git a/docs/data-studio/transforms/python-transforms.md b/docs/data-studio/transforms/python-transforms.mdindex b9396ef2751..aff313289ee 100644--- a/docs/data-studio/transforms/python-transforms.md+++ b/docs/data-studio/transforms/python-transforms.md@@ -120,9 +120,7 @@ To make a Python transform incremental:1. Go to the transform's page in **Data studio > Transforms**.2. Switch to **Settings** tab.-3. In **Column to check for new values**, select the column that Metabase should check to determine which values are new. See [Prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms) for more information on the requirements for that column.-- Unlike [Query transforms](./query-transforms.md), where you select an _output_ column as the column to check for new values, with Python transforms, you have to select a column from the _input_ tables as the column to check for new values.+3. In **Column to check for new values**, select the column in one of the source tables that Metabase should check to determine which values are new. Only some columns are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).## Current limitations of Python transformsdiff --git a/docs/data-studio/transforms/query-transforms.md b/docs/data-studio/transforms/query-transforms.mdindex f00a9850e95..976ce6eb835 100644--- a/docs/data-studio/transforms/query-transforms.md+++ b/docs/data-studio/transforms/query-transforms.md@@ -105,34 +105,50 @@ For a transform to run incrementally, you'll need to pick a column ("checkpoint"### Make a query transform incremental-To make a query transform incremental:+#### Use table variables for incremental SQL transforms-1. Go to the transform's page in **Data studio > Transforms**.-2. Switch to **Settings** tab.-3. In **Column to check for new values**, select the column that Metabase should check to determine which values are new. See [Prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms) for more information on the requirements for that column.+If you built your transform in the graphical query builder, you can skip right to [marking the transform as incremental](#mark-transform-as-incremental).- You have to select the column from the list of the columns of the _output_ tables. Note: this is different from [Python transforms](python-transforms.md), where you select an _input_ column as column to check for new values.+If you are writing your incremental transform in raw SQL, you'll need to add a [table variable](../../questions/native-editor/table-variables.md) into your SQL code, with the table variable replacing the table with the checkpoint column.- If you're using SQL, Metabase might tell you that your query is too complicated to automatically make the transform incremental. In this case, you need to add the filter for new values manually. For example, let's say you have a transform query:+For example, let's say you have a transform that retrieves order id, total and product title:- ```sql- SELECT id, total FROM orders;- ```+```sql+SELECT+ orders.id,+ orders.total,+ products.title+FROM+ orders JOIN products on orders.product_id = products.id+```- (This query is actually simple enough for Metabase to handle it automatically, we're just using it as an example)+To make this transform incrementally load the data based on new values of `orders.id` column, you need to:- If you want use the `id` column to check for new values, i.e. only write back the records with `id` greater than already existing `id`, you can add a manual filter like this:+1. Add a table variable, for example `{{orders_var}}` replacing `orders` in the `FROM` statement;+2. In the table variable settings, connect the table variable to the `orders` table;+3. Replace other references to the table in your query with either:+ - The name of the table variable (if you have "Emit table alias" toggled on in variable's setting).+ - Your own handcrafted alias for the variable.- ```sql- SELECT id, total FROM orders- {%raw%}[[WHERE id > {{checkpoint}}]]{% endraw %}- ```+So your query will look like this:- and then select `id` as the **Column to check for new values** in the incremental transform settings.+```sql+SELECT+ orders_var.id,+ orders_var.total,+ products.title+FROM+ {{orders_var}} JOIN products on orders_var.product_id = products.id+```++In this query,`orders_var` is connected to the `orders` table in variable settings, and "Emit table aliases" toggled on in the variables sidebar.- If you're using a timestamp column as a checkpoint, you'll need to explicitly cast it to timestamp:+Once your query has table aliases, you can mark the transform as incremental using the `orders.id` column in the transform's settings.- ```sql- SELECT created_at, total FROM orders- {%raw%}[[WHERE created_at > {{checkpoint}}::timestamp]]{% endraw %}- ```+#### Mark transform as incremental++To make a query transform incremental:++1. Go to the transform's page in **Data studio > Transforms**.+2. Switch to **Settings** tab.+3. In **Column to check for new values**, select the column in one of the source tables that Metabase should check to determine which values are new. Only some columns are eligible. See [prerequisites for incremental transforms](./transforms-overview.md#prerequisites-for-incremental-transforms).diff --git a/docs/data-studio/transforms/transforms-overview.md b/docs/data-studio/transforms/transforms-overview.mdindex 76f5e424254..2a2c1007363 100644--- a/docs/data-studio/transforms/transforms-overview.md+++ b/docs/data-studio/transforms/transforms-overview.md@@ -203,7 +203,6 @@ Incremental transforms only append new data since the previous transform run. Fo- There is a column in your data that Metabase can check for new values to determine which data is new. We'll refer to this as a "Checkpoint" column.- The checkpoint column has to have increasing values, like a sequential ID or timestamp column. Metabase will determine what "new" data is by looking for values that are _greater than_ already-written checkpoint values.-- The checkpoint column should be present in both input and output table.- Your schema is stable, meaning that the structure of the tables is not going to change from run to run.### Make a transform incrementaldiff --git a/docs/questions/native-editor/table-variables.md b/docs/questions/native-editor/table-variables.mdindex 43cf91371ae..00ae5f778d8 100644--- a/docs/questions/native-editor/table-variables.md+++ b/docs/questions/native-editor/table-variables.md@@ -37,7 +37,7 @@ FROM{% endraw %}```-Metabase inserts the table reference using the schema-qualified name (e.g., `public.orders`), so some databases like PostgreSQL will require you to use an alias. For example: `SELECT t.id FROM {% raw %}{{table}}{% endraw %} t`.+See [Referencing table variables](#reference-table-variables).### Setting the variable type to Table@@ -46,9 +46,40 @@ After adding a `{% raw %}{{variable}}{% endraw %}` to your query:1. Open the **Variables** sidebar (it should appear automatically).2. Change the variable type to **Table**.3. Under **Table to map to**, select a table from the picker (Required).+4. Depending on how you want to refer to the table variable in the query, toggle **Emit table alias** on or off, see [Referencing table variables](#reference-table-variables).When you run the query, Metabase replaces the variable with the selected table's schema and table name. To preview the code Metabase will run, click the **eye** icon.+## Reference table variables++There are two ways you can refer to the table variable in the rest of the query:++1. If you want to use the variable's name, you'll need to toggle _on_ **Emit table alias** in the variable's settings. Your queries will look like this:++ ```sql+ {% raw %}+ SELECT+ var_name.id,+ p.title+ FROM+ {{var_name}} JOIN products as p on var_name.product_id = p.id++ {% endraw %}+ ```++2. If you want to specify an alias, you must toggle **Emit table alias** _off_ and manually add aliases to your query. You may want to specify your own alias if you already have a long query with existing aliases, and you just want to swap the table for a table variable. Your queries will look like this:++ ```sql+ {% raw %}+ SELECT+ o.id,+ p.title+ FROM+ {{var_name}} as o JOIN products as p on o.product_id = p.id++ {% endraw %}+ ```+## Using table variables with snippetsOne neat thing you can do is combine table variables with [snippets](./snippets.md). You can write a generic query as a snippet and reuse the snippet in multiple questions, each mapped to a different table.