The Postgres development platform. Supabase gives you a dedicated Postgres database to build your web, mobile, and AI applications.
TypeScript aialternativeauthdatabasedenoembeddingsexamplefirebasenextjsoauth2pgvectorpostgispostgrespostgresqlpostgrestrealtimesupabasevectorswebsockets
Improve ETL docs (#46850)
Riccardo Busetti8519494741e5
diff --git a/apps/docs/content/guides/database/replication/bigquery.mdx b/apps/docs/content/guides/database/replication/bigquery.mdxindex c583f3a095e..b224b2185cb 100644--- a/apps/docs/content/guides/database/replication/bigquery.mdx+++ b/apps/docs/content/guides/database/replication/bigquery.mdx@@ -29,17 +29,22 @@ Before configuring BigQuery as a destination, set up the following in Google Clo3. **GCP service account key**: Create a [service account](https://cloud.google.com/iam/docs/keys-create-delete) with appropriate permissions- Go to **IAM & Admin > Service Accounts**- Click **Create Service Account**- - Grant the "BigQuery Data Editor" role+ - Grant the "BigQuery Data Editor" and "BigQuery Job User" roles- Create and download the JSON key fileRequired permissions:- `bigquery.datasets.get`+- `bigquery.jobs.create`- `bigquery.tables.create`+- `bigquery.tables.delete`- `bigquery.tables.get`- `bigquery.tables.getData`+- `bigquery.tables.list`- `bigquery.tables.update`- `bigquery.tables.updateData`+- `bigquery.routines.get`+- `bigquery.routines.list`## Configure BigQuery as a destination@@ -89,15 +94,21 @@ BigQuery replication requires each source table to have a primary key, and the pBigQuery primary keys are `NOT ENFORCED`, and BigQuery CDC supports composite primary keys with up to 16 columns. Your source primary key must stay unique and non-null because BigQuery uses it to match CDC rows.-Source tables must also use a BigQuery-compatible Postgres `REPLICA IDENTITY` setting:+Source tables must also use a BigQuery-compatible Postgres `REPLICA IDENTITY` setting. Most tables can keep the Postgres default, as long as they have a primary key and all primary-key columns are included in the publication.-- `DEFAULT` with a primary key is supported and works for most tables.-- `FULL` is supported and is recommended for tables with large `text`, `jsonb`, `bytea`, or other values that Postgres may store out-of-line using TOAST.-- `USING INDEX`, `NOTHING`, or `DEFAULT` without a primary key are not supported for BigQuery replication.+| Source table setting | BigQuery support | Guidance |+| ------------------------------------------------ | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `REPLICA IDENTITY DEFAULT` with a primary key | Supported | Recommended for most tables. BigQuery uses the replicated source primary key to apply upserts and deletes. |+| `REPLICA IDENTITY FULL` | Supported | Recommended for tables with large `text`, `jsonb`, `bytea`, or other values that Postgres may store out-of-line using TOAST, especially when those rows update. |+| `REPLICA IDENTITY USING INDEX` | Not supported | BigQuery CDC rows are keyed by the source primary key, not by an alternative unique index. |+| `REPLICA IDENTITY NOTHING` | Not supported | Updates and deletes do not include enough row identity for BigQuery to apply them safely. |+| `REPLICA IDENTITY DEFAULT` without a primary key | Not supported | BigQuery requires a source primary key. |For a general explanation of how replica identity affects update and delete events, see [How does replica identity affect updates and deletes?](/docs/guides/database/replication/external-replication-faq#how-does-replica-identity-affect-updates-and-deletes).-For updates, Postgres does not always send a complete old row through logical replication. It can also mark unchanged toasted values as `unchanged toast` instead of resending the value. The replication pipeline can reconstruct a complete update when the old row image contains the missing value, which is reliable with `REPLICA IDENTITY FULL`. BigQuery CDC upserts require a complete new row, so updates can fail for tables with toasted columns if the pipeline receives only a partial update row.+For updates, Postgres does not always send a complete old row through logical replication. It can also mark unchanged toasted values as `unchanged toast` instead of resending the value. BigQuery CDC upserts require a complete new row because omitted columns are not preserved in the destination. The replication pipeline can reconstruct a complete update when the old row image contains the missing value, which is reliable with `REPLICA IDENTITY FULL`.++If a BigQuery pipeline fails with an error about a partial update row, set `REPLICA IDENTITY FULL` on the affected source table and restart the pipeline. Changing replica identity only affects new WAL records, so a retained update that was written before the change may still need to be skipped by recreating the pipeline or re-copying the affected table.Check a table's current replica identity:@@ -138,16 +149,22 @@ Schema change support for BigQuery is currently in beta. External replication suSupported schema changes:-- Adding a column+- Adding a nullable column- Removing a column- Renaming a column+- Dropping a `NOT NULL` constraint+- Setting or dropping supported column default metadata-Unsupported schema changes:+Unsupported or limited schema changes:- Changing a column's data type-- Replicating column default values+- Adding `NOT NULL` with `SET NOT NULL`+- Filling existing rows for `ADD COLUMN ... DEFAULT`+- Unsupported default expressions++BigQuery requires added columns to be nullable. When a replicated `ADD COLUMN` includes a default, external replication can apply supported default metadata for future rows, but BigQuery does not backfill existing rows through that DDL. Existing destination rows remain `NULL` unless you run a separate backfill.-We plan to expand schema change support over time as the feature evolves.+Supported defaults are best-effort translations to BigQuery SQL. Unsupported defaults are skipped with a warning instead of failing replication.## Limitationsdiff --git a/apps/docs/content/guides/database/replication/external-replication-faq.mdx b/apps/docs/content/guides/database/replication/external-replication-faq.mdxindex cc09c6336bf..b511cb07198 100644--- a/apps/docs/content/guides/database/replication/external-replication-faq.mdx+++ b/apps/docs/content/guides/database/replication/external-replication-faq.mdx@@ -45,11 +45,13 @@ Schema change support is currently in beta and limited to the BigQuery destinatiSupported schema changes:-- Adding a column+- Adding a nullable column- Removing a column- Renaming a column+- Dropping a `NOT NULL` constraint+- Setting or dropping supported column default metadata-External replication does not currently support changing column data types or replicating column default values. See [BigQuery schema change support](/docs/guides/database/replication/bigquery#schema-change-support) for details.+External replication does not currently support changing column data types, adding `NOT NULL` with `SET NOT NULL`, or filling existing destination rows for `ADD COLUMN ... DEFAULT`. Supported defaults are applied as destination metadata for future rows where BigQuery can represent them. See [BigQuery schema change support](/docs/guides/database/replication/bigquery#schema-change-support) for details.## What happens when you disable external replication?@@ -69,6 +71,14 @@ Common reasons:Check your publication settings and verify your table meets the requirements.+## Why are partitioned tables replicated as separate tables?++Postgres controls this with the publication's `publish_via_partition_root` setting. If the setting is `false`, or if you created the publication manually with SQL and did not set it, Postgres publishes changes from the leaf partitions. External replication then creates destination tables for those leaf partitions. If `publish_via_partition_root = true`, Postgres publishes changes as the partition root, so external replication treats the partition hierarchy as the published partition root.++Publications created from the Dashboard replication flow use `publish_via_partition_root = true`.++See [Partitioned tables](/docs/guides/database/replication/external-replication-setup#partitioned-tables) for examples and the full behavior.+## How does replica identity affect updates and deletes?If inserts replicate but updates or deletes fail, check the table's `REPLICA IDENTITY` setting.@@ -132,6 +142,20 @@ Pipeline failures occur during the streaming phase when an error happens while rSee [Handling errors](/docs/guides/database/replication/external-replication-monitoring#handling-errors) for more details.+## Why is replication lag increasing?++Lag increases when Postgres produces WAL faster than the pipeline can confirm it has processed. Common causes include a slow or rate-limited destination, a pipeline issue, heavy source database activity, long transactions, network latency between the pipeline and source database, or a stopped/disconnected pipeline.++Open [**Database > Replication**](/dashboard/project/_/database/replication), click **View status**, and check **Waiting to sync**, **Room before pausing**, **Last check-in**, **Connected**, and **Slot status**. See [Dealing with replication lag](/docs/guides/database/replication/external-replication-monitoring#dealing-with-replication-lag) for the full investigation and response flow.++## What does a `Lost` slot status mean?++`Lost` means Postgres has already removed WAL files that the pipeline's replication slot needed. The pipeline cannot continue from that slot.++You can recreate the pipeline, or open the pipeline's **Advanced settings**, set **Invalidated slot behavior** to **Recreate**, and restart the pipeline. On restart, the pipeline creates a new replication slot and starts replication from scratch for all tables. This is required for consistency because the old slot can no longer provide every change the pipeline missed.++See [Slot statuses](/docs/guides/database/replication/external-replication-monitoring#slot-statuses) for all slot states and what to do next.+## Why is a table in error state?Table errors occur during the copy phase. To recover, click **View status**, find the affected table, and reset the table state. This will restart the table copy from the beginning.diff --git a/apps/docs/content/guides/database/replication/external-replication-monitoring.mdx b/apps/docs/content/guides/database/replication/external-replication-monitoring.mdxindex 12e460f88cf..eb22ca4c65c 100644--- a/apps/docs/content/guides/database/replication/external-replication-monitoring.mdx+++ b/apps/docs/content/guides/database/replication/external-replication-monitoring.mdx@@ -55,7 +55,35 @@ height={2146}#### Replication lag metrics-The status page shows replication lag metrics that help you determine how fast your pipeline is replicating data. These metrics are loaded directly from Postgres itself.+The status page shows replication lag metrics that help you determine how far the pipeline is behind Postgres. These metrics are loaded directly from Postgres replication slot state.++The destinations list also shows a compact lag value. This value is byte-based: it shows how much WAL the pipeline has not confirmed as flushed yet. A value of **Caught up** means the pipeline has confirmed every change currently available for its slot.++The detailed status page shows:++| Metric | What it means | What to watch for |+| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |+| **Waiting to sync** | Bytes of WAL between the pipeline's confirmed flush position and the current Postgres WAL position. This is the main byte-based replication lag. | A value that keeps growing means the pipeline is receiving changes more slowly than Postgres produces them. |+| **Room before pausing** | How much WAL can still accumulate before Postgres can no longer safely keep all WAL needed by the replication slot. This is controlled by `max_slot_wal_keep_size`. | A small or shrinking value means the slot is getting closer to being invalidated. `Unlimited` means Postgres is not reporting a slot WAL retention limit. |+| **Last check-in** | How long it has been since the pipeline last sent replication feedback to Postgres. | An old value can mean the pipeline is stopped, disconnected, overloaded, or unable to make progress. |+| **Connected** | Whether the pipeline's replication slot is active and currently being used. | `Not connected` while the pipeline should be running usually means you should check pipeline status and logs. |+| **Slot status** | How safely Postgres is keeping the WAL files the pipeline still needs. | `Unreserved` and `Lost` require action. See [Slot statuses](#slot-statuses). |++External replication uses one main pipeline replication slot for ongoing changes. During the initial copy phase, it can also create temporary table-sync replication slots. These temporary slots let multiple tables copy in parallel, make large table copies faster, and allow individual tables to be retried or copied again without restarting the whole pipeline.++Temporary table-sync slots show the same kind of lag and slot health metrics while they are active. After a table finishes copying and catches up, its temporary slot is removed and ongoing changes continue through the main pipeline slot. For overall replication health, focus first on the main pipeline slot.++#### Slot statuses++Replication slot status tells you whether Postgres is still retaining the WAL that the pipeline needs to continue from its current position.++| Status | Meaning |+| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| **Reserved** | Healthy. Postgres is keeping the WAL files this pipeline's replication slot needs, and they are within the normal WAL size limit. |+| **Extended** | Healthy, but growing. The slot is holding on to more WAL than usual, but Postgres is still keeping everything the pipeline needs. |+| **Unreserved** | At risk. Postgres is no longer reserving all WAL files this pipeline's replication slot needs. If the pipeline does not catch up soon, those files may be removed. |+| **Lost** | Broken. Some WAL files this pipeline's replication slot needs have already been removed. The pipeline can no longer continue from this slot. Recreate the pipeline, or set **Invalidated slot behavior** to **Recreate** in the pipeline's advanced settings and restart it. |+| **Unknown** | Postgres reported an unknown or unavailable state for this pipeline's replication slot. |#### Table states@@ -69,6 +97,57 @@ The pipeline status page also shows the state of individual tables being replica| **Live** | Table is now replicating data in near real-time || **Error** | Table has experienced an error during replication |+### Dealing with replication lag++Replication lag means the pipeline is behind the source database. Some lag is expected during the initial copy phase, after a burst of writes, or after restarting a stopped pipeline. Lag becomes a problem when it keeps increasing, when **Room before pausing** is running low, or when the slot status moves to **Unreserved** or **Lost**.++Lag can come from several places:++- **Destination throughput**: The destination is slow, rate-limited, unavailable, or rejecting writes.+- **Pipeline throughput**: The pipeline is overloaded, processing a very large transaction, or not performing as expected for the project workload.+- **Source database activity**: Postgres is producing WAL faster than the pipeline can consume it, often during bulk writes, migrations, or backfill jobs.+- **Network latency**: Latency or instability between the pipeline and source database can slow down WAL streaming.+- **Stopped or disconnected pipeline**: When a pipeline is stopped, disconnected, or failed, Postgres keeps WAL for the slot until the retention limit is reached.+- **Slow initial table copy**: A temporary table-sync slot can fall behind if a table is copied more slowly than new changes are written to that table.++#### Initial copy and table-sync slots++A common initial sync issue happens when a large or busy table is still in **Copying** while new rows keep being inserted or updated. The temporary table-sync slot needs to keep the changes that happen during the copy. If the copy is too slow compared to the table's write rate, the slot can move to **Unreserved** and then **Lost** if Postgres removes changes the copy still needs.++When a table-sync slot is lost, the affected table needs to be copied again. Tune the copy settings, then retry the table copy:++- Increase **Copy connections per table** when one large table is the bottleneck. This lets the pipeline copy chunks of that table in parallel, up to the point where the source database or network becomes the limit.+- Increase **Table sync workers** when several tables need to copy at the same time. Each worker can copy one table, and each worker uses an additional temporary replication slot during initial sync.+- If possible, run the initial copy during a quieter write period or reduce bulk writes until the table reaches **Live**.++After the affected table finishes copying and catches up, the temporary slot is deleted. The table then continues through the main pipeline replication slot.++#### Investigate the lag++1. Open [**Database > Replication**](/dashboard/project/_/database/replication) and check the destination's lag column.+2. Click **View status** and check **Waiting to sync**, **Room before pausing**, **Last check-in**, **Connected**, and **Slot status**.+3. Check table states. Tables in **Copying** can create temporary lag while the initial snapshot catches up to live changes. If a table-sync slot is **Unreserved** or **Lost**, tune copy parallelism and retry the affected table copy.+4. Open [**Logs > Replication**](/dashboard/project/_/logs/explorer) and look for destination errors, retries, rate limits, schema errors, or repeated restarts.+5. Compare the lag trend with recent database activity, such as imports, migrations, bulk updates, or long transactions.++#### Respond based on the slot status++| Slot status | What to do |+| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| **Reserved** | If **Waiting to sync** is stable or decreasing, continue monitoring. If it keeps increasing, check destination write performance, logs, and whether the publication includes more tables or write volume than expected. |+| **Extended** | Treat it as an early warning. Confirm the pipeline is connected, check logs for retries or destination slowness, and reduce avoidable write bursts if possible until the pipeline catches up. |+| **Unreserved** | Act quickly. The slot is at risk of losing required WAL. Check whether the pipeline is connected and making progress, fix destination or pipeline errors, and contact support if the lag continues to grow. |+| **Lost** | The pipeline cannot continue from the existing slot because required WAL has been removed. Recreate the pipeline, or set **Invalidated slot behavior** to **Recreate** in the pipeline's advanced settings and restart the pipeline. This creates a new slot and starts replication from scratch for all tables. |+| **Unknown** | Check replication logs for errors or missing slot details. If the status remains unknown while the pipeline should be running, contact support with the pipeline ID and recent log details. |++#### Reduce future lag risk++- Keep publications focused on the tables and operations you need at the destination.+- Avoid leaving pipelines stopped for long periods while the source database is still receiving writes.+- Schedule bulk updates, imports, and migrations during lower-traffic windows when possible.+- For BigQuery, verify that service account permissions, table requirements, and replica identity settings match the [BigQuery destination guide](/docs/guides/database/replication/bigquery).+- If the initial copy is the bottleneck, review **Table sync workers** and **Copy connections per table** in the pipeline's advanced settings. Increasing them can speed up copying, but it uses more database connections and replication slots.+### Handling errorsErrors can occur at two levels: per table or per pipeline.diff --git a/apps/docs/content/guides/database/replication/external-replication-setup.mdx b/apps/docs/content/guides/database/replication/external-replication-setup.mdxindex f1c58b4303d..f82edd4f457 100644--- a/apps/docs/content/guides/database/replication/external-replication-setup.mdx+++ b/apps/docs/content/guides/database/replication/external-replication-setup.mdx@@ -88,6 +88,44 @@ create publication pub_recent_ordersfor table orders where (created_at > '2024-01-01');```+##### Partitioned tables++External replication follows Postgres publication semantics for partitioned tables. The `publish_via_partition_root` publication setting controls whether changes from partitions are emitted as the partition root or as the leaf partitions.++| Publication setting | What gets replicated | Destination shape |+| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------- |+| `publish_via_partition_root = true` | Rows from the published partition root, including rows stored in its leaf partitions | One table matching the published partition root |+| `publish_via_partition_root = false` | Rows from the leaf partitions under the published partition root | One table per replicated leaf partition |+| Not set in SQL | Same as `false`, because Postgres defaults `publish_via_partition_root` to `false` | One table per replicated leaf partition |+| Publishing an individual leaf partition | The leaf partition itself, regardless of `publish_via_partition_root` | One table for that leaf partition |+| `FOR ALL TABLES` or `FOR TABLES IN SCHEMA` | Partition roots plus regular tables when `true`; leaf partitions plus regular tables when `false` or unset | Destination tables follow the effective Postgres publication table list |++For example, if `orders` is partitioned by month:++```sql+-- Replicate the whole partition hierarchy as the parent table.+create publication pub_orders_root+for table orders+with (publish_via_partition_root = true);++-- Replicate each leaf partition as its own table.+create publication pub_orders_leaves+for table orders+with (publish_via_partition_root = false);+```++Use `publish_via_partition_root = true` when you want analytics queries to read from a single destination table that has the parent table's schema. Use `false` when each partition should remain a separate destination table.++Publications created from the Dashboard replication flow use `publish_via_partition_root = true`. If you create or alter a publication manually with SQL, set this option explicitly so the destination shape matches what you expect.++On Postgres 15 and newer, row filters on partition publications apply during both the initial copy and streaming phases. External replication uses the row filter attached to the effective publication table entry: the published partition root when `publish_via_partition_root = true`, and the published leaf relation when `publish_via_partition_root = false`.++<Admonition type="note">++With `publish_via_partition_root = true`, truncating an individual leaf partition is not replicated as a truncate event for the published parent. If you need truncate replication, run `TRUNCATE` on the published partition root.++</Admonition>+#### Viewing publications in the DashboardAfter creating a publication via SQL, you can view it in the Dashboard:@@ -132,15 +170,17 @@ Follow these steps to configure your destination. Each destination has its own s5. Optionally expand **Advanced settings** to tune pipeline behavior. These settings apply to the pipeline rather than the destination:- | Setting | Default | Description |- | ------------------------------ | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |- | **Batch wait time** | `10000` milliseconds | Maximum time the pipeline waits to collect additional changes before flushing them. Lower values reduce replication latency. Higher values can improve batching efficiency. |- | **Table sync workers** | `4` workers | Number of tables copied in parallel during the initial snapshot phase. Each worker uses one replication slot, up to `N + 1` total replication slots while syncing. |- | **Copy connections per table** | `2` connections | Number of parallel database connections each table copy can use during the initial sync. Increasing this can speed up large table copies, but uses more database connections. |- | **Invalidated slot behavior** | `Error` | What the pipeline does when its replication slot is invalidated. **Error** blocks startup so you can recover manually. **Recreate** rebuilds the slot and starts replication from scratch. |+ | Setting | Default | Description |+ | ------------------------------ | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+ | **Batch wait time** | `10000` milliseconds | Maximum time the pipeline waits to collect additional changes before flushing them. Lower values reduce replication latency. Higher values can improve batching efficiency. |+ | **Table sync workers** | `4` workers | Number of tables copied in parallel during the initial snapshot phase. Each worker uses one replication slot, up to `N + 1` total replication slots while syncing. |+ | **Copy connections per table** | `2` connections | Number of parallel database connections each table copy can use during the initial sync. Increasing this can speed up large table copies, but uses more database connections. |+ | **Invalidated slot behavior** | `Error` | What the pipeline does when its replication slot is invalidated. **Error** blocks startup so you can recover manually. **Recreate** rebuilds the slot on the next pipeline restart and starts replication from scratch for all tables. |Leave these settings at their defaults unless you need to tune initial copy speed, latency, or recovery behavior.+ Use **Invalidated slot behavior** carefully. If **Recreate** is selected and the pipeline restarts after Postgres has invalidated the main replication slot, the pipeline creates a new slot and copies all replicated tables again. This full restart is required for consistency because the old slot can no longer provide every change the pipeline missed.+6. Click **Create and start** to begin replicationYour replication pipeline now starts copying data from your database to your destination.